symfony-doge/veslo

A vacancy aggregation engine

0.5.2 2021-02-09 10:23 UTC

Requires

• php: >=7.2
• bunny/bunny: ^0.4.1
• doctrine/doctrine-bundle: ^1.6
• doctrine/doctrine-fixtures-bundle: ^3.1
• doctrine/doctrine-migrations-bundle: ^1.0
• doctrine/orm: ^2.5
• guzzlehttp/guzzle: ^6.3
• incenteev/composer-parameter-handler: ^2.0
• knplabs/knp-menu-bundle: ^2.2
• knplabs/knp-paginator-bundle: ^2.8
• nelmio/alice: ^3.5
• sensio/distribution-bundle: ^5.0.19
• sensio/framework-extra-bundle: ^5.0.0
• shivas/versioning-bundle: ^3.1
• stof/doctrine-extensions-bundle: ^1.3
• symfony-doge/ministry-of-truth-client: ^0.1.1
• symfony/monolog-bundle: ^3.1.0
• symfony/polyfill-apcu: ^1.0
• symfony/swiftmailer-bundle: ^2.6.4
• symfony/symfony: 3.4.*
• twig/extensions: ^1.5
• twig/twig: ^1.0||^2.0

Requires (Dev)

• codeception/assert-throws: ^1.0
• codeception/codeception: ^4.1
• codeception/module-asserts: ^1.3
• codeception/module-webdriver: ^1.2
• codeception/specify: ^1.1
• codeception/verify: ^1.0
• php-ds/php-ds: ^1.2
• sensio/generator-bundle: ^3.0
• symfony/phpunit-bridge: ^3.0

GPL-3.0-or-later

• Pavel Petrov

README

                                                  ________________________  
                                                 /                        \ 
,--.                                            /                          |
|  |___________________________________________/_____________________      |
|  |_________________________________________________________________)     |
|  |                                           \                           |
`--'                                            \                          |
                                                 \________________________/ 
        Veslo: a vacancy aggregation engine

Installation

Docker

The preferred way to install is through docker-compose. You need to have a Docker daemon at least 17.05.0-ce (with build-time ARG in FROM) to successfully cook all containers.

Run an automated deploy script for local development with Docker.

$ bin/deploy_dev.sh

Roadmap configurations were not automatically loaded by the script, you need to insert them manually with a separate command.

$ docker-compose run --rm app bin/console doctrine:fixtures:load \
    --group roadmap.configuration.parameters \
    --append

Up to 120 sec may be required for starting up the vacancy parsing process.

Manual

You can clone and deploy the application with your own environment by providing .env file using .env.dev.dist as a template. Below are steps to prepare your application before accessing through a web server.

$ git clone [e-mail]:symfony-doge/veslo.git veslo && cd "$_"

// setup your own connection parameters
$ cp .env.dev.dist .env

$ cp app/config/parameters.yml.dist app/config/parameters.yml
$ cp webpack.env.dev.js.dist webpack.env.js

Check your environment for Symfony requirements and install PHP dependencies via Composer. You also need the php-ds extension for some services working with efficient PHP 7 data structures.

$ bin/symfony_requirements
$ composer install

Install Javascript dependencies and compile assets via Yarn (Node.js 10.x is required).

$ yarn install
$ yarn run build:dev

Apply database migrations.

$ bin/console doctrine:migrations:migrate latest

Load roadmap configurations.

$ bin/console doctrine:fixtures:load --group roadmap.configuration.parameters --append

Testing

Loading fixtures.

$ docker-compose run --rm app bin/console doctrine:fixtures:load --group test

Applying Codeception parameters.

$ cp parameters.codeception.dev.yml.dist parameters.codeception.yml

Executing tests.

$ docker-compose run --rm app bin/codecept run --steps

Workflow

FOUND (initial place)
|
| to_parse
 ------------> PARSED
               |
               | to_collect
                ------------> COLLECTED
                              |
                              | to_index
                               ------------> INDEXED

There is a set of log files suited for debugging an each workflow transition. var/logs contains:

API

$ docker-compose run --rm app bin/console veslo:anthill:digging hh php --iterations=1

Finds a vacancy from job site by roadmap (search plan) and configuration. Search result will be offered to parsing queue, according to current workflow (default veslo.app.workflow.vacancy_research.to_parse).

Available roadmaps are defined in AnthillBundle/Resources/config/roadmaps.yml and configurations in AnthillBundle/Resources/fixtures/roadmap/.

$ docker-compose run --rm app bin/console veslo:anthill:parsing --iterations=1

Polls a new raw vacancy data (html/json etc.) for parsing. Result will be offered to collecting queue (default veslo.app.workflow.vacancy_research.to_collect).

$ docker-compose run --rm app bin/console veslo:anthill:collecting --iterations=1

Grabs a parsed vacancy data (instance of AnthillBundle/Dto/Vacancy/RawDto) and decides whether should it be collected for analysis or not. Result will be persisted in the local storage and offered to the indexing queue (default veslo.app.workflow.vacancy_research.to_index)

$ docker-compose run --rm app bin/console veslo:sanity:indexing --iterations=1

Sends an accepted vacancy to the microservice for analysis (see ANALYSER_HOST, ANALYSER_PORT) and persists received metadata in the local storage. Remote service implements API defined by symfony-doge/ministry-of-truth-client bridge. Example (Go + Gin): symfony-doge/ministry-of-truth-cis.

Currently supported roadmaps:

• hh (configurations: php, javascript, golang, python, java, c++)

Usage example

www.veslo.it

See also

• symfony-doge/management-kit — A set of monitoring & management GUI tools to manage data & processes of this project.

Changelog

All notable changes to this project will be documented in CHANGELOG.md.