README

Improves performance of fos:elastica:populate command from FOSElasticaBundle by distributing the work among consumers.

This bundle is based on Enqueue Elastica Bundle, and provides a command to directly manage the "populate" with a set of sub-processes automatically. The performance gain depends on how much consumers you run. For example 10 consumers may give you 5 to 7 times better performance.

Installation

When installing the bundle, Enqueue ElasticaBundle is also automatically installed. It must then be configured in order to indicate the way in which the sub-processes will communicate.

Default enqueue.yaml:

enqueue:
    default:
        transport:
            dsn: '%env(resolve:ENQUEUE_DSN)%'
        client: ~
enqueue_elastica:
    transport: '%enqueue.default_transport%'
    doctrine: ~

Note: As long as you are on Symfony Flex you are done. If not, you have to do some extra things, like registering the bundle in your AppKernel class.

Usage

Run the populate command with some consumers (the more you run the better performance you might get):

php bin/console stm:fast-populate:populate --nb-subprocess=6

Customization

Options for the command

To limit the memory consumption of sub-processes you can use different parameters :

message-limit - Integer. Consume n messages and exit.
time-limit - Integer. Consume messages during this time.
memory-limit - Integer. Consume messages until process reaches this memory limit in MB.

You can also use the classic populate options : PopulateCommand.php

For example, to limit the memory consumption to 800MB for 8 sub-processes and process 1000 elements per page, you can run the command like this:

php bin/console app:elastica:populate --memory-limit=800 --max-per-page=1000 --nb-subprocess=8

Customizing the consumer

The QueuePagerPersister could be customized via options. The options could be customized in a listener subscribed on FOS\ElasticaBundle\Persister\Event\PrePersistEvent event for example.

Here's the list of available options:

max_per_page - Integer. Tells how many objects should be processed by a single worker at a time.
first_page - Integer. Tells from what page to start rebuilding the index.
last_page - Integer. Tells on what page to stop rebuilding the index.
populate_queue - String. It is a name of a populate queue. Workers should consume messages from it.
populate_reply_queue - String. It is a name of a reply queue. The command should consume replies from it. Persister tries to create a temporary queue if not set.
reply_receive_timeout - Float. A time a consumer waits for a message. In milliseconds.
limit_overall_reply_time - Int. Limits an overtime allowed processing time. Throws an exception if it is exceeded.

License

It is released under the MIT License.

steamulo / elastica-fast-populate-bundle

Maintainers

Details