ocramius / doctrine-batch-utils
A set of utilities to operate with Doctrine ORM's batch processing functionality
Fund package maintenance!
Ocramius
Installs: 1 161 050
Dependents: 7
Suggesters: 0
Security: 0
Stars: 324
Watchers: 7
Forks: 14
Open Issues: 4
Requires
- php: ~8.1.0 || ~8.2.0 || ~8.3.0
- doctrine/orm: ^3.2.1
- doctrine/persistence: ^3.3.3
Requires (Dev)
- doctrine/coding-standard: ^12.0.0
- doctrine/dbal: ^4.0.4
- phpunit/phpunit: ^9.6.19
- psalm/plugin-phpunit: ^0.19.0
- roave/infection-static-analysis-plugin: ^1.35.0
- vimeo/psalm: ^5.25.0
- 2.9.x-dev
- 2.8.x-dev
- 2.8.0
- 2.7.x-dev
- 2.7.0
- 2.6.x-dev
- 2.6.0
- 2.5.x-dev
- 2.5.0
- 2.4.x-dev
- 2.4.0
- 2.3.x-dev
- 2.3.0
- 2.2.x-dev
- 2.2.0
- 2.1.x-dev
- 2.1.0
- 2.0.x-dev
- 2.0.0
- 1.1.x-dev
- 1.1.0
- 1.0.x-dev
- 1.0.0
- dev-renovate/all-minor-patch
- dev-renovate/phpunit-phpunit-11.x
- dev-dependabot/composer/doctrine/orm-2.9.3
- dev-dependabot/add-v2-config-file
This package is auto-updated.
Last update: 2024-10-10 21:45:19 UTC
README
This repository attempts to ease the pain of dealing with batch-processing in the context of Doctrine ORM transactions.
This repository is maintained by Patrick Reimers (PReimers).
Installation
Supported installation method is via Composer:
composer require ocramius/doctrine-batch-utils
Current features
As it stands, the only implemented utility in this repository is an
IteratorAggregate
that
wraps around a DB transaction and calls
ObjectManager#flush()
and ObjectManager#clear()
on the given EntityManager
.
Example (array iteration)
It can be used as following:
use DoctrineBatchUtils\BatchProcessing\SimpleBatchIteratorAggregate; $object1 = $entityManager->find('Foo', 1); $object2 = $entityManager->find('Bar', 2); $iterable = SimpleBatchIteratorAggregate::fromArrayResult( [$object1, $object2], // items to iterate $entityManager, // the entity manager to operate on 100 // items to traverse before flushing/clearing ); foreach ($iterable as $record) { // operate on records here }
$record
freshness
Please note that the $record
inside the loop will always be "fresh"
(managed
state),
as the iterator re-fetches it on its own: this prevents you from having to
manually call ObjectManager#find()
on your own for every iteration.
Automatic flushing/clearing
In this example, the EntityManager
will be flushed and cleared only once,
but if there were more than 100 records, then it would flush (and clear) twice
or more.
Example (query/iterators)
The previous example is still not memory efficient, as we are operating on a pre-loaded array of objects loaded by the ORM.
We can use queries instead:
use DoctrineBatchUtils\BatchProcessing\SimpleBatchIteratorAggregate; $iterable = SimpleBatchIteratorAggregate::fromQuery( $entityManager->createQuery('SELECT f FROM Files f'), 100 // flush/clear after 100 iterations ); foreach ($iterable as $record) { // operate on records here }
Or our own iterator/generator:
use DoctrineBatchUtils\BatchProcessing\SimpleBatchIteratorAggregate; // This is where you'd persist/create/load your entities (a lot of them!) $results = function () { for ($i = 0; $i < 100000000; $i += 1) { yield new MyEntity($i); // note: identifier must exist in the DB } }; $iterable = SimpleBatchIteratorAggregate::fromTraversableResult( $results(), $entityManager, 100 // flush/clear after 100 iterations ); foreach ($iterable as $record) { // operate on records here } // eventually after all records have been processed, the assembled transaction will be committed to the database
Both of these approaches are much more memory efficient.