ang3/doctrine-orm-batch-process

Doctrine ORM Batch component

v1.0.0 2023-06-19 14:58 UTC

Requires

• php: >=8.1
• doctrine/orm: ^2.12

Requires (Dev)

• symfony/test-pack: ^1.0

MIT

• Joanis ROUANET

README

This component helps you to deal with batch-processing in the context of Doctrine ORM transactions.

The main problem with bulk operations is usually not to run out of memory and this is especially what the strategies presented here provide help with. -- Doctrine ORM documentation

The batch process component allows you to build bulk operations very easily with some advanced features.

Installation

Open a command console, enter your project directory and execute the following command to download the latest stable version of this bundle:

$ composer require ang3/doctrine-orm-batch-process

This command requires you to have Composer installed globally, as explained in the installation chapter of the Composer documentation.

Usage

A batch process is a basic loop using an iterator to do some logic with a handler. The iterator is mandatory and the handler is optional.

In all cases, the entity manager is flushed and cleared when the buffer size is reached (by default: 20).

Create a process

To create a process, use the constructor with the entity manager and an iterator, or use one of defined static shortcut methods as to your needs:

use Ang3\Doctrine\ORM\Batch\BatchProcess;

$myProcess = new BatchProcess($entityManager, $myIterator, $myHandler = null, $bufferSize = 20);

// For chaining calls
$myProcess = BatchProcess::create($entityManager, $myIterator);

/*
 * SHORTCUT METHODS FOR BUILT-IN ITERATORS
 */

// Iterate from identified entities
$myProcess = BatchProcess::iterateEntities($entityManager, $entityFqcn, array $identifiers);

// Iterate from an ORM query or query builder instance
$myProcess = BatchProcess::iterateQueryResult($query);

// Iterate from iterable or callable
$myProcess = BatchProcess::iterateData($entityManager, iterable|\Closure $data);

To execute the process:

$nbIterations = $myProcess->execute();

Advanced options

This component provides some useful advanced options.

Handlers

A handler is a class implementing the interface Ang3\Doctrine\ORM\Batch\Handler\BatchHandlerInterface. This class is called on each iteration. It allows you to persist/remove entities, or whatever you need with custom handlers.

This interface describes the method called by the batch process: __invoke(\Ang3\Doctrine\ORM\Batch\BatchIteration $iteration).

The component provides some useful built-in handlers.

Persist entity handler

This handler persists entities into database.

use Ang3\Doctrine\ORM\Batch\Handler\PersistEntityHandler;

$myHandler = PersistEntityHandler::new()
    // Handler options...
    ->skipInsertions() // Ignore new entities (without ID)
    ->skipUpdates() // Ignore stored entities (with ID)
    ->onPrePersist($myCallable) // This callable is called BEFORE each persist.
    ->onPostPersist($myCallable) // This callable is called AFTER each persist.
;

Callable arguments:

1. object The entity
2. Ang3\Doctrine\ORM\Batch\BatchIteration The iteration.

Remove entity handler

This handler removes entities into database.

use Ang3\Doctrine\ORM\Batch\Handler\RemoveEntityHandler;

$myHandler = RemoveEntityHandler::new()
    // Handler options...
    ->onPreRemove($myCallable) // This callable is called BEFORE each removing.
    ->onPostRemove($myCallable) // This callable is called AFTER each removing.
;

Callable arguments:

1. object The entity
2. Ang3\Doctrine\ORM\Batch\BatchIteration The iteration.

Callable handler

This handler holds a simple callable.

use Ang3\Doctrine\ORM\Batch\Handler\CallableHandler;

$myHandler = CallableHandler::new($myCallable);

Callable arguments:

1. mixed Iterator data
2. Ang3\Doctrine\ORM\Batch\BatchIteration The iteration.

Chain handler

This handler allows you to chain handlers in a specific order.

use Ang3\Doctrine\ORM\Batch\Handler\ChainHandler;

$myHandler = ChainHandler::new()
    // Handler options...
    ->append($myHandler1) // Add a handler at the end of the chain
    ->prepend($myHandler2) // Add a handler at the beginning of the chain
    ->clear() // Remove all handlers from the chain
;

Custom handler

Create a class implementing the interface Ang3\Doctrine\ORM\Batch\Handler\BatchHandlerInterface.

namespace App\Doctrine\Batch\Handler;

use Ang3\Doctrine\ORM\Batch\Handler\BatchHandlerInterface;

final class MyHandler implements BatchHandlerInterface
{
    public function __invoke(Iteration $iteration): void
    {
        // Data from the process iterator
        $data = $iteration->getData();
        
        // You can retrieve the entity manager from the iteration.
        $em = $iteration->getEntityManager();
    }
}

Use the trait Ang3\Doctrine\ORM\Batch\Handler\BatchHandlerTrait to enable options:

// ...
final class MyHandler implements BatchHandlerInterface
{
    // If you want to add options:
    use BatchHandlerTrait;
    
    // Internal constants used to define options
    private const OPTION_ENABLED = 'enabled';
    
    public function __invoke(Iteration $iteration): void
    {
        // Retrieves option values
        $enabled = $this->getOption(self::OPTION_ENABLED);
        
        // your logic with the option...
    }

    // Return the handler to allow chaining calls
    public function enableOption(bool $enabled = true): self
    {
       $this->setOption(self::OPTION_ENABLED, $enabled);
    
       return $this;
    }
}

Buffer size

You can modify the default buffer size:

$myProcess->setBufferSize(50); // By default: 20

Disable ID generators

You can turn off the ID generator for one or more entities, useful to migrate data.

$myProcess->disableIdGenerator(MyClass1::class, MyClass2::class/*, ...*/);

You can restore some disabled ID generators like below:

$myProcess->restoreIdGenerator(MyClass1::class, MyClass2::class/*, ...*/);

// ... Or restore all directly
$myProcess->restoreAllIdGenerators();

Transactional entities

The entity manager could be cleared many times. That's why during the process, all entities loaded before the execution will be detached from the manager.

$myLoadedEntity; // attached
$myProcess->execute();
dump($myLoadedEntity); // Probably detached - If you persist, a new entity could be created!

To avoid reloading your entities, this component helps you to keep your entities in memory (by variable reference) in order to reload it automatically.

$myLoadedEntity; // attached
$myProcess
    ->addTransactionalEntity($myLoadedEntity) // This variable is passed by reference
    ->execute();
dump($myLoadedEntity); // reloaded by the process on each flush/clear

On flush

You can pass a callable to the batch to execute some logic on each flush/clear operations.

$myProcess->onFlush($myCallable);

Callable arguments:

1. Ang3\Doctrine\ORM\Batch\BatchProcess The current batch
2. Ang3\Doctrine\ORM\Batch\BatchIteration|null The iteration (NULL in case of last flush/clear).