oliverde8/php-etl-bundle

Allow usage of the PHP-ETL library in symfony framework.

v1.1.0-alpha8 2024-11-10 11:20 UTC

Requires

• php: >=8.0
• ext-json: *
• oliverde8/php-etl: ^v1.2.0-alpha1|^v2.0.0-alpha1
• symfony/framework-bundle: ^5.4|^6.0|^7.0
• symfony/messenger: ^5.4|^6.0|^7.0

Requires (Dev)

• symfony/console: ^6.0|^7.0
• symfony/monolog-bundle: ^3.7

MIT

• Oliver de Cramer

• symfony-ux

README

The Php etl bundle allows the usage of Oliver's PHP Etl library in symfony.

You should also check the PHP ETL's Easy Admin Bundle to have interfaces.

Installation

1. Install using composer

2. in /config/ create a directory etl

3. Enable bundle:

    \Oliverde8\PhpEtlBundle\Oliverde8PhpEtlBundle::class => ['all' => true],

1. Optional: Enable queue if you wish to allow users from the easy admin panel to do executions.

framework:
  messenger:
    routing:
        "Oliverde8\PhpEtlBundle\Message\EtlExecutionMessage": async

1. Optional: Enable creation of individual files for each log by editing the monolog.yaml

etl:
    type: service
    id: Oliverde8\PhpEtlBundle\Services\ChainExecutionLogger
    level: debug
    channels: ["!event"]

Usage

Creating an ETL chain

First read the documentation of the PHP ETL

Each chain is declare in a single file. The name of the chain is the name of the file created in /config/etl/. Example:

chain:
  "Dummy Step":
    operation: rule-engine-transformer
    options:
      add: true
      columns:
        test:
          rules:
            - get : {field: [0, 'uid']}

Executing a chain

./bin/console etl:execute demo '[["test1"],["test2"]]' '{"opt1": "val1"}'

The first argument is the input, depending on your chain it can be empty. The second are parameters that will be available in the context of each link in the chain.

Additional commands

Get a definition

./bin/console etl:get-definition demo

Adding your own chain operation

To add your own chain operation you need 2 classes. The operation itself that we will call MyVendor\Etl\Operation\OurTestOperation, and a MyVendor\Etl\OperationFactory\OurTestOperationFactory factory to create it. The factory allows us to configure the operation and inject service to our operation.

All operations needs to implement DataChainOperationInterface; they can extend AbstractChainOperation.

All factories needs to extend Oliverde8\Component\PhpEtl\Builder\Factories\AbstractFactory.

The operation is a Model and not a service, you therefore need to add the path to the exclusions so that it's not made a service by symfony:

App\:
  resource: '../src/'
  exclude:
    - '../src/Etl/Operation'

Factories needs to be tagged `etl.operation-factory\ . To remove the need to tag all your factories you can add the following line your your services.yaml file

    MyVendor\Etl\OperationFactory\:
        resource: '../src/Etl/OperationFactory/'
        tags: ['etl.operation-factory']

For more information on how the etl works and how to create operations check the Php Etl Documentation