README

BoutDeCode Sylius ETL Plugin

Integrate a full ETL (Extract, Transform, Load) pipeline system into the Sylius admin panel.

Overview

This plugin adds a full-featured ETL workflow management system to the Sylius admin panel. It allows store administrators to define reusable Workflows (named chains of ordered ETL Steps), and execute them as Pipelines — either manually, via file upload, or on a schedule.

The plugin provides the ETL infrastructure (workflow management, pipeline execution, history tracking). Step implementations are defined in your application.

Features

Workflow management — create and configure reusable chains of ETL steps from the admin panel
Pipeline execution — run workflows as pipelines with JSON input, file upload, or scheduled execution
Execution history — detailed per-step and per-pipeline run history with status tracking
Step registry — register your own step services tagged with etl.step
State machine — pipeline lifecycle management with reset/execute transitions
Dedicated logging — separate pipeline log channel for ETL activity
Sylius admin integration — ETL section added to the admin sidebar with grid views for Workflows and Pipelines

Screenshots

Workflow list

Workflow editor — configure ordered steps with their configuration

Pipeline list

Pipeline detail — inspect pipeline and configuration

Execute a pipeline — provide a file upload, JSON input or a scheduled date; the workflow configuration can also be overridden at runtime per step

Pipeline run history — per-step status and execution timeline

Requirements

PHP ^8.1
Sylius ^1.14
Symfony ^6.4
Symfony Messenger with an async transport

Installation

1. Require the plugin via Composer

composer require boutdecode/sylius-etl-plugin

2. Enable the plugin

Add the plugin to your config/bundles.php:

return [
    // ...
    BoutDeCode\SyliusETLPlugin\BoutDeCodeSyliusETLPlugin::class => ['all' => true],
];

3. Import the plugin configuration

PHP — create or update config/packages/bout_de_code_sylius_etl.php:

<?php

declare(strict_types=1);

use Symfony\Component\DependencyInjection\Loader\Configurator\ContainerConfigurator;

return static function (ContainerConfigurator $containerConfigurator): void {
    $containerConfigurator->import('@BoutDeCodeETLCoreBundle/Resources/config/config.yaml');
    $containerConfigurator->import('@BoutDeCodeSyliusETLPlugin/config/config.php');
};

YAML — create or update config/packages/bout_de_code_sylius_etl.yaml:

imports:
    - { resource: '@BoutDeCodeETLCoreBundle/Resources/config/config.yaml' }
    - { resource: '@BoutDeCodeSyliusETLPlugin/config/config.php' }

4. Import the admin routes

PHP — add to config/routes.php (or config/routes/sylius_admin.php):

<?php

declare(strict_types=1);

use Symfony\Component\Routing\Loader\Configurator\RoutingConfigurator;

return static function (RoutingConfigurator $routingConfigurator): void {
    $routingConfigurator->import('@BoutDeCodeSyliusETLPlugin/config/routes/admin.php');
};

YAML — add to config/routes.yaml (or config/routes/sylius_admin.yaml):

bout_de_code_sylius_etl_admin:
    resource: '@BoutDeCodeSyliusETLPlugin/config/routes/admin.php'

5. Configure Symfony Messenger

Ensure an async transport is configured in config/packages/messenger.yaml:

framework:
    messenger:
        transports:
            async: '%env(MESSENGER_TRANSPORT_DSN)%'

6. Import the admin assets

Add a dedicated entry in your webpack.config.js:

const path = require('path');

Encore
    // ...
    .addEntry(
        'boutdecode-etl-plugin-admin-entry',
        path.resolve(
            __dirname,
            'vendor/boutdecode/sylius-etl-plugin/assets/admin/entrypoint.js'
        )
    )

Then include the compiled entry in your admin layout (e.g. templates/bundles/SyliusAdminBundle/layout.html.twig):

{{ encore_entry_script_tags('boutdecode-etl-plugin-admin-entry') }}
{{ encore_entry_link_tags('boutdecode-etl-plugin-admin-entry') }}

Then install and build the assets:

bin/console assets:install
yarn encore dev

7. Run database migrations

bin/console doctrine:migrations:migrate

Usage

Workflows

Navigate to Admin > ETL > Workflows to create a new Workflow. A workflow defines an ordered chain of steps, each with a step type code and a JSON configuration.

Step type codes correspond to services tagged etl.step registered in your application. See the etl-core documentation for instructions on how to create and register custom steps.

Pipelines

Navigate to Admin > ETL > Pipelines to create and execute pipelines from existing workflows.

When creating a pipeline you can provide:

A JSON input payload
A file upload (e.g. a CSV file)
A JSON configuration override to customize step parameters at runtime
A scheduled date/time for deferred execution

Logging

Pipeline execution is logged to a dedicated file:

var/log/{env}_bout_de_code_sylius_etl_plugin.log

Testing

PHPUnit

vendor/bin/phpunit

Behat (non-JS scenarios)

vendor/bin/behat --strict --tags="~@javascript&&~@mink:chromedriver"

Behat (JS scenarios)

Install Symfony CLI.

Start Headless Chrome:

google-chrome-stable --enable-automation --disable-background-networking --no-default-browser-check --no-first-run --disable-popup-blocking --disable-default-apps --allow-insecure-localhost --disable-translate --disable-extensions --no-sandbox --enable-features=Metal --headless --remote-debugging-port=9222 --window-size=2880,1800 --proxy-server='direct://' --proxy-bypass-list='*' http://127.0.0.1

Start the test application server:

symfony server:ca:install
APP_ENV=test symfony server:start --port=8080 --daemon

Run Behat:

vendor/bin/behat --strict --tags="@javascript,@mink:chromedriver"

Static Analysis

vendor/bin/phpstan analyse -c phpstan.neon -l max src/

Coding Standards

vendor/bin/ecs check

License

This plugin is released under the MIT License.

boutdecode / sylius-etl-plugin

Maintainers

Package info

Statistics

Security