README

Php etl is a ETL that will allow transformation of data in a simple and efficient way.

It comes in 2 php components :

Rule engine

The rule engine allows to have configuration based transformations to transform a particular data. This is integrated inside the ETL with the RuleTransformOperation.

The rule engine can be used in standalone, see docs

The ETL Chain

An ETL chain is described by chain operations and Items. The chain operation holds the logic, this means it can:

Extract data, (possibly duplicate the item)
Transform data
Load the data somewhere.

Data will propagate throught the ETL operations using Items, there can be different type of items (We will detail this later.)

Chain operations consumes one item in order to create a new item or an iterator. The purpose is to always process data individually. For example if we are importing customers we try to never have the data of more than one customer in memory.

Examples of how it works

We will have more detailed real use cases with sample data a bit further in the document.

In the simplest case the chains receive an iterator containing 2 items in input, both items are processed by each chain operation. This could be for example a list of customer. Each operation changes the items.

In the following example the iterator sends a single item. The first operation will then send GroupedItems containing 2 items. The first item could be a customer, and then we fetch each order of the customer in the operation1.

We can also group items, to make aggregations. The chain receives an iterator containg 2 items, the first operation processes both items. It breaks the chain for the first item, and returns an aggregation of item1 & item 2. This can be used to count the number of customers. This kind of grouping can use more memory and should therefore be used with care.