camillebaronnet / php-etl
v0.5.0
2020-04-26 20:25 UTC
Requires
- php: ^7.1
- ext-curl: *
- symfony/serializer: ^4.3 || ^5.0
Requires (Dev)
- phpunit/phpunit: ^7.5
- symfony/var-dumper: ^5.0
This package is auto-updated.
Last update: 2024-10-27 06:34:04 UTC
README
Create simple imports with the Extract, Transform, Load pattern.
Installation
composer require camillebaronnet/php-etl
Usage
This example extract some Github's repositories, apply some transformations
<?php use Camillebaronnet\ETL\Etl; use Camillebaronnet\ETL\Extractor\Http; use Camillebaronnet\ETL\Loader\DebugLoader; use Camillebaronnet\ETL\Transformer\DateTime; use Camillebaronnet\ETL\Transformer\Decode; use Camillebaronnet\ETL\Transformer\Flatten; use Camillebaronnet\ETL\Transformer\Map; use Camillebaronnet\ETL\Transformer\Sleep; $etl = (new Etl) ->extract(Http::class, ['url' => 'https://api.github.com/users/camillebaronnet/repos']) ->add(Decode::class) ->add(Sleep::class, ['seconds' => .2]) ->add(Flatten::class, ['glue' => '_']) ->add(Map::class, [ 'fields' => [ 'id', 'name', 'full_name' => 'fullName', 'owner_login' => 'ownerLogin', 'owner_url' => 'ownerUrl', 'url', 'ssh_url' => 'sshUrl', 'created_at' => 'createdAt' ] ]) ->add(DateTime::class, [ 'fields' => ['createdAt'], 'from' => 'Y-m-d\TH:i:s\Z', 'to' => 'd/m/Y', ]) ; $etl->process(DebugLoader::class);
The process explained
-
EXTRACT : Extract can output one or more items
-
TRANFORM : A transform step takes the result of the previous step (extractor or transformer) apply an operation and optionally split the input into several subsets of items (example with Decode).
-
LOADER : A loader can by placed at the end of the pipeline or between transformers. Several Loader can be setting up.
Collection
Extractors
Transformers
Loaders
Extendable
You can easily create your own custom Extractors, Transformers, Loader or Strategy by implementing the corresponding interface.
Submit yours. Send a pull-request