umpirsky/extraload

Powerful ETL library.

0.1.0 2016-03-12 17:11 UTC

Requires

• php: >=5.6
• symfony/event-dispatcher: ^3.0

Requires (Dev)

• behat/behat: v3.1.0rc1
• behat/mink-selenium2-driver: ^1.3
• doctrine/dbal: ^2.5
• phpspec/phpspec: ^2.4
• phpunit/phpunit: ^5.1
• symfony/console: ^3.0
• symfony/property-access: ^3.0

Suggests

• doctrine/dbal: Allows loading data into database using DBAL loader
• symfony/console: Allows displaying tabular data in console
• symfony/property-access: Allows applying transformer on property by path

MIT

• Saša Stamenković

• import
• convert
• export
• transform
• extract
• etl
• load

README

symfony upgrade fixer • twig gettext extractor • wisdom • centipede • permissions handler • extraload • gravatar • locurro • country list • transliterator

Extraload

Powerful ETL library.

Examples

Dumping CSV data into the console

Input data is given in csv format:

"99921-58-10-7", "Divine Comedy", "Dante Alighieri"
"9971-5-0210-0", "A Tale of Two Cities", "Charles Dickens"
"960-425-059-0", "The Lord of the Rings", "J. R. R. Tolkien"
"80-902734-1-6", "And Then There Were None", "Agatha Christie"

With:

(new DefaultPipeline(
    new CsvExtractor(
        new \SplFileObject('books.csv')
    ),
    new NoopTransformer(),
    new ConsoleLoader(
        new Table(new ConsoleOutput())
    )
))->process();

It can be dumped as table to console:

+---------------+--------------------------+------------------+
| 99921-58-10-7 | Divine Comedy            | Dante Alighieri  |
| 9971-5-0210-0 | A Tale of Two Cities     | Charles Dickens  |
| 960-425-059-0 | The Lord of the Rings    | J. R. R. Tolkien |
| 80-902734-1-6 | And Then There Were None | Agatha Christie  |
+---------------+--------------------------+------------------+

In this example NoopTransformer is used, but various transformations can be applied. Transformers can also be chained using TransformerChain.

Dumping a Doctrine query into the console

First of all make sure to load the fixtures into a database -- this example works with MySQL:

mysql> source /home/standard/projects/Extraload/fixtures/mysql/books.sql

So the following code:

(new DefaultPipeline(
    new QueryExtractor($conn, 'SELECT * FROM books'),
    new NoopTransformer(),
    new ConsoleLoader(
        new Table($output = new ConsoleOutput())
    )
))->process();

Will dump these results to the console:

+---------------+--------------------------+------------------+
| 99921-58-10-7 | Divine Comedy            | Dante Alighieri  |
| 9781847493583 | La Vita Nuova            | Dante Alighieri  |
| 9971-5-0210-0 | A Tale of Two Cities     | Charles Dickens  |
| 960-425-059-0 | The Lord of the Rings    | J. R. R. Tolkien |
| 80-902734-1-6 | And Then There Were None | Agatha Christie  |
+---------------+--------------------------+------------------+

Dumping a Doctrine prepared query into the console

The following code:

// ...

$sql = "SELECT * FROM books WHERE author = :author";
$values = [
    [
        'parameter' => ':author',
        'value' => 'Dante Alighieri',
        'data_type' => PDO::PARAM_STR // optional
    ]
];

(new DefaultPipeline(
    new QueryExtractor($conn, $sql, $values),
    new NoopTransformer(),
    new ConsoleLoader(
        new Table($output = new ConsoleOutput())
    )
))->process();

Will dump these results to the console:

+---------------+---------------+-----------------+
| 99921-58-10-7 | Divine Comedy | Dante Alighieri |
| 9781847493583 | La Vita Nuova | Dante Alighieri |
+---------------+---------------+-----------------+

Dumping a Doctrine query into a table

The following code:

// ...

(new DefaultPipeline(
    new QueryExtractor($conn, 'SELECT * FROM books'),
    new NoopTransformer(),
    new DbalLoader($conn, 'my_books')
))->process();

Will dump the results into the my_books table:

mysql> select * from my_books;
+----------------+--------------------------+----------------------------+
| isbn           | title                    | author                     |
+----------------+--------------------------+----------------------------+
| 9781503262140  | Faust                    | Johann Wolfgang von Goethe |
| 978-0156949606 | The Waves                | Virgina Woolf              |
| 99921-58-10-7  | Divine Comedy            | Dante Alighieri            |
| 9781847493583  | La Vita Nuova            | Dante Alighieri            |
| 9971-5-0210-0  | A Tale of Two Cities     | Charles Dickens            |
| 960-425-059-0  | The Lord of the Rings    | J. R. R. Tolkien           |
| 80-902734-1-6  | And Then There Were None | Agatha Christie            |
+----------------+--------------------------+----------------------------+
7 rows in set (0.00 sec)

See more examples.

2. Inspiration

Inspired by php-etl and petl.