umpirsky/extraload

Powerful ETL library.

0.1.0 2016-03-12 17:11 UTC

README

symfony upgrade fixertwig gettext extractorwisdomcentipedepermissions handlerextraloadgravatarlocurrocountry listtransliterator

Extraload Build Status Scrutinizer Code Quality

Powerful ETL library.

Examples

Dumping CSV data into the console

Input data is given in csv format:

"99921-58-10-7", "Divine Comedy", "Dante Alighieri"
"9971-5-0210-0", "A Tale of Two Cities", "Charles Dickens"
"960-425-059-0", "The Lord of the Rings", "J. R. R. Tolkien"
"80-902734-1-6", "And Then There Were None", "Agatha Christie"

With:

(new DefaultPipeline(
    new CsvExtractor(
        new \SplFileObject('books.csv')
    ),
    new NoopTransformer(),
    new ConsoleLoader(
        new Table(new ConsoleOutput())
    )
))->process();

It can be dumped as table to console:

+---------------+--------------------------+------------------+
| 99921-58-10-7 | Divine Comedy            | Dante Alighieri  |
| 9971-5-0210-0 | A Tale of Two Cities     | Charles Dickens  |
| 960-425-059-0 | The Lord of the Rings    | J. R. R. Tolkien |
| 80-902734-1-6 | And Then There Were None | Agatha Christie  |
+---------------+--------------------------+------------------+

In this example NoopTransformer is used, but various transformations can be applied. Transformers can also be chained using TransformerChain.

Dumping a Doctrine query into the console

First of all make sure to load the fixtures into a database -- this example works with MySQL:

mysql> source /home/standard/projects/Extraload/fixtures/mysql/books.sql

So the following code:

(new DefaultPipeline(
    new QueryExtractor($conn, 'SELECT * FROM books'),
    new NoopTransformer(),
    new ConsoleLoader(
        new Table($output = new ConsoleOutput())
    )
))->process();

Will dump these results to the console:

+---------------+--------------------------+------------------+
| 99921-58-10-7 | Divine Comedy            | Dante Alighieri  |
| 9781847493583 | La Vita Nuova            | Dante Alighieri  |
| 9971-5-0210-0 | A Tale of Two Cities     | Charles Dickens  |
| 960-425-059-0 | The Lord of the Rings    | J. R. R. Tolkien |
| 80-902734-1-6 | And Then There Were None | Agatha Christie  |
+---------------+--------------------------+------------------+

Dumping a Doctrine prepared query into the console

The following code:

// ...

$sql = "SELECT * FROM books WHERE author = :author";
$values = [
    [
        'parameter' => ':author',
        'value' => 'Dante Alighieri',
        'data_type' => PDO::PARAM_STR // optional
    ]
];

(new DefaultPipeline(
    new QueryExtractor($conn, $sql, $values),
    new NoopTransformer(),
    new ConsoleLoader(
        new Table($output = new ConsoleOutput())
    )
))->process();

Will dump these results to the console:

+---------------+---------------+-----------------+
| 99921-58-10-7 | Divine Comedy | Dante Alighieri |
| 9781847493583 | La Vita Nuova | Dante Alighieri |
+---------------+---------------+-----------------+

Dumping a Doctrine query into a table

The following code:

// ...

(new DefaultPipeline(
    new QueryExtractor($conn, 'SELECT * FROM books'),
    new NoopTransformer(),
    new DbalLoader($conn, 'my_books')
))->process();

Will dump the results into the my_books table:

mysql> select * from my_books;
+----------------+--------------------------+----------------------------+
| isbn           | title                    | author                     |
+----------------+--------------------------+----------------------------+
| 9781503262140  | Faust                    | Johann Wolfgang von Goethe |
| 978-0156949606 | The Waves                | Virgina Woolf              |
| 99921-58-10-7  | Divine Comedy            | Dante Alighieri            |
| 9781847493583  | La Vita Nuova            | Dante Alighieri            |
| 9971-5-0210-0  | A Tale of Two Cities     | Charles Dickens            |
| 960-425-059-0  | The Lord of the Rings    | J. R. R. Tolkien           |
| 80-902734-1-6  | And Then There Were None | Agatha Christie            |
+----------------+--------------------------+----------------------------+
7 rows in set (0.00 sec)

See more examples.

2. Inspiration

Inspired by php-etl and petl.