README

Soong\Console provides basic Symfony Console integration for the Soong framework.

Install

Via Composer

$ composer require soong/console

Usage

The bin/soong script is use to run console commands. Three commands are provied:

migrate

The migrate command takes the name of a migration task defined in the config directory and performs the migration process (extracting data from the configured source and loading it into the configured destination).

$ bin/soong migrate arraytosql
Executing arraytosql

rollback

The rollback command takes the name of a migration task defined in the config directory and rolls back the migration process (removing any data from the destination which was previously loaded by the migrate command).

$ bin/soong rollback arraytosql
Executing arraytosql

status

The status command shows the migration status of any specified migration tasks (or all configured migration tasks, when no task name is provied).

$ bin/soong status arraytosql
+------------+-------+-----------+-------------+
| Task       | Total | Processed | Unprocessed |
+------------+-------+-----------+-------------+
| arraytosql | 4     | 0         | 4           |
+------------+-------+-----------+-------------+

Demos

To setup for demoing Soong ETL:

Create an empty database for testing.
Import data/extractsource.sql to the database (table to be populated by the first demo).
Import data/beer.sql to the database (tables to be populated for the second demo).
Edit each of the files in config/ - where indicated, replace the sample credentials with those for the test database.

Demo 1:

Execute bin/soong migrate arraytosql
Look at the extractsource table to see the data populated, and that the ids have been assigned consecutively.
Look at the map_arraytosql table to see the mapping from source to destination keys.
Execute bin/soong migrate sqltocsv
Observe CSV data output to the terminal with configured transformations applied.
Execute bin/soong rollback arraytosql
Observe that the extractsource and map_arraytosql tables are now empty.

Demo 2:

Execute bin/soong migrate beertopics
Observe the beer_terms table is populated from CSV data - in particular, see how the 'red ale' reference to its 'ale' parent has been converted to the numeric ID assigned to the 'ale' row in the database.
Execute bin/soong migrate beeraccounts
Observe the beer_users table - in particular, see how the ValueLookup transformer converted the boolean values in the pro column to strings in the taster column.
Execute bin/soong migrate beercontent
Observe the beer table - in particular, see how the relationships to users/accounts was maintained even though the IDs for the users changed (also see the map_beeraccounts table).
Execute bin/soong rollback beercontent
Observe how the beer and map_beercontent tables are now empty.

Change log

Please see CHANGELOG for more information on what has changed recently.

Testing

$ composer test

Contributing

Please see CONTRIBUTING and CODE_OF_CONDUCT for details.

Todo

Fix composer.json/autoloading! We should not need to require all the other libraries here...

Security

If you discover any security related issues, please email soong@virtuoso-performance.com instead of using the issue tracker.

Credits

License

The MIT License (MIT). Please see License File for more information.

soong / console

Maintainers

Details