bentools/cpimport-bundle

Symfony Bundle - Use file watchers with MariaDb Columnstore CpImport utility

dev-master / 1.0.x-dev 2018-04-12 11:49 UTC

Requires

• php: ^7.1.3
• symfony/config: ~3.0||~4.0
• symfony/console: ~3.0||~4.0
• symfony/dependency-injection: ~3.0||~4.0
• symfony/http-kernel: ~3.0||~4.0
• symfony/lock: ~3.0||~4.0
• symfony/process: ~3.0||~4.0
• symfony/yaml: ~3.0||~4.0

Requires (Dev)

• symfony/dotenv: ~4.0
• symfony/framework-bundle: ~3.0||~4.0
• symfony/var-dumper: ~4.0

MIT

• symfony
• php
• file watcher
• mariadb columnstore
• cpimport

README

MariaDb ColumnStore is a very powerful, column-oriented MySQL storage, powerful for analytics and Big Data.

It has the drawback of extremely slow write operations, but comes with cpimport, a binary that can load CSV files with millions of rows into the table of your choice, in a few seconds.

This Symfony bundle enables a file watcher that will look into a directory, pick up new CSV files when they come, run cpimport and delete them.

Installation

composer require bentools/cpimport-bundle:1.0.x-dev

Configuration

Add the bundle to your kernel. Then, map your directories to database/tables like this:

# config/packages/cpimport.yaml (or app/config.yml in Symfony 3)

cpimport:
    cpimport_bin: /usr/local/mariadb/columnstore/bin/cpimport
    watch:
        /path/to/csv_sales:
            database: my_shop
            table: sales_day
            options:
                delimiter: ','
                enclosure: '"'
                timeout: 60

Usage

To automatically process new files in configured directories:

php bin/console cpimport:watch

To manually process a CSV file through cpimport:

php bin/console cpimport:run my_shop sales_day path/to/file.csv --delimiter=','