keboola/storage-api-client

4.18.0 2016-12-05 19:10 UTC

README

Latest Stable Version License Total Downloads Build Status

Simple PHP wrapper library for Keboola Storage REST API

Installation

Library is available as composer package. To start using composer in your project follow these steps:

Install composer

curl -s http://getcomposer.org/installer | php
mv ./composer.phar ~/bin/composer # or /usr/local/bin/composer

Create composer.json file in your project root folder:

{
    "require": {
        "php" : ">=5.4.0",
        "keboola/storage-api-client": "2.12.*"
    }
}

Install package:

composer install

Add autoloader in your bootstrap script:

require 'vendor/autoload.php';

Read more in Composer documentation

Usage examples

Table write:

require 'vendor/autoload.php';

use Keboola\StorageApi\Client,
    Keboola\Csv\CsvFile;

$client = new Client([
  'token' => 'YOUR_TOKEN',
]);
$csvFile = new CsvFile(__DIR__ . '/my.csv', ',', '"');
$client->writeTableAsync('in.c-main.my-table', $csvFile);

Table export to file:

require 'vendor/autoload.php';

use Keboola\StorageApi\Client,
  Keboola\StorageApi\TableExporter;

$client = new Client(['token' => 'YOUR_TOKEN',]);

$exporter = new TableExporter($client);
$exporter->exportTable('in.c-main.my-table', './in.c-main.my-table.csv', []);

Tests

Warning: Never run this tests on production project with real data, always create project for testing purposes!!!

The main purpose of these test is "black box" test driven development of Keboola Connection. These test guards the API implementation.

Tests are divided into multiple test suites:

  • common - tests all storage backend unrelated features like configurations, events, file uploads, tokens management
  • backend-mysql - tests project with mysql storage backend
  • backend-redshift - tests project with redshift storage backend

Common test suite

This test suite expects following environment variables set:

You can export variables manually or you can create and fill file set-env.sh as copy of attached set-env.template.sh.

Than you can run tests:

source ./set-env.sh && php ./vendor/bin/phpunit --testsuite common

Mysql backend test suite

This test suite expects following environment variables set:

  • STORAGE_API_URL - URL of Keboola Storage API (https://connection.keboola.com/)
  • STORAGE_API_TOKEN - Storage API token associated to user (Admin master token) with all permissions. Project must have mysql set as default backend.

You can export variables manually or you can create and fill file set-env.mysql.sh as copy of attached set-env.mysql.template.sh.

Than you can run tests:

source ./set-env.mysql.sh && php ./vendor/bin/phpunit --testsuite backend-mysql

Redshift backend test suite

Reshift tests require a cluster connected to Storage API and credentials.

These credentials are used in tests to simulate "copy from table" transformations behaviour. Then you can create your Redshift user. Connect to your Redshift database sapi_YOURPROJECTID and run queries:

CREATE USER test_user PASSWORD '***';
GRANT ALL PRIVILEGES ON DATABASE sapi_YOURPROJECTID TO test_user;
GRANT ALL PRIVILEGES ON SCHEMA "in.c-api-tests" TO test_user;
GRANT ALL PRIVILEGES ON SCHEMA "out.c-api-tests" TO test_user;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA "in.c-api-tests" TO test_user;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA "out.c-api-tests" TO test_user;

This test suite expects following environment variables set:

  • STORAGE_API_URL - URL of Keboola Storage API (https://connection.keboola.com/)
  • STORAGE_API_TOKEN - Storage API token associated to user (Admin master token) with all permissions. Project must have Redshift set as default backend.
  • REDSHIFT_HOSTNAME - hostname of storage backend Redshift cluster
  • REDSHIFT_USER - previously created user
  • REDSHIFT_PASSWORD - previously created passowrd

You can export variables manually or you can create and fill file set-env.redshift.sh as copy of attached set-env.redshift.template.sh.

Than you can run tests:

source ./set-env.redshift.sh && php ./vendor/bin/phpunit --testsuite backend-redshift

Snowflake backend test suite

This test suite expects following environment variables set:

  • STORAGE_API_URL - URL of Keboola Storage API (https://connection.keboola.com/)
  • STORAGE_API_TOKEN - Storage API token associated to user (Admin master token) with all permissions. Project must have snowflake set as default backend.

Tests are also testing provisioning of workspaces and direct connection to snowflake so it requires snowflake odbc driver. You can download odbc driver from Snowflake Console and save it snowflake_linux_x8664_odbc.tgz to this repository root folder.

You can run these tests in docker with drivers installed:

source ./set-env.snowflake.sh && docker-compose run --rm tests sh -c 'composer install && ./vendor/bin/phpunit --testsuite backend-snowflake-part-1'

Mixed backend test suite

Project can support multiple backends, this is useful for migrations from one backend to another. These tests require project with all backend assigned (mysql, redshift, snowflake).

This test suite expects following environment variables set:

You can export variables manually or you can create and fill file set-env.mixed.sh as copy of attached set-env.mixed.template.sh.

Than you can run tests:

source ./set-env.mixed.sh && php ./vendor/bin/phpunit --testsuite backend-mixed

Versioning

semver.org is followed.

Release History

See the CHANGELOG.