keboola/db-extractor-common

Common library from Keboola Database Extractors

16.1.0 2023-01-19 08:49 UTC

Requires

• php: >=7.4
• ext-json: *
• ext-pdo: *
• keboola/common-exceptions: ^1.1
• keboola/csv: ^3.2
• keboola/db-extractor-adapter: ^1.9
• keboola/db-extractor-config: ^1.8
• keboola/db-extractor-ssh-tunnel: ^1.1.5
• keboola/db-extractor-table-format: ^3.2
• keboola/php-component: ^9.0
• keboola/php-datatypes: ^6.0
• keboola/php-utils: ^2.3||^3.0||^4.0
• keboola/retry: ^0.5
• monolog/monolog: ^2.0
• nette/utils: ^3.0
• phpunit/phpunit: ^9.1
• pimple/pimple: ^3.3
• psr/log: ^1.1

Requires (Dev)

• ihsw/toxiproxy-php-client: ^2.0
• keboola/coding-standard: >=13.0
• keboola/php-temp: ^2.0
• php-parallel-lint/php-parallel-lint: ^1.2
• phpstan/phpstan: ^0.12.14
• symfony/debug: ^4.3

MIT

• keboola

README

Common classes for creating vendor specific database extractors.

Extractors using DBC

• MySQL
• MSSQL
• PgSQL
• Oracle (private repository)
• Impala
• Firebird
• DB2
• Redshift
• Snowflake

Development and running tests

docker-compose build
docker-compose run --rm tests  # runs the tests

Usage

Add the library to your component's composer:

php composer.phar require db-extractor-common

composer.json

{
  "require": "db-extractor-common": ^8.0
}

Usage

Create entrypoint script file run.php like this one for Mysql extractor: https://github.com/keboola/db-extractor-mysql/blob/master/src/run.php

Note that as of version 7, configuration rows are supported so it is not necessary to support .yml configs or table array configurations.

The $config is loaded from the config.json file. You have to provide values for the data_dir and extractor_class keys. extractor_class is the main class of derived extractor, it should extend Keboola\DbExtractor\Extractor\Extractor.

You will need to implement the following methods:

• createConnection(array $params)
• testConnection()
• simpleQuery(array $table, array $columns = array()): string
• getTables(?array $tables = null): array;

Note that to support identifier sanitation, the getTables method should return a sanitizedName property for each column. This sanitizedName should be created using Keboola\php-utils\sanitizeColumnName

If you want to implement incremental fetching, you must implement
validateIncrementalFetching(array $table, string $columnName, ?int $limit = null): void
Please see the sample in the Common class: https://github.com/keboola/db-extractor-common/blob/master/src/Keboola/DbExtractor/Extractor/Common.php#L52

The namespace of your extractor class shoud be Keboola\DbExtractor\Extractor and the name of the class should corespond to DB vendor name i.e. PgSQL, Oracle, Impala, Firebrid, DB2 and so on.

Please check the existing implementations above for help getting started.

Check username

Check username functionality compares - value of the KBC_REALUSER environment variable - and username of the database user from the configuration.

If enabled, values must be same to run the component.

If a service account is used, check is skipped. For more information see UsernameChecker class.

Example image / stack parameters

{
  "checkUsername": {
    "enabled": true
  }
}

Service account can be defined by regular expression:

{
  "checkUsername": {
    "enabled": true,
    "serviceAccountRegexp": "~^service_~i"
  }
}

Or user account can be defined by regular expression (inverted logic):

{
  "checkUsername": {
    "enabled": true,
    "userAccountRegexp": "~^[0-9]{3}_"
  }
}

License

MIT licensed, see LICENSE file.