keboola/db-extractor-adapter

Set of connection adapters for DB extractors.

1.17.0 2026-08-25 15:09 UTC

Requires

• php: >=8.2
• ext-iconv: *
• keboola/common-exceptions: ^1.0
• keboola/csv: ^3.2
• keboola/db-extractor-config: ^1.17
• keboola/db-extractor-table-format: ^3.7
• keboola/retry: ^0.5
• keboola/ssh-tunnel: ^2.2
• psr/log: ^1.1

Requires (Dev)

• ext-json: *
• ihsw/toxiproxy-php-client: ^2.0
• keboola/coding-standard: >=9.0
• keboola/php-temp: ^2.0
• phpstan/phpstan: ^1.10
• phpunit/phpunit: ^9.1

Suggests

• ext-odbc: Needed to support ODBC connection
• ext-pdo: Needed to support PDO connection

MIT

• Keboola

Published on Packagist.org on 2026-08-25 15:09 UTC

README

This library contains a common interface for connecting to and data extracting from, various sources:

• It is intended for use with db-extractor-common.
• It supports PDO and ODBC connections for now.
• The interfaces defined in this library can be easily used to support other methods, e.g. cli BCP tool.

Main Classes

• Interface DbConnection is an abstraction that represents a connection to the database.
  • Abstract class BaseDbConnection contains common code and retry mechanisms.
  • Class PdoDbConnection implements connection using PDO extension.
  • Class OdbcDbConnection implements connection using ODBC extension.
• Interface QueryResult is an abstraction that represents query result - rows returned from database.
  • Class PdoQueryResult represents result from PDO connection.
  • Class OdbcQueryResult represents result from ODBC connection.
• Interface ExportAdapter is an abstraction which defines how the data is to be extracted.
  • Based on ExportConfig, ExportResult is generated. The rows are written to the specified CSV file.
  • By implementing this interface, it is possible to add support for CLI tools for export.
  • Abstract class BaseExportAdapter contains common code.
  • Class PdoExportAdapter implements export for PDO connection.
  • Class OdbcExportAdapter implements export for ODBC connection.
  • Class FallbackExportAdapter allows you to use multiple adapters. If one fails, then fallback adapter is used.
• Interface QueryFactory used to generate SQL query from ExportConfig. It is used if query is not set in the config.
  • Class DefaultQueryFactory is base implementation for MySQL/MariaDb compatible SQL dialects.
• Class QueryResultCsvWriter used to write rows from the QueryResult to the specified CSV file.

Incremental Fetching

When incrementalFetchingColumn is set, DefaultQueryFactory builds the WHERE clause from the incremental-fetching config on ExportConfig. The dialect-specific factories in the individual extractors either inherit this logic or mirror it. The lower/upper bounds are resolved by WindowBoundResolver (from db-extractor-config) and quoted for the target column type (INTEGER, NUMERIC, FLOAT or TIMESTAMP).

incrementalFetchingMode selects the strategy. It is optional and defaults to watermark, so configs without it produce exactly the same query as before this feature was added.

• watermark (default) — WHERE column >= <last fetched value> (the value stored in state). On the first run there is no watermark yet, so no WHERE is emitted (full fetch).

  • incrementalFetchingLookback (optional) lowers that bound by a fixed margin — column >= (watermark − N) — so a row committed slightly after its own timestamp is re-scanned on a later run. It is a duration for TIMESTAMP (e.g. "20 minutes") or a number for numeric columns. Being watermark-anchored, it has no dependency on the current time.

• window — column >= start [AND column <= end], ignoring the watermark, for a bounded or segmented backfill. incrementalFetchingStart / incrementalFetchingEnd accept a relative ("20 minutes ago", "now") or absolute ("2021-01-01", "1000") value. At least one bound is required — window mode with neither incrementalFetchingStart nor incrementalFetchingEnd throws a UserException rather than silently degrading to an unfiltered full-table scan.

The modes are mutually exclusive; keys belonging to the other mode are ignored. incrementalFetchingLimit cannot be combined with a window or a lookback — a window would keep returning the first page of a fixed range and never advance, and a lookback would move the watermark backwards; both throw a UserException. It remains valid with plain watermark mode (chunked forward fetching). Cross-cutting validation (e.g. requiring a primary key when a lookback/window re-fetches rows) lives in db-extractor-common.

Development

Clone this repository and init the workspace with following command:

git clone https://github.com/keboola/db-extractor-adapter
cd db-extractor-adapter
docker compose build
docker compose run --rm dev composer install --no-scripts

Run the test suite using this command:

docker compose run --rm dev composer tests

License

MIT licensed, see LICENSE file.