keboola/openlineage-generator

Generates jobs data for a service with OpenLineage API

1.1.0 2023-10-30 08:47 UTC

Requires

• php: ^8.1
• guzzlehttp/guzzle: ^7.0
• psr/log: ^1.1

Requires (Dev)

• keboola/coding-standard: ^15.0
• php-parallel-lint/php-parallel-lint: ^1.3
• phpstan/phpstan: ^1.4

MIT

README

This library will generate jobs metadata obtained from Job Queue API open-api-lineage endpoint for a data lineage service supporting OpenLineage API (e.g. Marquez).

Installation

The library is available as composer package. To start using this library in your project follow these steps:

Install package:

composer require keboola/openlineage-generator

Add autoloader in your bootstrap script:

require 'vendor/autoload.php';

Read more in Composer documentation

Usage

$openLineageWriter = new OpenLineageWriter(
            $queueClient, // keboola job queue client
            $openLineageClient, // http client for open lineage api
            $logger,
            $createdTimeFrom, //DateTimeImmutablee object from which jobs will be imported
            $config->getOpenLineageEndpoint(), // url of open lineage api
            $config->getJobNameAsConfig() // boolean, if true, job name will be used as config name
        );

$openLineageWriter->write();

Development

Clone this repository and init the workspace with following command:

git clone https://github.com/keboola/openlineage-generator
cd openlineage-generator
docker-compose build
docker-compose run --rm dev composer install --no-scripts

Run the CI checks using this command:

docker-compose run --rm dev composer ci

Integration

For information about deployment and integration with KBC, please refer to the deployment section of developers documentation

License

MIT licensed, see LICENSE file.