laravelplus/etl-manifesto

declarative ETL tool

v1.0.0 2025-04-16 20:57 UTC

Requires

• php: ^8.3
• illuminate/database: ^12.0
• illuminate/support: ^12.0
• symfony/yaml: ^6.4|^7.0

Requires (Dev)

• laravel/pint: ^1.13
• orchestra/testbench: ^10.0
• phpunit/phpunit: ^11.0
• vlucas/phpdotenv: ^5.5

MIT

• Nejc Cotic

• laravelPlus
• etl-manifesto

README

Features

• 🚀 YAML-based manifest configuration
• 🔄 Flexible data transformation pipeline
• 📊 Support for complex SQL aggregations
• 🔌 Multiple export formats (CSV, JSON)
• 🔗 Automatic relationship handling
• 🎯 Group by and custom functions
• 🛡️ Error handling and validation
• 📝 Detailed logging

Installation

You can install the package via composer:

composer require laravelplus/etl-manifesto

Basic Usage

1. Create a manifest file (e.g., manifests/etl.yml):

etl:
  - id: monthly_user_summary
    name: Monthly User Purchase Summary
    description: Generate monthly user purchase statistics

    source:
      entities:
        - users
        - orders
        - payments

      relationships:
        - users hasMany orders
        - orders hasOne payments

      conditions:
        - orders.created_at: last_month

      mapping:
        - id: users.id
        - name: users.name
        - email: users.email
        - total_orders:
            function: count
            column: orders.id
        - total_spent:
            function: sum
            column: payments.amount

      group_by:
        - users.id
        - users.name
        - users.email

    output:
      format: csv
      path: exports/monthly_user_summary.csv

1. Process the ETL manifest in your code:

use Laravelplus\EtlManifesto\EtlManifesto;

$etl = new EtlManifesto();
$results = $etl->loadManifest('manifests/etl.yml')->process();

Manifest Configuration

Source Configuration

Define your data sources and their relationships:

source:
  entities:
    - table_name
    - another_table
  
  relationships:
    - table_name hasMany related_table
    - table_name belongsTo parent_table

Mapping Functions

Available mapping functions:

• count: Count records
• sum: Sum values
• avg: Calculate average
• min: Find minimum value
• max: Find maximum value
• concat: Concatenate strings
• custom: Define custom transformations

Transformations

Apply transformations to your data:

transform:
  - field_name: lower
  - another_field: upper
  - date_field: format_date

Export Options

Supported export formats:

• CSV with custom delimiters and encoding
• JSON with formatting options
• Custom export handlers

Advanced Usage

Custom Transformers

Create custom transformers by extending the DataTransformer class:

use Laravelplus\EtlManifesto\Services\DataTransformer;

class CustomTransformer extends DataTransformer
{
    public function transform($value, $options)
    {
        // Your custom transformation logic
    }
}

Error Handling

The package provides detailed error handling:

try {
    $results = $etl->loadManifest('manifests/etl.yml')->process();
} catch (\Exception $e) {
    // Handle errors
}

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

The MIT License (MIT). Please see License File for more information.

Credits

• [Your Name]
• [All Contributors]

Security

If you discover any security related issues, please email [e-mail] instead of using the issue tracker.

Laravel Package Boilerplate

This package was generated using the Laravel Package Boilerplate.