README

A high-performance Laravel package for exporting 300M+ records with professional PDF support (9000+ pages without breaking). Supports CSV, XLSX, PDF, DOCX, and SQL formats.

Overview

TurboStream Export Engine is designed for Laravel applications that need to export massive datasets (300M+ records) efficiently without memory issues. It uses chunked queries to process data in batches and leverages Laravel Queues with Redis for background processing.

Key Features

300M+ Records Support: Optimized chunk sizes for massive datasets (10K-20K per chunk)
9000+ Page PDFs: Professional PDF generation without breaking
Memory Efficient: Uses cursor() instead of chunk() to stream records without loading all into memory
5 Export Formats: CSV, XLSX, PDF, DOCX, SQL
PDF Full Features: Subtotals, grand totals, colspan, rowspan, custom headers
Async Processing: Background jobs via Laravel Queues with Redis
Real-time Progress: Track export progress via Redis cache
Filter Names in Filename: Downloaded files include applied filters in filename
Multiple Queue Drivers: Redis (recommended), Database, or Sync
Auto Chunk Sizing: Automatically adjusts chunk size based on data volume
Laravel Native: Integrates seamlessly with Laravel 9, 10, 11, 12, and 13

Requirements

PHP 8.1+
Laravel 9.0, 10.0, 11.0, 12.0, or 13.0
Redis (recommended) or Database queue driver
ext-json, ext-mbstring

Dependencies (All Included)

Format	Package	Description
CSV	League CSV	Best for 300M+ records
XLSX	PhpSpreadsheet	Reports & Excel files
PDF	TCPDF	9000+ page PDFs with all features
DOCX	PhpWord	Word documents
SQL	Built-in	Database backup/migration

Installation

1. Install via Composer

composer require turbostream/export-engine

2. Publish Configuration

php artisan vendor:publish --tag=turbo-export

3. Configure Environment

Add Redis configuration to your .env file:

CACHE_DRIVER=redis
QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
REDIS_PORT=6379

4. Run Migrations (optional - for database queue)

php artisan migrate

Quick Start

Basic Usage

use TurboStreamExport\Facades\ExportFacade;
use App\Models\User;

$exportId = ExportFacade::createExport([
    'model' => User::class,
    'columns' => ['id', 'name', 'email', 'created_at'],
    'format' => 'csv',
    'filename' => 'users_export',
]);

Using Filters

$exportId = ExportFacade::createExport([
    'model' => \App\Models\User::class,
    'columns' => ['id', 'name', 'email', 'status'],
    'format' => 'csv',
    'filters' => [
        ['status', '=', 'active'],
        ['created_at', '>=', '2026-01-01'],
    ],
]);

Filename with filters: When filters are applied, the downloaded file includes filter details:

users_export_filtered_status=_active_created_at=_>=_2026-01-01.csv

Check Progress

$progress = ExportFacade::getProgress($exportId);

echo $progress['progress'];      // 75
echo $progress['status'];        // 'processing' or 'completed'
echo $progress['filters'];      // Array of applied filters
echo $progress['filter_summary']; // 'status=_active'

Download File

$downloadUrl = ExportFacade::getDownloadUrl($exportId);
// Returns signed URL valid for 1 hour

Export Formats

This package supports 5 export formats for different use cases:

┌─────────────────────────────────────────────────────────────────────────────┐
│                        5 EXPORT FORMATS SUPPORTED                            │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐      │
│  │    CSV   │  │   XLSX   │  │   PDF    │  │   DOCX   │  │   SQL    │      │
│  ├──────────┤  ├──────────┤  ├──────────┤  ├──────────┤  ├──────────┤      │
│  │    📄    │  │    📊    │  │    📑    │  │    📝    │  │  🗄️     │      │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘      │
│       │             │             │             │             │          │
│       ▼             ▼             ▼             ▼             ▼          │
│  Streaming      Memory-       TCPDF          Memory-       Streaming     │
│  (300M+ OK)     based         (Full Features)based         (300M+ OK)     │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

CSV (Recommended for Large Data)

┌────────────────────────────────────────────────────────┐
│  Format:  comma-separated-values.csv                   │
│  Speed:   ⚡⚡⚡⚡⚡ (FASTEST)                            │
│  Memory:  ~1GB (constant - streaming)                  │
│  Best for: 300M+ records, data backup, API export     │
│  Install:  ✅ Included (League CSV)                   │
└────────────────────────────────────────────────────────┘

XLSX (Excel)

┌────────────────────────────────────────────────────────┐
│  Format:  spreadsheet.xlsx                             │
│  Speed:   ⚡⚡⚡ (moderate)                             │
│  Memory:  ~2GB (grows with records)                   │
│  Best for: Reports, sharing, printing                 │
│  Install:  ✅ Included (PhpSpreadsheet)               │
└────────────────────────────────────────────────────────┘

PDF (Professional Documents)

┌────────────────────────────────────────────────────────┐
│  Format:  document.pdf                                 │
│  Speed:   ⚡⚡ (slower)                                 │
│  Memory:  ~2-4GB (grows with records)                 │
│  Best for: Invoices, reports, 9000+ page documents   │
│                                                        │
│  Features (all included):                             │
│  ✅ Subtotals per group   ✅ Grand Total              │
│  ✅ Colspan & Rowspan     ✅ Custom Headers/Footers   │
│  ✅ 9000+ pages           ✅ Professional layouts    │
│  Install:  ✅ TCPDF included (no extra package)        │
└────────────────────────────────────────────────────────┘

DOCX (Word)

┌────────────────────────────────────────────────────────┐
│  Format:  document.docx                                │
│  Speed:   ⚡⚡ (slower)                                 │
│  Memory:  ~2GB (grows with records)                   │
│  Best for: Documentation, letters, contracts          │
│  Install:  ✅ Included (PhpWord)                       │
└────────────────────────────────────────────────────────┘

SQL (Database)

┌────────────────────────────────────────────────────────┐
│  Format:  dump.sql                                     │
│  Speed:   ⚡⚡⚡⚡⚡ (fast)                              │
│  Memory:  ~1GB (constant - streaming)                  │
│  Best for: DB backup, migration, re-import            │
│  Install:  ✅ Built-in (no extra package)             │
└────────────────────────────────────────────────────────┘

Quick Decision Guide

┌────────────────────────────────────────────────────────────────────┐
│                    WHICH FORMAT TO CHOOSE?                         │
├────────────────────────────────────────────────────────────────────┤
│                                                                    │
│  How many records?                                                 │
│                                                                    │
│  ┌─────────────┐                                                  │
│  │  100M+     │ ──────▶  CSV (streaming, memory efficient)       │
│  └─────────────┘                                                  │
│                                                                    │
│  ┌─────────────┐                                                  │
│  │  1M - 100M  │ ──────▶  CSV (recommended)                      │
│  └─────────────┘         or SQL                                    │
│                                                                    │
│  ┌─────────────┐                                                  │
│  │  < 1M       │ ──────▶  Any format (CSV/XLSX/PDF/DOCX/SQL)     │
│  └─────────────┘                                                  │
│                                                                    │
│  ─────────────────────────────────────────────────────────         │
│                                                                    │
│  What is the purpose?                                              │
│                                                                    │
│  • Data backup/migration ──▶ CSV or SQL                           │
│  • Excel report ──────────▶ XLSX                                   │
│  • Print document ────────▶ PDF                                    │
│  • Word document ────────▶ DOCX                                   │
│  • Share with non-tech ──▶ XLSX                                    │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

API Reference

Create Export

ExportFacade::createExport(array $config);

Configuration Options:

Parameter	Type	Required	Description
model	string	Yes	Full model class path
columns	array	Yes	Columns to export
format	string	No	Export format (default: 'csv')
filename	string	No	Custom filename (without extension)
filters	array	No	Query where clauses
chunk_size	integer	No	Records per chunk (auto-calculated)

Example:

$exportId = ExportFacade::createExport([
    'model' => \App\Models\Transaction::class,
    'columns' => ['id', 'amount', 'status', 'created_at'],
    'format' => 'xlsx',
    'filename' => 'transactions_report',
    'filters' => [
        ['status', '=', 'completed'],
        ['amount', '>', 100],
        ['created_at', '>=', '2026-01-01'],
        ['category_id', 'IN', [1, 2, 3]],
    ],
]);

Get Progress

ExportFacade::getProgress(string $exportId): array;

Returns:

{
    "export_id": "550e8400-e29b-41d4-a716-446655440000",
    "progress": 75,
    "total": 100000000,
    "status": "processing",
    "file_path": null,
    "filters": [
        ["status", "=", "completed"],
        ["created_at", ">=", "2026-01-01"]
    ],
    "filter_summary": "status=_completed_created_at=_>=_2026-01-01",
    "updated_at": "2026-03-19T10:30:00Z"
}

When completed:

{
    "export_id": "550e8400-e29b-41d4-a716-446655440000",
    "progress": 100,
    "total": 100000000,
    "status": "completed",
    "file_path": "exports/transactions_report_filtered_status=_completed.csv",
    "filters": [["status", "=", "completed"]],
    "filter_summary": "status=_completed",
    "updated_at": "2026-03-19T12:30:00Z"
}

Get Download URL

ExportFacade::getDownloadUrl(string $exportId, int $minutes = 60): string;

Returns a signed URL valid for the specified duration.

List Exports

ExportFacade::listExports(int $limit = 10): array;

Delete Export

ExportFacade::deleteExport(string $exportId): bool;

PDF Export (300M+ Records, 9000+ Pages)

This package uses TCPDF with advanced features for professional PDF generation:

┌─────────────────────────────────────────────────────────────────────────┐
│                    PDF EXPORT CAPABILITIES                              │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ✅ 300M+ Records Support    │  ✅ Colspan & Rowspan                    │
│  ✅ 9000+ Pages Without Break│  ✅ Auto Subtotals                       │
│  ✅ Grand Total              │  ✅ Custom Headers/Footers              │
│  ✅ Memory Optimized        │  ✅ Professional Layouts               │
│                                                                         │
│  No additional packages needed - TCPDF included by default!           │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

When to Use PDF

Use Case	Records	Pages	Recommended
Invoice generation	<50K	<100	PDF
Financial reports	<100K	<500	PDF
Large data export	300M+	9000+	CSV (recommended)
Database backup	300M+	-	SQL

PDF Features

use TurboStreamExport\Contracts\Drivers\PdfExportDriver;

$driver = new PdfExportDriver();

// Set report information
$driver->setReportInfo('Annual Financial Report 2021-2026', [
    'company' => 'ABC Corporation',
    'start_date' => '2021-01-01',
    'end_date' => '2026-03-31',
]);

// Enable automatic subtotals by department
$driver->setGroupBy('department');

// Define columns with numeric formatting
$columns = ['id', 'employee', 'department', 'year', 'basic', 'allowances', 'gross', 'tax', 'net'];
$driver->setNumericColumns(['basic', 'allowances', 'gross', 'tax', 'net']);

// Write header
$driver->writeHeader($columns);

// Add custom row with colspan for year header
$driver->addCustomRow([
    0 => [
        'TEXT' => '═══ YEAR 2026 ═══',
        'COLSPAN' => 9,
        'STYLE' => 'subtotal',
        'FONT_WEIGHT' => 'B',
        'TEXT_ALIGN' => 'C',
        'BACKGROUND_COLOR' => [52, 152, 219],
    ]
]);

// Write data rows
foreach ($employees as $employee) {
    $driver->writeRow([...]);
}

// Grand total added automatically at end
$driver->finalize($filePath);

PDF Methods Reference

Method	Description
`setReportInfo($name, $filters)`	Set report title and filter info
`setGroupBy($column)`	Enable auto-subtotals when column changes
`setNumericColumns($columns)`	Format numeric columns with commas
`writeHeader($columns)`	Write table header
`writeRow($data)`	Write single data row
`addCustomRow($cells)`	Add custom row with colspan/rowspan
`addColspanRow($data, $colspan, $text, $style)`	Add row with merged cells
`addEmptyRow()`	Add blank row for spacing
`finalize($filePath)`	Save PDF to file

Real-World PDF Examples

1. Employee Salary Report with Subtotals by Department

$driver = new PdfExportDriver();
$driver->setReportInfo('Employee Salary Report', [
    'department' => 'All',
    'month' => 'March 2026',
]);

$driver->setGroupBy('department');
$driver->setNumericColumns(['basic_salary', 'house_rent', 'medical', 'gross', 'net']);

$columns = ['id', 'name', 'department', 'basic_salary', 'house_rent', 'medical', 'gross', 'net'];
$driver->writeHeader($columns);

foreach ($employees as $emp) {
    $driver->writeRow([
        $emp->id,
        $emp->name,
        $emp->department,
        $emp->basic_salary,
        $emp->house_rent,
        $emp->medical,
        $emp->gross_salary,
        $emp->net_salary,
    ]);
}

$driver->finalize(storage_path('app/exports/salary_report.pdf'));
// Output: Department subtotals + Grand total at end

2. Sales Report with Monthly Section Headers (Colspan)

$driver = new PdfExportDriver();
$driver->setReportInfo('Quarterly Sales Report Q1 2026', [
    'quarter' => 'Q1',
    'year' => 2026,
]);

$driver->setGroupBy('month');
$columns = ['product', 'jan', 'feb', 'mar', 'total'];
$driver->setNumericColumns(['jan', 'feb', 'mar', 'total']);
$driver->writeHeader($columns);

foreach ($months as $month) {
    // Add month header with colspan
    $driver->addCustomRow([
        0 => [
            'TEXT' => "━━━ $month ━━━",
            'COLSPAN' => 5,
            'STYLE' => 'header',
            'TEXT_ALIGN' => 'C',
            'BACKGROUND_COLOR' => [41, 128, 185],
        ]
    ]);
    
    foreach ($products as $product) {
        $driver->writeRow([...]);
    }
}

$driver->finalize($filePath);

3. Inventory Report with Rowspan for Categories

$driver = new PdfExportDriver();
$driver->setReportInfo('Inventory Status Report');

$columns = ['category', 'product', 'quantity', 'value'];
$driver->writeHeader($columns);

$currentCategory = '';
foreach ($inventory as $item) {
    if ($item->category !== $currentCategory) {
        // First item of category - use rowspan
        $driver->addCustomRow([
            0 => [
                'TEXT' => $item->category,
                'ROWSPAN' => $categoryCount[$item->category],
                'STYLE' => 'header',
                'BACKGROUND_COLOR' => [149, 165, 166],
            ],
            1 => ['TEXT' => $item->name],
            2 => ['TEXT' => $item->quantity, 'TEXT_ALIGN' => 'R'],
            3 => ['TEXT' => number_format($item->value), 'TEXT_ALIGN' => 'R'],
        ]);
        $currentCategory = $item->category;
    } else {
        $driver->writeRow(['', $item->name, $item->quantity, $item->value]);
    }
}

$driver->finalize($filePath);

Memory Optimization for Large PDFs

For 9000+ page PDFs with 300M+ records:

// In config/turbo-export.php
return [
    'memory_limit_pdf' => '4G',  // Increase for large PDFs
    
    // For very large exports, process in chunks
    'large_data_chunk_size' => 20000,
    'log_progress_interval' => 500000,
];

Cell Configuration Options

Option	Type	Description
TEXT	string	Cell text
COLSPAN	integer	Columns to span
ROWSPAN	integer	Rows to span
STYLE	string	header, body, subtotal, grandtotal
TEXT_ALIGN	string	L, R, C
FONT_WEIGHT	string	B for bold
FONT_SIZE	integer	Font size in points
TEXT_COLOR	array	RGB [R, G, B]
BACKGROUND_COLOR	array	RGB [R, G, B]
BORDER_SIZE	float	Border width
PADDING	integer	Cell padding

// Write data rows foreach ($employees as $employee) { $driver->writeRow([ $employee->id, $employee->name, $employee->department, $employee->basic_salary, $employee->house_rent, $employee->medical, $employee->gross_salary, $employee->deductions, $employee->net_salary, ]); }

// Add custom section header with colspan $driver->addCustomRow([ 0 => [ 'TEXT' => '★ Report Summary - Last 5 Years Financial Data ★', 'COLSPAN' => 9, 'STYLE' => 'subtotal', 'FONT_WEIGHT' => 'B', 'TEXT_ALIGN' => 'C', 'BACKGROUND_COLOR' => [68, 114, 196], ] ]);

// Grand total added automatically at end

// Finalize and save $filePath = storage_path('app/exports/salary_report.pdf'); $driver->finalize($filePath);


### Methods Reference

#### setReportInfo(string $name, array $filters = [])

Set the report title and filters to be displayed.

```php
$driver->setReportInfo('Monthly Sales Report', [
    'start_date' => '2026-01-01',
    'end_date' => '2026-03-31',
    'region' => 'Dhaka'
]);

setGroupBy(string $column)

Enable automatic subtotals when a column value changes.

// Subtotals will be added automatically when department changes
$driver->setGroupBy('department');

// Can also group by year, category, region, etc.
$driver->setGroupBy('year');

setNumericColumns(array $columns)

Define which columns contain numeric values (for formatting).

$driver->setNumericColumns([
    'basic_salary', 
    'house_rent', 
    'gross_salary', 
    'net_salary'
]);

addCustomRow(array $cellData)

Add a custom row with full control over each cell.

// Section header spanning all columns
$driver->addCustomRow([
    0 => [
        'TEXT' => 'Quarterly Summary - Q1 2026',
        'COLSPAN' => 8,
        'STYLE' => 'subtotal',
        'FONT_WEIGHT' => 'B',
        'TEXT_ALIGN' => 'C',
    ]
]);

// Custom row with specific cell values
$driver->addCustomRow([
    0 => ['TEXT' => 'Section A', 'STYLE' => 'header'],
    1 => ['TEXT' => '', 'STYLE' => 'header'],
    2 => ['TEXT' => 'Total Amount', 'STYLE' => 'header', 'TEXT_ALIGN' => 'R'],
    3 => ['TEXT' => '1,234,567', 'STYLE' => 'subtotal', 'TEXT_ALIGN' => 'R'],
    4 => ['TEXT' => '', 'STYLE' => 'header'],
]);

addColspanRow(array $data, int $colspan, string $text, string $style = 'subtotal')

Add a row with a cell spanning multiple columns.

// Create a row where first 3 columns are merged
$driver->addColspanRow(
    ['', '', '', 'Value 1', 'Value 2'],  // Cell data
    3,                                     // Span 3 columns
    'Merged Header Text',                   // Text for merged cell
    'subtotal'                             // Style
);

addEmptyRow()

Add a blank row for visual separation.

$driver->addEmptyRow();

Cell Configuration Options

Each cell in addCustomRow() supports these options:

Option	Type	Description
TEXT	string	Cell text content
COLSPAN	integer	Number of columns to span
ROWSPAN	integer	Number of rows to span
STYLE	string	Style name (header, body, subtotal, grandtotal)
TEXT_ALIGN	string	Alignment: L, R, C
VERTICAL_ALIGN	string	Vertical: T, M, B
FONT_WEIGHT	string	'B' for bold
FONT_SIZE	integer	Font size in points
TEXT_COLOR	array	RGB [R, G, B]
BACKGROUND_COLOR	array	RGB [R, G, B]
BORDER_SIZE	float	Border width
PADDING_TOP	integer	Top padding
PADDING_BOTTOM	integer	Bottom padding

Example: Financial Report with All Features

use TurboStreamExport\Contracts\Drivers\PdfExportDriver;

$driver = new PdfExportDriver();
$driver->setReportInfo('Annual Financial Report 2021-2026', [
    'start_date' => '2021-01-01',
    'end_date' => '2026-03-31',
    'company' => 'ABC Corporation'
]);

// Group by department for subtotals
$driver->setGroupBy('department');

// Define columns with numeric ones
$columns = ['id', 'employee', 'department', 'year', 'basic', 'allowances', 'gross', 'tax', 'net'];
$driver->setNumericColumns(['basic', 'allowances', 'gross', 'tax', 'net']);

// Write header
$driver->writeHeader($columns);

// Data by year and department
$years = [2021, 2022, 2023, 2024, 2025, 2026];
$departments = ['HR', 'IT', 'Finance', 'Operations', 'Marketing'];

foreach ($years as $year) {
    // Year header with colspan
    $driver->addCustomRow([
        0 => [
            'TEXT' => "═══ YEAR $year ═══",
            'COLSPAN' => 9,
            'STYLE' => 'subtotal',
            'FONT_WEIGHT' => 'B',
            'TEXT_ALIGN' => 'C',
            'BACKGROUND_COLOR' => [52, 152, 219],
        ]
    ]);
    
    foreach ($departments as $dept) {
        // Write employee rows (subtotals added automatically when department changes)
        foreach ($employees as $emp) {
            if ($emp->department === $dept && $emp->year === $year) {
                $driver->writeRow([
                    $emp->id,
                    $emp->name,
                    $emp->department,
                    $emp->year,
                    $emp->basic,
                    $emp->allowances,
                    $emp->gross,
                    $emp->tax,
                    $emp->net,
                ]);
            }
        }
    }
}

// Grand total row added automatically at the end
$filePath = storage_path('app/exports/financial_report.pdf');
$driver->finalize($filePath);

Configuration

Publish the configuration file:

php artisan vendor:publish --tag=turbo-export --force

config/turbo-export.php

return [
    'disk' => env('EXPORT_DISK', 'local'),
    
    'chunk_size' => env('EXPORT_CHUNK_SIZE', 5000),
    
    'large_data_chunk_size' => env('EXPORT_LARGE_DATA_CHUNK_SIZE', 10000),
    
    'queue' => env('EXPORT_QUEUE', 'exports'),
    
    'retention_hours' => env('EXPORT_RETENTION_HOURS', 24),
    
    'max_records' => env('EXPORT_MAX_RECORDS', 100000000),
    
    'formats' => [
        'csv',
        'xlsx',
        'pdf',
        'docx',
        'sql',
    ],
    
    'default_format' => env('EXPORT_DEFAULT_FORMAT', 'csv'),
    
    // Default memory limit for CSV/SQL (streaming exports)
    'memory_limit' => env('EXPORT_MEMORY_LIMIT', '1G'),
    
    // Format-specific memory limits for memory-based exports
    'memory_limit_xlsx' => env('EXPORT_MEMORY_LIMIT_XLSX', '2G'),
    'memory_limit_pdf' => env('EXPORT_MEMORY_LIMIT_PDF', '2G'),
    'memory_limit_docx' => env('EXPORT_MEMORY_LIMIT_DOCX', '2G'),
    
    'batch_commit_size' => env('EXPORT_BATCH_COMMIT_SIZE', 50000),
    
    'include_filter_in_filename' => env('EXPORT_INCLUDE_FILTER_IN_FILENAME', true),
    
    'download_expiry_minutes' => env('EXPORT_DOWNLOAD_EXPIRY_MINUTES', 60),
    
    'drivers' => [
        'csv' => \TurboStreamExport\Contracts\Drivers\CsvExportDriver::class,
        'xlsx' => \TurboStreamExport\Contracts\Drivers\XlsxExportDriver::class,
        'pdf' => \TurboStreamExport\Contracts\Drivers\PdfExportDriver::class,
        'docx' => \TurboStreamExport\Contracts\Drivers\DocxExportDriver::class,
        'sql' => \TurboStreamExport\Contracts\Drivers\SqlExportDriver::class,
    ],
    
    'large_data_threshold' => env('EXPORT_LARGE_DATA_THRESHOLD', 1000000),
    
    'enable_progress_logging' => env('EXPORT_PROGRESS_LOGGING', true),
    
    'log_progress_interval' => env('EXPORT_LOG_PROGRESS_INTERVAL', 100000),
];

Memory Management

The package uses cursor() instead of chunk() to stream database records without loading all into memory:

Format	Export Type	Default Memory	Notes
CSV	Streaming	1GB	Best for 100M+ records
SQL	Streaming	1GB	Best for database backup
XLSX	Memory-based	2GB	Uses running totals
PDF	Memory-based	2GB	Slow for 50K+ records
DOCX	Memory-based	2GB	Works well with 47K+ records

Testing Package

For a complete Laravel application demonstrating the export engine in action, see the testing package:

Testing App: https://github.com/moshiur1412/turbo-export-app

This testing package includes:

Full Laravel application setup
Example models and controllers
Export API routes and views
Queue worker configuration
Sample data generators

Testing

# Run all tests
composer test

# Run with coverage
composer test:coverage

# Run unit tests only
composer test:unit

# Run feature tests only
composer test:feature

# Run large data tests (memory/performance)
composer test:large

# Or using Pest directly
./vendor/bin/pest
./vendor/bin/pest --coverage
./vendor/bin/pest tests/Unit
./vendor/bin/pest tests/LargeData

Test Coverage

Suite	Description	Files
Unit	Driver tests, Service tests, Job tests	ExportDriverTest, ExportServiceTest, ProcessExportJobTest
Feature	Integration tests, Filter tests	ExportServiceFeatureTest
LargeData	Performance tests, Memory tests	LargeDataExportTest

Writing Tests

use TurboStreamExport\Contracts\Drivers\CsvExportDriver;
use TurboStreamExport\Services\ExportService;

class ExportServiceTest extends TestCase
{
    public function test_csv_driver_returns_correct_format(): void
    {
        $driver = new CsvExportDriver();
        
        $this->assertEquals('csv', $driver->getFormat());
        $this->assertEquals('text/csv', $driver->getContentType());
        $this->assertEquals('csv', $driver->getFileExtension());
    }
    
    public function test_filter_summary_generation(): void
    {
        $service = new ExportService('local', [new CsvExportDriver()]);
        
        $filters = [
            ['status', '=', 'active'],
            ['category_id', '=', '5'],
        ];
        
        // Access private method via reflection
        $reflection = new \ReflectionClass($service);
        $method = $reflection->getMethod('buildFilterSummary');
        $method->setAccessible(true);
        
        $summary = $method->invoke($service, $filters);
        
        $this->assertStringContainsString('status', $summary);
        $this->assertStringContainsString('active', $summary);
    }
}

Large Data Testing

public function test_csv_driver_handles_large_batch(): void
{
    $driver = new CsvExportDriver();
    $filePath = $this->tempDir . '/large_export.csv';
    $handle = fopen($filePath, 'w');
    
    $columns = ['id', 'name', 'email', 'status', 'created_at'];
    $driver->writeHeader($columns, $handle);
    
    $recordCount = 100000;
    $startTime = microtime(true);
    
    for ($i = 1; $i <= $recordCount; $i++) {
        $driver->writeRow([
            $i,
            "User Name $i",
            "user$i@example.com",
            $i % 2 === 0 ? 'active' : 'inactive',
            '2026-01-15 10:30:00',
        ], $handle);
    }
    
    $driver->finalize($handle, $filePath);
    
    $duration = microtime(true) - $startTime;
    
    $this->assertFileExists($filePath);
    $this->assertLessThan(30, $duration, 'Large batch export took too long');
}

Architecture

Export Process Flow

┌─────────────────────────────────────────────────────────────────────┐
│                        USER REQUEST                                 │
│  ExportFacade::createExport([                                       │
│    'model' => User::class,                                         │
│    'format' => 'csv',                                               │
│    'filters' => [['status', '=', 'active']]                        │
│  ])                                                                  │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         VALIDATION                                   │
│  • Validate model exists                                           │
│  • Check columns exist                                             │
│  • Validate format (csv/xlsx/pdf/docx/sql)                        │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      DISPATCH JOB                                   │
│  ProcessExportJob dispatched to 'exports' queue                   │
│  Job ID = exportId (UUID)                                          │
└────────────────────────────────┬────────────────────────────────────┘
                                 │  async
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    QUEUE WORKER (Redis)                             │
│                                                                     │
│  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐          │
│  │ Worker 1     │   │ Worker 2     │   │ Worker N     │          │
│  │ ProcessExport│   │ ProcessExport│   │ ProcessExport│          │
│  └──────────────┘   └──────────────┘   └──────────────┘          │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      CHUNK PROCESSING                                │
│                                                                     │
│   Auto Chunk Sizing:                                                │
│   ┌─────────────┬─────────────┬─────────────┬─────────────┐      │
│   │   < 1M      │  1M - 10M   │ 10M - 100M  │   100M+     │      │
│   │  5,000 rec  │  10,000 rec │ 15,000 rec  │ 20,000 rec  │      │
│   └─────────────┴─────────────┴─────────────┴─────────────┘      │
│                                                                     │
│   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐            │
│   │   cursor()  │──▶│   Write     │──▶│   Progress  │            │
│   │  (stream)   │   │   Batch     │   │   Update   │            │
│   └─────────────┘   └─────────────┘   └─────────────┘            │
│        │                 │                 │                      │
│        ▼                 ▼                 ▼                      │
│   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐            │
│   │  Memory     │   │   File      │   │    Redis    │            │
│   │  Efficient  │   │   Output    │   │    Cache    │            │
│   └─────────────┘   └─────────────┘   └─────────────┘            │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      FILE GENERATED                                 │
│                                                                     │
│   exports/users_export_filtered_status=_active_2026-01-01.csv     │
│                                                                     │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      DOWNLOAD READY                                  │
│                                                                     │
│   ExportFacade::getDownloadUrl($exportId)                          │
│   → Signed URL (valid 1 hour)                                      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

5 Export Formats - When to Use Which

┌────────────────────────────────────────────────────────────────────────────┐
│                        CHOOSE YOUR FORMAT                                  │
└────────────────────────────────┬───────────────────────────────────────────┘
                                 │
        ┌────────────────────────┼────────────────────────────────┐
        │                        │                                │
        ▼                        ▼                                ▼
┌───────────────┐      ┌─────────────────┐           ┌─────────────────┐
│   CSV         │      │   XLSX          │           │   SQL           │
│   (FASTEST)   │      │   (Excel)       │           │   (Database)    │
├───────────────┤      ├─────────────────┤           ├─────────────────┤
│ Best for:     │      │ Best for:       │           │ Best for:       │
│ • 100M+ rec  │      │ • Reports       │           │ • DB Migration  │
│ • Data backup │      │ • Sharing       │           │ • Backup        │
│ • API export  │      │ • Printing      │           │ • Re-import     │
├───────────────┤      ├─────────────────┤           ├─────────────────┤
│ Memory: 1GB   │      │ Memory: 2GB     │           │ Memory: 1GB     │
│ Stream: YES   │      │ Stream: NO      │           │ Stream: YES     │
└───────────────┘      └─────────────────┘           └─────────────────┘
        │                        │                                │
        └────────────────────────┼────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────┐      ┌─────────────────┐
│   PDF           │      │   DOCX          │
│   (Professional) │      │   (Word)        │
├─────────────────┤      ├─────────────────┤
│ Best for:       │      │ Best for:       │
│ • 9000+ pages   │      │ • Documentation │
│ • Invoices      │      │ • Letters       │
│ • Reports       │      │ • Contracts     │
├─────────────────┤      ├─────────────────┤
│ TCPDF: Full     │      │ Memory: 2GB      │
│ Memory: 2-4GB   │      │ Stream: NO      │
│ Stream: NO      │      └─────────────────┘
└─────────────────┘

Memory Usage by Format

┌─────────────────────────────────────────────────────────────────────────┐
│                        MEMORY EFFICIENCY                                │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  CSV / SQL              ████████████████░░░░░░░░░░░  ~1GB (Streaming)  │
│                                                                         │
│  XLSX                   █████████████████████████░░  ~2GB               │
│                                                                         │
│  PDF (TCPDF)           █████████████████████████░░  ~2-4GB             │
│                                                                         │
│  DOCX                  █████████████████████████░░  ~2GB                │
│                                                                         │
├─────────────────────────────────────────────────────────────────────────┤
│  KEY INSIGHT:                                                           │
│  • CSV/SQL use STREAMING - constant memory (best for 100M+)            │
│  • XLSX/PDF/DOCX load ALL data - memory grows with records             │
│  • For 100M+ records, ALWAYS use CSV!                                   │
└─────────────────────────────────────────────────────────────────────────┘

Chunk Size Strategy

Records	Chunk Size	Memory
< 1M	5,000	~50MB
1M - 10M	10,000	~100MB
10M - 100M	15,000	~200MB
100M+	20,000	~512MB

Directory Structure

turbostream/export-engine/
├── src/
│   ├── Contracts/
│   │   ├── ExportDriverInterface.php
│   │   ├── ExportableInterface.php
│   │   └── Drivers/
│   │       ├── CsvExportDriver.php
│   │       ├── XlsxExportDriver.php
│   │       ├── PdfExportDriver.php       # Unified: simple, subtotals, colspan, 100M+ records
│   │       ├── DocxExportDriver.php
│   │       └── SqlExportDriver.php
│   ├── Facades/
│   │   └── ExportFacade.php
│   ├── Http/Controllers/
│   │   └── ExportController.php
│   ├── Jobs/
│   │   └── ProcessExportJob.php
│   ├── Providers/
│   │   └── TurboStreamExportServiceProvider.php
│   └── Services/
│       └── ExportService.php
├── config/
│   └── turbo-export.php
├── tests/
│   ├── Unit/
│   │   ├── ExportDriverTest.php
│   │   ├── ExportServiceTest.php
│   │   └── ProcessExportJobTest.php
│   ├── Feature/
│   │   └── ExportServiceFeatureTest.php
│   └── LargeData/
│       └── LargeDataExportTest.php
├── routes/
│   └── api.php
├── composer.json
├── phpunit.xml
└── README.md

Queue Workers

Start Queue Worker

php artisan queue:work redis --queue=exports

High Priority Queue

For urgent exports:

php artisan queue:work redis --queue=exports-high,exports

Production Setup (Supervisor)

[program:export-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/artisan queue:work redis --queue=exports --sleep=3 --tries=3
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/log/export-worker.log

Extending the Package

Custom Export Driver

Create your own driver by implementing ExportDriverInterface:

<?php

namespace App\Export\Drivers;

use TurboStreamExport\Contracts\ExportDriverInterface;

class CustomExportDriver implements ExportDriverInterface
{
    public function getFormat(): string
    {
        return 'custom';
    }

    public function getContentType(): string
    {
        return 'application/custom';
    }

    public function getFileExtension(): string
    {
        return 'ext';
    }

    public function writeHeader(array $columns, $handle): void
    {
        // Write header
    }

    public function writeRow(array $data, $handle): void
    {
        // Write row
    }

    public function writeBatch($records, array $columns, $handle): void
    {
        // Write batch
    }

    public function finalize($handle, string $filePath): string
    {
        // Finalize
        return $filePath;
    }
}

Register Custom Driver

// In your ServiceProvider
use TurboStreamExport\Facades\ExportFacade;

ExportFacade::extendDriver('custom', CustomExportDriver::class);

Troubleshooting

"Class Redis not found"

Install the Redis PHP extension or use predis:

composer require predis/predis

Queue Issues

Export stuck in "processing"

Check if queue worker is running:

php artisan queue:work redis --queue=exports

Check Laravel logs for errors:

tail -f storage/logs/laravel.log

Verify Redis connection:

php artisan tinker
Redis::ping();

Clear stuck queue jobs

If jobs are stuck in the queue or failed:

# Clear all pending jobs from exports queue
php artisan queue:clear --queue=exports

# Retry failed jobs
php artisan queue:retry

# Delete all failed jobs
php artisan queue:flush

# View failed jobs
php artisan queue:failed

Queue worker not processing

If the worker is running but jobs aren't being processed:

Check Redis connection in .env:

REDIS_HOST=127.0.0.1
REDIS_PORT=6379
QUEUE_CONNECTION=redis

Restart the worker:

# Stop existing worker (Ctrl+C)
# Start fresh worker
php artisan queue:work redis --queue=exports

For high-volume exports, run worker with verbose output:

php artisan queue:work redis --queue=exports -vvv

Memory issues with large exports

For XLSX/PDF/DOCX with large datasets, increase memory limits:

# Default for CSV/SQL (streaming)
EXPORT_MEMORY_LIMIT=1G

# Higher for XLSX/PDF/DOCX (memory-based)
EXPORT_MEMORY_LIMIT_XLSX=2G
EXPORT_MEMORY_LIMIT_PDF=2G
EXPORT_MEMORY_LIMIT_DOCX=2G

The XLSX driver now uses running totals instead of storing all records in memory, reducing memory usage significantly.

Performance issues

For 100M+ records:

EXPORT_LARGE_DATA_CHUNK_SIZE=20000
EXPORT_LOG_PROGRESS_INTERVAL=500000

Security

If you discover security vulnerabilities, please email moshiur.bro@gmail.com instead of using the public issue tracker.

License

The TurboStream Export Engine is open-sourced software licensed under the MIT license.

Changelog

See CHANGELOG.md for version history.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

turbostream / export-engine

Maintainers

Package info

Statistics

Security

README

Overview

Key Features

Requirements

Dependencies (All Included)

Installation

1. Install via Composer

2. Publish Configuration

3. Configure Environment

4. Run Migrations (optional - for database queue)

Quick Start

Basic Usage

Using Filters

Check Progress

Download File

Export Formats

CSV (Recommended for Large Data)

XLSX (Excel)

PDF (Professional Documents)

DOCX (Word)

SQL (Database)

Quick Decision Guide

API Reference

Create Export

Get Progress

Get Download URL

List Exports

Delete Export

PDF Export (300M+ Records, 9000+ Pages)

When to Use PDF

PDF Features

PDF Methods Reference

Real-World PDF Examples

1. Employee Salary Report with Subtotals by Department

2. Sales Report with Monthly Section Headers (Colspan)

3. Inventory Report with Rowspan for Categories

Memory Optimization for Large PDFs

Cell Configuration Options

setGroupBy(string $column)

setNumericColumns(array $columns)

addCustomRow(array $cellData)

addColspanRow(array $data, int $colspan, string $text, string $style = 'subtotal')

addEmptyRow()

Cell Configuration Options

Example: Financial Report with All Features

Configuration

config/turbo-export.php

Memory Management

Testing Package

Testing

Test Coverage

Writing Tests

Large Data Testing

Architecture

Export Process Flow

5 Export Formats - When to Use Which

Memory Usage by Format

Chunk Size Strategy

Directory Structure

Queue Workers

Start Queue Worker

High Priority Queue

Production Setup (Supervisor)

Extending the Package

Custom Export Driver

Register Custom Driver

Troubleshooting

"Class Redis not found"

Queue Issues

Export stuck in "processing"

Clear stuck queue jobs

Queue worker not processing

Memory issues with large exports

Performance issues

Security