README

Framework-agnostic data import engine with transformation, validation, and transaction management.

The Nexus\Import package provides a high-integrity, modular import system for processing CSV, JSON, XML, and Excel files with configurable field mappings, data transformations, validation rules, duplicate detection, and flexible transaction strategies.

Features

🔄 Multiple Import Formats: CSV, JSON, XML, Excel (via Atomy integration)
🎯 Field Mapping: Map source fields to target fields with transformations
✨ Data Transformations: 13 built-in transformation rules (trim, upper, lower, date formatting, type casting, etc.)
✅ Validation Engine: Required fields, email, numeric, min/max, length, date validation
🔍 Duplicate Detection: Internal (within import file) and external (against existing data)
💾 Transaction Strategies: TRANSACTIONAL (single transaction), BATCH (transaction per batch), STREAM (no transaction)
🎭 Import Modes: CREATE, UPDATE, UPSERT, DELETE, SYNC
📊 Comprehensive Error Reporting: Row-level errors with severity (WARNING, ERROR, CRITICAL)
🚀 Memory Efficient: Streaming support for large datasets

Installation

composer require azaharizaman/nexus-import

Quick Start

use Nexus\Import\Services\ImportManager;
use Nexus\Import\ValueObjects\{ImportFormat, ImportMode, ImportStrategy, FieldMapping};
use Nexus\Import\Parsers\CsvParser;

// 1. Create import manager
$importManager = new ImportManager(
    processor: $processor,  // ImportProcessorInterface
    authorizer: null,
    context: null,
    logger: $logger  // PSR-3 LoggerInterface
);

// 2. Register parsers
$importManager->registerParser(ImportFormat::CSV, new CsvParser());

// 3. Define field mappings
$mappings = [
    new FieldMapping(
        sourceField: 'customer_name',
        targetField: 'name',
        required: true,
        transformations: ['trim', 'capitalize']
    ),
    new FieldMapping(
        sourceField: 'email_address',
        targetField: 'email',
        required: true,
        transformations: ['trim', 'lower']
    ),
    new FieldMapping(
        sourceField: 'signup_date',
        targetField: 'created_at',
        transformations: [
            'parse_date:Y-m-d',
            'date_format:Y-m-d H:i:s'
        ]
    )
];

// 4. Define validation rules
$validationRules = [
    new ValidationRule('email', 'email', 'Invalid email format'),
    new ValidationRule('name', 'required', 'Name is required'),
    new ValidationRule('name', 'max_length', 'Name too long', 255)
];

// 5. Import data
$result = $importManager->import(
    filePath: '/path/to/customers.csv',
    format: ImportFormat::CSV,
    handler: $customerHandler,  // ImportHandlerInterface
    mappings: $mappings,
    mode: ImportMode::CREATE,
    strategy: ImportStrategy::BATCH,
    transactionManager: $transactionManager,  // TransactionManagerInterface
    validationRules: $validationRules
);

// 6. Check results
echo "Success: {$result->successCount}\n";
echo "Failed: {$result->failedCount}\n";
echo "Skipped: {$result->skippedCount}\n";
echo "Error Rate: {$result->getSuccessRate()}%\n";

// 7. Display errors
foreach ($result->getErrorsByField() as $field => $errors) {
    echo "Field '{$field}': " . count($errors) . " errors\n";
}

Core Components

1. Value Objects

ImportFormat

enum ImportFormat: string
{
    case CSV = 'csv';
    case JSON = 'json';
    case XML = 'xml';
    case EXCEL = 'excel';
    
    public function requiresExternalParser(): bool; // true for EXCEL
}

ImportMode

enum ImportMode: string
{
    case CREATE = 'create';      // Insert new records only
    case UPDATE = 'update';      // Update existing records only
    case UPSERT = 'upsert';      // Insert or update
    case DELETE = 'delete';      // Delete existing records
    case SYNC = 'sync';          // Full synchronization
    
    public function canCreate(): bool;
    public function canUpdate(): bool;
    public function canDelete(): bool;
}

ImportStrategy

enum ImportStrategy: string
{
    case TRANSACTIONAL = 'transactional';  // Single transaction, rollback on error
    case BATCH = 'batch';                  // Transaction per batch, continue on failure
    case STREAM = 'stream';                // No transaction wrapper, row-by-row
    
    public function isMemoryEfficient(): bool;
}

FieldMapping

new FieldMapping(
    sourceField: 'source_column',
    targetField: 'target_field',
    required: true,
    defaultValue: 'default',
    transformations: ['trim', 'upper', 'slug']  // Applied in order
);

2. Built-in Transformation Rules

Category	Rule	Description	Example
String	`trim`	Remove whitespace	`" Hello "` → `"Hello"`
	`upper`	Uppercase	`"hello"` → `"HELLO"`
	`lower`	Lowercase	`"HELLO"` → `"hello"`
	`capitalize`	Capitalize words	`"hello world"` → `"Hello World"`
	`slug`	URL-safe slug	`"Hello World"` → `"hello-world"`
Type	`to_bool`	Convert to boolean	`"yes"` → `true`
	`to_int`	Convert to integer	`"42"` → `42`
	`to_float`	Convert to float	`"3.14"` → `3.14`
	`to_string`	Convert to string	`42` → `"42"`
Date	`parse_date:format`	Parse date	`"2024-01-15"` with format `Y-m-d`
	`date_format:format`	Format date	Format to `Y-m-d H:i:s`
Utility	`default:value`	Fallback value	`null` → `"N/A"`
	`coalesce:val1,val2`	First non-null	`null, "default"` → `"default"`

3. Validation Rules

new ValidationRule(field: 'email', type: 'email', message: 'Invalid email');
new ValidationRule(field: 'age', type: 'numeric', message: 'Must be numeric');
new ValidationRule(field: 'age', type: 'min', message: 'Min 18', constraint: 18);
new ValidationRule(field: 'age', type: 'max', message: 'Max 100', constraint: 100);
new ValidationRule(field: 'name', type: 'required', message: 'Name required');
new ValidationRule(field: 'name', type: 'min_length', message: 'Too short', constraint: 3);
new ValidationRule(field: 'name', type: 'max_length', message: 'Too long', constraint: 255);
new ValidationRule(field: 'birthdate', type: 'date', message: 'Invalid date');
new ValidationRule(field: 'active', type: 'boolean', message: 'Must be true/false');

4. Import Handler

Implement ImportHandlerInterface to define domain-specific persistence logic:

use Nexus\Import\Contracts\ImportHandlerInterface;
use Nexus\Import\ValueObjects\ImportMode;

final class CustomerImportHandler implements ImportHandlerInterface
{
    public function handle(array $data, ImportMode $mode): void
    {
        match($mode) {
            ImportMode::CREATE => $this->repository->create($data),
            ImportMode::UPDATE => $this->repository->update($data),
            ImportMode::UPSERT => $this->repository->upsert($data),
            ImportMode::DELETE => $this->repository->delete($data),
            ImportMode::SYNC => $this->repository->sync($data)
        };
    }

    public function getUniqueKeyFields(): array
    {
        return ['email'];  // Duplicate detection on email
    }

    public function getRequiredFields(): array
    {
        return ['name', 'email'];
    }

    public function supportsMode(ImportMode $mode): bool
    {
        return $mode !== ImportMode::DELETE;  // Don't allow deletions
    }

    public function exists(array $uniqueData): bool
    {
        return $this->repository->existsByEmail($uniqueData['email']);
    }

    public function validateData(array $data): array
    {
        // Custom domain validation
        $errors = [];
        
        if (isset($data['age']) && $data['age'] < 18) {
            $errors[] = 'Customer must be 18 or older';
        }
        
        return $errors;
    }
}

5. Transaction Manager

Implement TransactionManagerInterface for database transaction management:

use Nexus\Import\Contracts\TransactionManagerInterface;

final class LaravelTransactionManager implements TransactionManagerInterface
{
    public function begin(): void
    {
        DB::beginTransaction();
    }

    public function commit(): void
    {
        DB::commit();
    }

    public function rollback(): void
    {
        DB::rollBack();
    }

    public function savepoint(string $name): void
    {
        DB::statement("SAVEPOINT {$name}");
    }

    public function rollbackToSavepoint(string $name): void
    {
        DB::statement("ROLLBACK TO SAVEPOINT {$name}");
    }

    public function inTransaction(): bool
    {
        return DB::transactionLevel() > 0;
    }

    public function getTransactionLevel(): int
    {
        return DB::transactionLevel();
    }
}

Transaction Strategies

TRANSACTIONAL Strategy

Use Case: Critical imports where all-or-nothing is required (e.g., financial data)
Behavior: Single transaction wrapping entire import, rollback on any critical error
Memory: Holds all changes in memory until commit
Error Handling: One critical error fails entire import

$result = $importManager->import(
    strategy: ImportStrategy::TRANSACTIONAL,
    transactionManager: $transactionManager  // REQUIRED
);

BATCH Strategy

Use Case: Large imports where partial success is acceptable
Behavior: Transaction per batch (500 rows default), continue on batch failure
Memory: Moderate - processes in chunks
Error Handling: Failed batches are skipped, others continue

$result = $importManager->import(
    strategy: ImportStrategy::BATCH,
    transactionManager: $transactionManager  // OPTIONAL
);

STREAM Strategy

Use Case: Very large files with minimal memory footprint
Behavior: Row-by-row processing, no transaction wrapper
Memory: Minimal - processes one row at a time
Error Handling: Row failures don't affect other rows

$result = $importManager->import(
    strategy: ImportStrategy::STREAM
    // No transactionManager needed
);

Error Handling

Error Severity Levels

WARNING: Non-critical issue, row processed successfully
ERROR: Critical issue, row skipped
CRITICAL: System failure, entire import may fail (TRANSACTIONAL mode)

Accessing Errors

// Get all errors
$allErrors = $result->getAllErrors();

// Get errors by severity
$errorCounts = $result->getErrorCountBySeverity();
// ['WARNING' => 5, 'ERROR' => 12, 'CRITICAL' => 0]

// Get errors by field
$fieldErrors = $result->getErrorsByField();
// ['email' => [ImportError, ImportError], 'age' => [ImportError]]

// Get errors by row
$rowErrors = $result->getErrorsByRow();
// [1 => [ImportError], 5 => [ImportError, ImportError]]

// Check success rate
    return $successRate = $result->getSuccessRate();  // 87.5%

Available Interfaces

The package defines 10 core interfaces for dependency injection:

Primary Interfaces

Interface	Purpose	Implementation Location
`ImportManagerInterface`	❌ Not provided - Use concrete `ImportManager`	Consumer defines if abstraction needed
`ImportParserInterface`	Parse import files (CSV, JSON, XML, Excel)	Package provides CSV/JSON/XML; Consumer implements Excel
`TransactionManagerInterface`	Database transaction management	Consumer implements (Laravel, Symfony, etc.)
`ImportHandlerInterface`	Domain-specific persistence logic	Consumer implements per entity type
`ImportProcessorInterface`	Process import with strategy enforcement	Package provides `ImportProcessor`

Engine Interfaces

Interface	Purpose	Provided Implementation
`TransformerInterface`	Apply transformation rules to data	`DataTransformer`
`FieldMapperInterface`	Map source fields to target fields	`FieldMapper`
`ImportValidatorInterface`	Validate import definitions	`DefinitionValidator`
`DuplicateDetectorInterface`	Detect duplicate records	`DuplicateDetector`

Optional Interfaces

Interface	Purpose	Required?
`ImportAuthorizerInterface`	Authorization checks	❌ Optional - Pass `null` if not needed
`ImportContextInterface`	Tenant/context management	❌ Optional - Pass `null` if not needed

See full API documentation: docs/api-reference.md

Integration Examples

Laravel Integration

// app/Services/Import/CustomerImportHandler.php
use Nexus\Import\Contracts\ImportHandlerInterface;

final class CustomerImportHandler implements ImportHandlerInterface
{
    public function __construct(
        private readonly CustomerRepository $repository
    ) {}

    public function handle(array $data, ImportMode $mode): void
    {
        match($mode) {
            ImportMode::CREATE => $this->repository->create($data),
            ImportMode::UPSERT => $this->repository->upsert($data),
        };
    }

    public function getUniqueKeyFields(): array
    {
        return ['email'];
    }
}

// routes/web.php
Route::post('/import/customers', function (Request $request, ImportManager $importManager) {
    $result = $importManager->import(
        filePath: $request->file('import_file')->getRealPath(),
        format: ImportFormat::CSV,
        handler: app(CustomerImportHandler::class),
        mappings: [/* field mappings */],
        mode: ImportMode::UPSERT
    );
    
    return response()->json([
        'success' => $result->successCount,
        'failed' => $result->failedCount,
        'errors' => $result->getAllErrors()
    ]);
});

See complete integration guide: docs/integration-guide.md

See working examples: docs/examples/

Advanced Usage

Custom Transformations

use Nexus\Import\Core\Engine\DataTransformer;

$transformer = new DataTransformer();

// Register custom rule
$transformer->registerRule('encrypt', function($value) {
    return encrypt($value);
});

// Use in mapping
new FieldMapping(
    sourceField: 'ssn',
    targetField: 'encrypted_ssn',
    transformations: ['trim', 'encrypt']
);

Auto-Mapping

use Nexus\Import\Core\Engine\FieldMapper;

$fieldMapper = new FieldMapper($transformer);

// Automatically map matching field names
$autoMappings = $fieldMapper->autoMap(
    sourceHeaders: ['customer_name', 'email_address', 'phone_number'],
    targetFields: ['customer_name', 'email_address', 'phone']
);

// Result: Maps customer_name and email_address (exact match)
// phone_number doesn't match 'phone', must be manually defined

Duplicate Detection

use Nexus\Import\Core\Engine\DuplicateDetector;

$duplicateDetector = new DuplicateDetector();

// Detect duplicates within import file
$duplicates = $duplicateDetector->detectInternal(
    rows: $definition->rows,
    uniqueKeyFields: ['email']
);

// Detect duplicates against existing data
$externalDuplicate = $duplicateDetector->detectExternal(
    row: $data,
    uniqueKeyFields: ['email'],
    existsCheck: fn($data) => Customer::where('email', $data['email'])->exists(),
    rowNumber: 1
);

Integration with Atomy (Laravel)

1. Create ExcelParser

// apps/Atomy/app/Services/Import/ExcelParser.php
namespace App\Services\Import;

use Nexus\Import\Contracts\ImportParserInterface;
use Nexus\Import\ValueObjects\{ImportDefinition, ImportMetadata, ImportFormat};
use PhpOffice\PhpSpreadsheet\IOFactory;

final class ExcelParser implements ImportParserInterface
{
    public function parse(string $filePath, ImportMetadata $metadata): ImportDefinition
    {
        $spreadsheet = IOFactory::load($filePath);
        $worksheet = $spreadsheet->getActiveSheet();
        
        $headers = [];
        $rows = [];
        
        foreach ($worksheet->getRowIterator() as $index => $row) {
            $cellIterator = $row->getCellIterator();
            $cellIterator->setIterateOnlyExistingCells(false);
            
            $data = [];
            foreach ($cellIterator as $cell) {
                $data[] = $cell->getValue();
            }
            
            if ($index === 1) {
                $headers = $data;
            } else {
                $rows[] = array_combine($headers, $data);
            }
        }
        
        return new ImportDefinition($headers, $rows, $metadata);
    }

    public function supports(ImportFormat $format): bool
    {
        return $format === ImportFormat::EXCEL;
    }
}

2. Register in Service Provider

// apps/Atomy/app/Providers/ImportServiceProvider.php
namespace App\Providers;

use Illuminate\Support\ServiceProvider;
use Nexus\Import\Services\{ImportManager, ImportProcessor};
use Nexus\Import\Parsers\{CsvParser, JsonParser, XmlParser};
use Nexus\Import\ValueObjects\ImportFormat;
use App\Services\Import\{ExcelParser, LaravelTransactionManager};

final class ImportServiceProvider extends ServiceProvider
{
    public function register(): void
    {
        // Bind TransactionManager
        $this->app->singleton(TransactionManagerInterface::class, LaravelTransactionManager::class);
        
        // Bind ImportManager
        $this->app->singleton(ImportManager::class, function ($app) {
            $manager = new ImportManager(
                processor: $app->make(ImportProcessor::class),
                authorizer: null,
                context: null,
                logger: $app->make(LoggerInterface::class)
            );
            
            // Register native parsers
            $manager->registerParser(ImportFormat::CSV, new CsvParser());
            $manager->registerParser(ImportFormat::JSON, new JsonParser());
            $manager->registerParser(ImportFormat::XML, new XmlParser());
            
            // Register Excel parser (Atomy-specific)
            $manager->registerParser(ImportFormat::EXCEL, new ExcelParser());
            
            return $manager;
        });
    }
}

Architecture

The Nexus\Import package follows strict framework-agnostic principles:

Pure PHP: No Laravel dependencies in package code
Contract-Driven: All external dependencies via interfaces
Immutable VOs: Value objects are readonly and validated
Error Collection: Transformations/validations collect errors (don't throw) for batch completion
Separation of Concerns: Processor enforces strategies, Handler handles persistence

Package Structure

packages/Import/
├── src/
│   ├── Contracts/              # 10 interfaces
│   │   ├── ImportParserInterface.php
│   │   ├── TransactionManagerInterface.php
│   │   ├── TransformerInterface.php
│   │   ├── FieldMapperInterface.php
│   │   ├── ImportValidatorInterface.php
│   │   ├── ImportHandlerInterface.php
│   │   ├── ImportProcessorInterface.php
│   │   ├── DuplicateDetectorInterface.php
│   │   ├── ImportAuthorizerInterface.php
│   │   └── ImportContextInterface.php
│   ├── Core/Engine/           # 6 engine components
│   │   ├── DataTransformer.php
│   │   ├── FieldMapper.php
│   │   ├── DefinitionValidator.php
│   │   ├── DuplicateDetector.php
│   │   ├── ErrorCollector.php
│   │   └── BatchProcessor.php
│   ├── Exceptions/            # 7 exceptions
│   ├── Parsers/               # 3 native parsers (CSV/JSON/XML)
│   ├── Services/              # 2 services (Manager/Processor)
│   └── ValueObjects/          # 9 VOs (4 enums, 5 classes)
└── composer.json

Testing

Running Package Tests

# Run all tests
composer test

# Run with coverage
composer test:coverage

# Run specific test suite
./vendor/bin/phpunit --filter TransformerTest

Current Test Status

Coverage: 0% (Tests pending implementation)
Planned Tests: ~65 tests (50 unit, 15 integration)
Target Coverage: 90%+

See test documentation: TEST_SUITE_SUMMARY.md

Writing Custom Tests

use PHPUnit\Framework\TestCase;
use Nexus\Import\Core\Engine\DataTransformer;

final class CustomTransformationTest extends TestCase
{
    public function test_custom_transformation_rule(): void
    {
        $transformer = new DataTransformer();
        
        $transformer->registerRule('reverse', fn($value) => strrev($value));
        
        $result = $transformer->apply('hello', ['reverse']);
        
        $this->assertSame('olleh', $result);
    }
}

Documentation

Package Documentation

Getting Started Guide - Quick start with Laravel integration
API Reference - Complete interface and class documentation
Integration Guide - Laravel, Symfony, and vanilla PHP integration
Basic Usage Example - Simple customer CSV import
Advanced Usage Example - Complex scenarios with validation

Project Documentation

Implementation Summary - Development metrics and progress
Requirements - 78 documented requirements with traceability
Test Suite Summary - Test coverage and planned tests
Valuation Matrix - Package valuation ($160K estimated value)

License

MIT License. See LICENSE file for details.

Contributing

This package is part of the Nexus ERP monorepo. See main repository for contribution guidelines.

Version: 1.0.0
Status: Production Ready
Maintained By: Nexus Development Team
Last Updated: November 25, 2024

azaharizaman / nexus-import

Maintainers

Package info

Statistics

Security