README

A CSV import engine for Laravel Eloquent models.

Installation

You can install the package via Composer:

composer require yossuf/laravel-import

You may publish all of the package's resources at once:

php artisan vendor:publish --tag="laravel-import"

Or, you may publish each resource individually:

Publishing the Configuration File

php artisan vendor:publish --tag="laravel-import-config"

The published config/import.php covers the tracking table, the queue and the default sizes, each with an environment variable:

return [
    // Where tracking records live and how long they are kept. A null
    // connection uses the application's default; a null prune_after
    // keeps every record.
    'tracking' => [
        'database' => env('IMPORT_CONNECTION'),
        'table' => env('IMPORT_TABLE', 'IMPORT'),
        'prune_after' => env('IMPORT_PRUNE_AFTER'),
    ],

    // Imports run as queued jobs; the "sync" connection runs them inline.
    // The lock values tune how the import jobs of one import wait for
    // each other; expire_after has to exceed the time a single one takes.
    'queue' => [
        'connection' => env('IMPORT_QUEUE_CONNECTION'),
        'queue' => env('IMPORT_QUEUE'),
        'release_after' => (int) env('IMPORT_QUEUE_RELEASE_AFTER', 5),
        'expire_after' => (int) env('IMPORT_QUEUE_EXPIRE_AFTER', 300),
    ],

    // Rows per queued job for ShouldChunkRead importers whose chunkSize()
    // returns null.
    'chunk_size' => (int) env('IMPORT_CHUNK_SIZE', 500),

    // Rows per upsert statement for ShouldBulkWrite importers whose
    // bulkSize() returns null.
    'bulk_size' => (int) env('IMPORT_BULK_SIZE', 500),
];

IMPORT_CONNECTION=mysql
IMPORT_TABLE=IMPORT
IMPORT_QUEUE_CONNECTION=redis
IMPORT_QUEUE=imports
IMPORT_QUEUE_RELEASE_AFTER=5
IMPORT_QUEUE_EXPIRE_AFTER=300
IMPORT_CHUNK_SIZE=500
IMPORT_BULK_SIZE=500

Running the Migrations

The package loads its migrations automatically, so migrating is enough:

php artisan migrate

Publish them instead if you want the files in your application, for example to change the table name before it is created:

php artisan vendor:publish --tag="laravel-import-migrations"
php artisan migrate

Pruning Old Tracking Records

Tracking records are kept forever by default. Set prune_after to a relative period Carbon understands to let Laravel's model:prune command delete the older ones:

IMPORT_PRUNE_AFTER="30 days"

"6 months" and "1 year" work just as well. A record is aged by its created_at and pruned whatever state it ended in, so an import abandoned in Waiting or Processing is cleaned up too.

An invalid period fails the prune run with an InvalidArgumentException naming the value: a typo such as "30" or "banana months", or a period that is not in the past, like "-30 days" or "0 days", which would prune every record. Leaving prune_after unset is the only way to switch pruning off. Confirm a new period with --pretend before scheduling it.

The command only auto-discovers models in your own app/Models, so name this one explicitly when scheduling it:

use Illuminate\Support\Facades\Schedule;
use Yossuf\Laravel\Import\Models\Import;

Schedule::command('model:prune', [
    '--model' => [Import::class],
])->daily();

Check what would be deleted without deleting it:

php artisan model:prune --model="Yossuf\Laravel\Import\Models\Import" --pretend

Usage

Once the IMPORT table is migrated:

1. Write an importer for your model

Implement Yossuf\Laravel\Import\Contracts\ModelImport to say which Eloquent model a CSV file is imported into and how each row maps onto it. The package reads the file for you.

namespace App\Imports;

use App\Models\Product;
use Yossuf\Laravel\Import\Concerns\Importable;
use Yossuf\Laravel\Import\Contracts\ModelImport;

class ProductImport implements ModelImport
{
    use Importable;

    public function model(): string
    {
        return Product::class;
    }

    public function rules(): array
    {
        return [
            'sku' => ['required', 'string', 'max:255'],
            'name' => ['required', 'string', 'max:255'],
        ];
    }

    public function map(array $row): array
    {
        return [
            'sku' => $row['sku'],
            'name' => $row['name'],
        ];
    }

    public function uniqueBy(): array
    {
        return ['sku'];
    }
}

Importable supplies readerOptions(). Override it when the file calls for it:

public function readerOptions(): array
{
    return ['delimiter' => ';', 'enclosure' => "'", 'encoding' => 'ISO-8859-1'];
}

An importer like the one above is read row by row, each row imported as it is read. To read a large file in chunks, each imported by its own queued job, implement ShouldChunkRead as well. Importable already supplies chunkSize() returning null, which means config('import.chunk_size'); override it to pick a size:

use Yossuf\Laravel\Import\Contracts\ModelImport;
use Yossuf\Laravel\Import\Contracts\ShouldChunkRead;

class ProductImport implements ModelImport, ShouldChunkRead
{
    use Importable;

    // ...

    public function chunkSize(): ?int
    {
        return 1000; // instead of config('import.chunk_size')
    }
}

Rows are written one upsert statement per row unless the importer implements ShouldBulkWrite, which groups a chunk's rows into statements of bulkSize() rows. The two contracts are independent. Importable supplies bulkSize() returning null as well, for config('import.bulk_size'):

use Yossuf\Laravel\Import\Contracts\ShouldBulkWrite;

class ProductImport implements ModelImport, ShouldChunkRead, ShouldBulkWrite
{
    use Importable;

    // ...

    public function bulkSize(): ?int
    {
        return 250; // instead of config('import.bulk_size')
    }
}

The model needs nothing from the package. Rows are written with Eloquent's upsert(), which bypasses model events, observers, attribute casts, set-mutators and $fillable/$guarded, but still fills in generated keys and timestamps:

Model's key	Who assigns it
Generated by the model (`HasUuids`, `HasUlids`, or any model overriding `newUniqueId()`)	Eloquent calls the model's `newUniqueId()` for each row whose `map()` leaves the key out
Auto-incrementing	The database — the engine never prefills it
Natural (e.g. a string code)	Your importer's `map()` — return the key as one of the attributes

map() receives only the columns rules() names, with the values as validated. A column the file has but the rules leave out never reaches map(), so a file cannot smuggle in an attribute the importer did not ask for, even when map() returns the row as is. Give every column you read a rule, if only ['present', 'nullable'].

map() has to return every uniqueBy() column; an importer that leaves one out fails the import with a LogicException naming the column.

Because casts and mutators do not run, map() has to return each value the way the column stores it. A hashed or encrypted cast, a json cast or a set-mutator on the model has no effect on an import, so a password mapped as is would be stored in plain text. Do in map() what the cast would have done:

use Illuminate\Support\Facades\Crypt;
use Illuminate\Support\Facades\Hash;

public function rules(): array
{
    return [
        'email' => ['required', 'email', 'max:255'],
        'password' => ['required', 'string', 'min:8'],
        'api_token' => ['present', 'nullable', 'string'],
        'roles' => ['present', 'nullable', 'string'],
    ];
}

public function map(array $row): array
{
    return [
        'email' => $row['email'],
        'password' => Hash::make($row['password']),
        'api_token' => $row['api_token'] === null ? null : Crypt::encryptString($row['api_token']),
        'roles' => $row['roles'] === null ? '[]' : json_encode(explode('|', $row['roles'])),
    ];
}

Reading the model back works as usual: the hashed, encrypted and array casts on User decode what map() stored, because the stored values are what User::create() would have written.

2. Run an import

use App\Imports\ProductImport;
use Yossuf\Laravel\Import\Facades\Import;

$record = Import::dispatch($file, new ProductImport());

Import::dispatch() is the package's only entry point. It hands the file to a queued job rather than reading it itself.

$file is trusted: the package hashes it, counts its lines, reads it and records the full path on the tracking record. Build the path yourself, for example from a Storage disk, and never from a user's original filename or a path a request supplies. The file is always parsed as CSV, whatever its extension.

3. How an import runs

Import::dispatch() returns straight away with a Waiting record; a queued job reads the file and settles the record when every import job is done:

$record = Import::dispatch($file, new ProductImport());

$record->state;         // ImportState::Waiting
$record->id;            // track it from here

The record also stores the file's MD5 hash, taken at dispatch. The producer job recomputes it before reading and fails the import if it differs, so a file replaced between dispatch and processing is never imported under the old record. When the dispatching process cannot read the file, no hash is stored and the check is skipped.

Poll a fresh copy for progress:

$record->fresh()->progress();   // approximate while the file is read, exact once settled
$record->fresh()->state;        // Waiting → Processing → Success | Failed
$record->fresh()->rows_created;

Every row is counted as created, updated or skipped, so the three add up to rows_total once the import is done. Two rows sharing a uniqueBy() value both count, though the table ends up with one row.

The file is parsed once, streamed by one producer job. Before streaming, the producer counts the file's lines in a single fast pass and stores that, minus the header, as a provisional rows_total, so progress() has a denominator from the first row. The estimate is an upper bound — a quoted field spanning lines counts once per line — and is replaced by the exact count once the last row has been read. The two optional contracts decide how rows get from the file to the database:

Importer implements	Reading	Writing
neither	row by row, each row imported inline as it is read	one statement per row
`ShouldChunkRead`	in chunks, each queued as an import job	one statement per row
`ShouldBulkWrite`	row by row, inline	one statement per row, since a job holds a single row
both	in chunks, each queued as an import job	statements of the bulk size

The chunk size has to be at least 1 and is resolved in this order:

the chunk argument to Import::dispatch()
the importer's chunkSize(), when it returns one
config('import.chunk_size'), 500 by default

The bulk size is resolved the same way, from the bulk argument, then bulkSize(), then config('import.bulk_size'), and also has to be at least 1. Either argument is ignored for an importer without the matching contract.

Two things follow from the table. ShouldBulkWrite on its own is allowed but changes nothing: every import job carries one row, so every statement writes one row. And a bulk size larger than the chunk size is capped by it: an import job never holds more than one chunk, so a chunk of 500 with a bulk size of 1000 is still written in one statement of 500.

A row repeated in one file counts as created then updated when the two land in different statements, and as two created rows when one bulk statement holds both.

Import::dispatch($file, new ProductImport(), chunkSize: 1000, bulkSize: 250);

The producer opens $file on the worker, not in the process that called Import::dispatch(), so the path has to point at storage the worker can reach — a shared disk, not a request-local temp file. A path the worker cannot open settles the record Failed with the reader's "File does not exist" message.

Both jobs carry your ModelImport instance in their payload, so write it as a named class with no closures or open resources; an anonymous class, or one holding a file handle, fatals at dispatch.

Import jobs of one import run one at a time, guarded by WithoutOverlapping, so the created/updated counts stay exact. Imports of different files still run in parallel.

Your app needs Laravel's job_batches table and a cache store that supports locks. Configure where the jobs go:

IMPORT_QUEUE_CONNECTION=redis
IMPORT_QUEUE=imports

Set IMPORT_QUEUE_CONNECTION=sync to run an import inline, as the package's own tests do. The returned record is then already settled, but a failing import rethrows after marking the record Failed, so a synchronous call needs a try/catch. On a real queue the same failure surfaces in the worker and the batch's catch() callback settles the record.

The record's error holds the exception message. A database failure is recorded by the driver's message alone: the SQL statement with its bound values and the connection's host and database that Laravel appends are left out, so the tracking table never keeps a copy of the rows that failed. The driver's own message may still name the one value it rejected, as in a duplicate key error.

A chunked import's rows travel to the workers in the job payload, so a large chunk size with wide rows makes for large payloads; lower the chunk size if your queue backend complains. The bulk size is the matching knob for the database: a statement of many wide rows can exceed a driver's bound-parameter limit. An import without ShouldChunkRead never puts rows on the queue; it writes inside the producer job, one row at a time.

License

Laravel Import is open-sourced software licensed under the Apache-2.0 license.

yossuf / laravel-import

Package info

Security

Statistics