README

A comprehensive Laravel package for efficient database record archival to cold storage systems like AWS S3, with support for multiple formats and automated scheduling.

Features

Multi-storage Support: AWS S3, local filesystem, and extensible for other providers
Multiple Formats: JSON and Parquet export formats
Configurable Criteria: Flexible record selection based on age, status, or custom conditions
Scheduled Archival: Integration with Laravel Task Scheduler
Memory Efficient: Processes large datasets in configurable chunks
Comprehensive Logging: Detailed operation logs and database tracking
Artisan Commands: Easy-to-use CLI interface
Multi-table Support: Archive multiple tables with different configurations

Installation

Install the package via Composer:

composer require onamfc/laravel-database-archiver

Publish the configuration file:

php artisan vendor:publish --tag=db-archiver-config

Publish and run the migrations:

php artisan vendor:publish --tag=db-archiver-migrations
php artisan migrate

Configuration

Configure your archival settings in config/db-archiver.php:

Basic Storage Configuration

'storage' => [
    's3' => [
        'driver' => 's3',
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
        'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
        'bucket' => env('DB_ARCHIVER_S3_BUCKET'),
    ],
    'local' => [
        'driver' => 'local',
        'root' => storage_path('app/archives'),
    ],
],

Table Configuration

'tables' => [
    'add-table-name-here' => [
        'enabled' => true,
        'criteria' => [
            'column' => 'created_at',
            'operator' => '<',
            'value' => '6 months ago',
        ],
        'format' => 'json',
        'storage' => 's3',
        'path' => 'archives/users/{date}',
        'schedule' => 'daily',
        'delete_after_archive' => false,
    ],
    'add-another-table-name-here' => [
        'enabled' => true,
        'criteria' => [
            'column' => 'created_at',
            'operator' => '<',
            'value' => '3 months ago',
        ],
        'format' => 'parquet',
        'storage' => 's3',
        'path' => 'archives/logs/{date}',
        'schedule' => 'weekly',
        'delete_after_archive' => true,
        'additional_criteria' => [
            ['column' => 'level', 'operator' => '=', 'value' => 'debug'],
        ],
    ],
],

Environment Variables

Add these to your .env file:

# Storage Configuration
DB_ARCHIVER_STORAGE=s3
DB_ARCHIVER_S3_BUCKET=your-archive-bucket
DB_ARCHIVER_FORMAT=json
DB_ARCHIVER_CHUNK_SIZE=1000

# AWS Credentials (if using S3)
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_DEFAULT_REGION=us-east-1

# Logging
DB_ARCHIVER_LOGGING=true
DB_ARCHIVER_LOG_CHANNEL=daily

Usage

Artisan Commands

Archive a Single Table

# Archive records from the users table
php artisan archive:table users

# Dry run to see what would be archived
php artisan archive:table users --dry-run

# Force archival even if disabled in config
php artisan archive:table users --force

Archive All Configured Tables

# Archive all enabled tables
php artisan archive:all

# Dry run for all tables
php artisan archive:all --dry-run

Check Archival Status

# Status for all tables
php artisan archive:status

# Status for specific table
php artisan archive:status users

Programmatic Usage

use onamfc\LaravelDbArchiver\Services\ArchiveService;

class YourController extends Controller
{
    public function archiveData(ArchiveService $archiveService)
    {
        // Archive a specific table
        $result = $archiveService->archiveTable('users');
        
        // Archive all configured tables
        $results = $archiveService->archiveAll();
        
        // Get status
        $status = $archiveService->getStatus();
        
        return response()->json($result);
    }
}

Scheduled Archival

The package automatically registers scheduled tasks based on your table configurations. The following schedules are supported:

daily - Runs daily at midnight
weekly - Runs weekly on Sunday at midnight
monthly - Runs monthly on the 1st at midnight
Custom cron expressions (e.g., 0 2 * * * for 2 AM daily)

Advanced Configuration

Custom Criteria Values

You can use dynamic values in your criteria:

'criteria' => [
    'column' => 'created_at',
    'operator' => '<',
    'value' => '6 months ago', // Carbon-parseable string
],

// Or use a closure for complex logic
'criteria' => [
    'column' => 'status',
    'operator' => '=',
    'value' => function () {
        return config('app.archive_status');
    },
],

Multiple Criteria

'additional_criteria' => [
    ['column' => 'status', 'operator' => '=', 'value' => 'inactive'],
    ['column' => 'last_login', 'operator' => '<', 'value' => '1 year ago'],
],

Custom Storage Paths

Use dynamic placeholders in storage paths:

'path' => 'archives/{table}/{date}', // archives/users/2024-01-15
'path' => 'backups/{table}/year={Y}/month={m}', // backups/users/year=2024/month=01

Storage Adapters

AWS S3

'storage' => [
    's3' => [
        'driver' => 's3',
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
        'region' => env('AWS_DEFAULT_REGION'),
        'bucket' => env('DB_ARCHIVER_S3_BUCKET'),
        'endpoint' => env('AWS_ENDPOINT'), // For S3-compatible services
    ],
],

Local Storage

'storage' => [
    'local' => [
        'driver' => 'local',
        'root' => storage_path('app/archives'),
    ],
],

Data Formats

JSON Format

Stores data as formatted JSON files:

[
    {
        "id": 1,
        "name": "John Doe",
        "email": "john@example.com",
        "created_at": "2024-01-01T00:00:00Z"
    }
]

Parquet Format

Stores data in Parquet format for efficient compression and analytics.

Monitoring and Logging

Database Logging

All archival operations are logged to the archive_logs table:

$logs = \onamfc\LaravelDbArchiver\Models\ArchiveLog::where('table_name', 'users')
    ->latest()
    ->get();

File Logging

Operations are also logged to your configured log channel:

'logging' => [
    'enabled' => true,
    'channel' => 'daily',
    'level' => 'info',
],

Error Handling

The package includes comprehensive error handling:

Storage connectivity issues
Invalid configurations
Data transformation errors
Permission problems
Memory limitations

Errors are logged and can trigger notifications if configured.

Security Considerations

Use IAM roles with minimal required permissions for S3 access
Enable server-side encryption for stored archives
Regularly rotate access keys
Monitor access logs for archived data
Consider using S3 bucket policies for additional security

Performance Optimization

Adjust chunk_size based on your available memory
Use appropriate S3 storage classes (Standard-IA, Glacier)
Schedule archival during low-traffic periods
Monitor database performance during archival operations
Consider using database indexes on archival criteria columns

Testing

# Run package tests
vendor/bin/phpunit

# Test with specific configuration
php artisan archive:table users --dry-run

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

onamfc / laravel-database-archiver

Maintainers

Details