onamfc / laravel-database-archiver
A Laravel package for efficient database record archival to cold storage
Requires
- php: ^8.1
- aws/aws-sdk-php: ^3.0
- illuminate/console: ^10.0|^11.0
- illuminate/database: ^10.0|^11.0
- illuminate/support: ^10.0|^11.0
- league/flysystem-aws-s3-v3: ^3.0
Requires (Dev)
- orchestra/testbench: ^8.0|^9.0
- phpunit/phpunit: ^10.0
This package is auto-updated.
Last update: 2025-09-11 21:41:56 UTC
README
A comprehensive Laravel package for efficient database record archival to cold storage systems like AWS S3, with support for multiple formats and automated scheduling.
Features
- Multi-storage Support: AWS S3, local filesystem, and extensible for other providers
- Multiple Formats: JSON and Parquet export formats
- Configurable Criteria: Flexible record selection based on age, status, or custom conditions
- Scheduled Archival: Integration with Laravel Task Scheduler
- Memory Efficient: Processes large datasets in configurable chunks
- Comprehensive Logging: Detailed operation logs and database tracking
- Artisan Commands: Easy-to-use CLI interface
- Multi-table Support: Archive multiple tables with different configurations
Installation
Install the package via Composer:
composer require onamfc/laravel-database-archiver
Publish the configuration file:
php artisan vendor:publish --tag=db-archiver-config
Publish and run the migrations:
php artisan vendor:publish --tag=db-archiver-migrations php artisan migrate
Configuration
Configure your archival settings in config/db-archiver.php
:
Basic Storage Configuration
'storage' => [ 's3' => [ 'driver' => 's3', 'key' => env('AWS_ACCESS_KEY_ID'), 'secret' => env('AWS_SECRET_ACCESS_KEY'), 'region' => env('AWS_DEFAULT_REGION', 'us-east-1'), 'bucket' => env('DB_ARCHIVER_S3_BUCKET'), ], 'local' => [ 'driver' => 'local', 'root' => storage_path('app/archives'), ], ],
Table Configuration
'tables' => [ 'add-table-name-here' => [ 'enabled' => true, 'criteria' => [ 'column' => 'created_at', 'operator' => '<', 'value' => '6 months ago', ], 'format' => 'json', 'storage' => 's3', 'path' => 'archives/users/{date}', 'schedule' => 'daily', 'delete_after_archive' => false, ], 'add-another-table-name-here' => [ 'enabled' => true, 'criteria' => [ 'column' => 'created_at', 'operator' => '<', 'value' => '3 months ago', ], 'format' => 'parquet', 'storage' => 's3', 'path' => 'archives/logs/{date}', 'schedule' => 'weekly', 'delete_after_archive' => true, 'additional_criteria' => [ ['column' => 'level', 'operator' => '=', 'value' => 'debug'], ], ], ],
Environment Variables
Add these to your .env
file:
# Storage Configuration DB_ARCHIVER_STORAGE=s3 DB_ARCHIVER_S3_BUCKET=your-archive-bucket DB_ARCHIVER_FORMAT=json DB_ARCHIVER_CHUNK_SIZE=1000 # AWS Credentials (if using S3) AWS_ACCESS_KEY_ID=your-access-key AWS_SECRET_ACCESS_KEY=your-secret-key AWS_DEFAULT_REGION=us-east-1 # Logging DB_ARCHIVER_LOGGING=true DB_ARCHIVER_LOG_CHANNEL=daily
Usage
Artisan Commands
Archive a Single Table
# Archive records from the users table php artisan archive:table users # Dry run to see what would be archived php artisan archive:table users --dry-run # Force archival even if disabled in config php artisan archive:table users --force
Archive All Configured Tables
# Archive all enabled tables php artisan archive:all # Dry run for all tables php artisan archive:all --dry-run
Check Archival Status
# Status for all tables php artisan archive:status # Status for specific table php artisan archive:status users
Programmatic Usage
use onamfc\LaravelDbArchiver\Services\ArchiveService; class YourController extends Controller { public function archiveData(ArchiveService $archiveService) { // Archive a specific table $result = $archiveService->archiveTable('users'); // Archive all configured tables $results = $archiveService->archiveAll(); // Get status $status = $archiveService->getStatus(); return response()->json($result); } }
Scheduled Archival
The package automatically registers scheduled tasks based on your table configurations. The following schedules are supported:
daily
- Runs daily at midnightweekly
- Runs weekly on Sunday at midnightmonthly
- Runs monthly on the 1st at midnight- Custom cron expressions (e.g.,
0 2 * * *
for 2 AM daily)
Advanced Configuration
Custom Criteria Values
You can use dynamic values in your criteria:
'criteria' => [ 'column' => 'created_at', 'operator' => '<', 'value' => '6 months ago', // Carbon-parseable string ], // Or use a closure for complex logic 'criteria' => [ 'column' => 'status', 'operator' => '=', 'value' => function () { return config('app.archive_status'); }, ],
Multiple Criteria
'additional_criteria' => [ ['column' => 'status', 'operator' => '=', 'value' => 'inactive'], ['column' => 'last_login', 'operator' => '<', 'value' => '1 year ago'], ],
Custom Storage Paths
Use dynamic placeholders in storage paths:
'path' => 'archives/{table}/{date}', // archives/users/2024-01-15 'path' => 'backups/{table}/year={Y}/month={m}', // backups/users/year=2024/month=01
Storage Adapters
AWS S3
'storage' => [ 's3' => [ 'driver' => 's3', 'key' => env('AWS_ACCESS_KEY_ID'), 'secret' => env('AWS_SECRET_ACCESS_KEY'), 'region' => env('AWS_DEFAULT_REGION'), 'bucket' => env('DB_ARCHIVER_S3_BUCKET'), 'endpoint' => env('AWS_ENDPOINT'), // For S3-compatible services ], ],
Local Storage
'storage' => [ 'local' => [ 'driver' => 'local', 'root' => storage_path('app/archives'), ], ],
Data Formats
JSON Format
Stores data as formatted JSON files:
[ { "id": 1, "name": "John Doe", "email": "john@example.com", "created_at": "2024-01-01T00:00:00Z" } ]
Parquet Format
Stores data in Parquet format for efficient compression and analytics.
Monitoring and Logging
Database Logging
All archival operations are logged to the archive_logs
table:
$logs = \onamfc\LaravelDbArchiver\Models\ArchiveLog::where('table_name', 'users') ->latest() ->get();
File Logging
Operations are also logged to your configured log channel:
'logging' => [ 'enabled' => true, 'channel' => 'daily', 'level' => 'info', ],
Error Handling
The package includes comprehensive error handling:
- Storage connectivity issues
- Invalid configurations
- Data transformation errors
- Permission problems
- Memory limitations
Errors are logged and can trigger notifications if configured.
Security Considerations
- Use IAM roles with minimal required permissions for S3 access
- Enable server-side encryption for stored archives
- Regularly rotate access keys
- Monitor access logs for archived data
- Consider using S3 bucket policies for additional security
Performance Optimization
- Adjust
chunk_size
based on your available memory - Use appropriate S3 storage classes (Standard-IA, Glacier)
- Schedule archival during low-traffic periods
- Monitor database performance during archival operations
- Consider using database indexes on archival criteria columns
Testing
# Run package tests vendor/bin/phpunit # Test with specific configuration php artisan archive:table users --dry-run
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.