README

Production-grade Craft CMS 4/5 module for migrating assets from AWS S3 to DigitalOcean Spaces with checkpoint/resume, rollback capabilities, and zero-downtime support.

✨ Highlights

Production-Ready: Battle-tested migration toolkit with enterprise-grade reliability
Checkpoint/Resume System: Survive interruptions and resume from exactly where you left off
Complete Rollback: Full rollback capabilities with comprehensive change logs
Zero Dependencies: Works with just Craft CMS - no additional packages required
Auto-Bootstrap: PSR-4 autoloaded - no manual configuration in config/app.php
Memory Efficient: Handles 100,000+ assets with intelligent batch processing
Comprehensive Dashboard: Web-based Control Panel for orchestration and monitoring

🎯 Key Features

Migration Capabilities

14 Specialized Controllers for different migration phases
Batch Processing with configurable batch sizes for memory efficiency
Dry Run Mode to test migrations without making changes
Progress Tracking with real-time ETA and throughput metrics
Error Recovery with automatic retry logic and health checks
Idempotent Operations - safe to run multiple times

Control Panel Dashboard

Web-based interface for migration orchestration
Real-time status monitoring
Command execution from the browser
Checkpoint inspection tools
Live log streaming
Connection testing

Developer Experience

Centralized configuration via MigrationConfig helper
40+ type-safe configuration methods
Comprehensive inline documentation
Extensive architecture documentation
Custom Twig filters included

📋 Requirements

PHP: 8.0 or higher
Craft CMS: 4.0+ or 5.0+
AWS S3: Source bucket with read access
DigitalOcean Spaces: Target bucket with write access

🚀 Installation

1. Install via Composer

composer require csabourin/craft-s3-spaces-migration

The plugin will auto-install and automatically create config/migration-config.php if it doesn't exist.

2. Configure Environment Variables

Add these to your .env file:

# Migration Environment
MIGRATION_ENV=dev

# DigitalOcean Spaces Credentials
DO_S3_ACCESS_KEY=your_spaces_access_key
DO_S3_SECRET_KEY=your_spaces_secret_key
DO_S3_BUCKET=your-bucket-name
DO_S3_BASE_URL=https://your-bucket.tor1.digitaloceanspaces.com
DO_S3_BASE_ENDPOINT=https://tor1.digitaloceanspaces.com

See .env.example for all available options.

3. Update Migration Config

Edit config/migration-config.php:

$awsSource = [
    'bucket' => 'your-aws-bucket',      // ← Update this
    'region' => 'us-east-1',             // ← Update this
];

4. Verify Installation

./craft s3-spaces-migration/migration-check/check

📖 Usage

Console Commands

Pre-Flight Check

./craft s3-spaces-migration/migration-check/check

Migrate Assets (Dry Run)

./craft s3-spaces-migration/image-migration/migrate --dryRun=1

Migrate Assets (Production)

./craft s3-spaces-migration/image-migration/migrate

Replace URLs in Database

./craft s3-spaces-migration/url-replacement/replace-s3-urls

Switch Filesystems

./craft s3-spaces-migration/filesystem-switch/to-do

Post-Migration Diagnostics

./craft s3-spaces-migration/migration-diag/diagnose

Web Dashboard

Access the migration dashboard in your Control Panel:

Control Panel → Migration → Dashboard

The dashboard provides:

Step-by-step migration guidance
Real-time progress monitoring
Command execution interface
Configuration validation
Checkpoint management

🗺️ Migration Workflow

The complete migration process follows these phases:

Pre-Flight Validation - Verify configuration and connectivity
Database URL Replacement - Update S3 URLs in content tables
Template URL Replacement - Update hardcoded URLs in Twig templates
File Migration - Copy physical assets from AWS to DigitalOcean
Filesystem Switch - Switch volumes to use DigitalOcean Spaces
Configuration Updates - Update plugin configs and field settings
Transform Management - Discover and pre-generate image transforms
Post-Migration Verification - Validate successful migration

Each phase is modular and can be executed independently or in sequence.

OptimisedImages transform cleanup (new CLI module)

Before running the file migration or cleanup commands, purge stale transform files that live inside underscore-prefixed folders (for example _1200x800/hero.jpg) within the Optimised Images volume (ID 4). This prevents copying millions of auto-generated transforms to the new filesystem and keeps the migration delta small.

# Preview everything that would be removed
./craft s3-spaces-migration/transform-cleanup/clean --dryRun=1

# Execute the cleanup
./craft s3-spaces-migration/transform-cleanup/clean --dryRun=0

Each run saves a JSON report under storage/runtime/transform-cleanup/ so you can audit which files were targeted.

📂 Module Structure

modules/
├── module.php                     # Module entry point
├── controllers/                   # Web controllers
│   ├── DefaultController.php
│   └── MigrationController.php    # Dashboard controller
├── console/controllers/           # 14 console controllers
│   ├── MigrationCheckController.php
│   ├── ImageMigrationController.php
│   ├── FilesystemSwitchController.php
│   └── ... (11 more)
├── helpers/
│   └── MigrationConfig.php        # Centralized configuration
└── templates/s3-spaces-migration/
    └── dashboard.twig             # Control Panel interface

📚 Documentation

ARCHITECTURE.md - System design and technical details
DASHBOARD.md - Control Panel dashboard guide
PRODUCTION_RUNBOOK.md - Production deployment and troubleshooting
CHANGELOG.md - Version history
CONTRIBUTING.md - Contribution guidelines
SECURITY.md - Security policy and best practices

🔧 Configuration

The module uses a centralized configuration system with three layers:

Environment Variables (.env) - Credentials and secrets
Configuration File (config/migration-config.php) - Migration settings
Helper Class (MigrationConfig) - Type-safe access to config values

Key Configuration Options

return [
    'aws' => [
        'bucket' => 'your-source-bucket',
        'region' => 'us-east-1',
    ],
    'do' => [
        'bucket' => App::env('DO_S3_BUCKET'),
        'region' => 'tor1',
        'accessKey' => App::env('DO_S3_ACCESS_KEY'),
        'secretKey' => App::env('DO_S3_SECRET_KEY'),
    ],
    'migration' => [
        'batchSize' => 100,
        'checkpointFrequency' => 50,
        'maxRetries' => 3,
    ],
];

🛡️ Safety Features

Dry Run Mode: Test all operations before execution
Checkpoint System: Automatic progress saving every N items
Rollback Capability: Complete undo with change logs
Health Checks: Continuous validation during migration
Error Recovery: Automatic retry with exponential backoff
Audit Logging: Comprehensive logs of all operations

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for details.

Development Setup

Fork and clone the repository
Install dependencies: composer install
Set up a test Craft CMS installation
Configure test AWS S3 and DigitalOcean Spaces accounts
Run pre-flight checks: ./craft s3-spaces-migration/migration-check/check

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Built for Craft CMS by Pixel & Tonic
Inspired by real-world enterprise migration needs
Tested on production sites with millions of assets

💬 Support

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: Architecture Guide

🔗 Links

Repository: https://github.com/csabourin/do-migration
Packagist: https://packagist.org/packages/csabourin/craft-s3-spaces-migration
Craft CMS: https://craftcms.com/
DigitalOcean Spaces: https://www.digitalocean.com/products/spaces

Made with ❤️ for the Craft CMS community

csabourin / craft-s3-spaces-migration

Maintainers

Details