README

Automated Magento 2 Testing with AI-Powered Failure Analysis

Automated smoke and full testing for Magento 2 with AI-powered failure analysis using Google Gemini or OpenAI.

✨ Features

🧪 Automated Testing

Smoke Tests - Quick 5-page health checks (Homepage, Search, Product, Cart, Checkout)
Full Tests - Complete 8-step flow including API checkout with customer creation
Real-time Progress - Live updates via AJAX polling
Cron Integration - Automated scheduled testing

🤖 AI-Powered Analysis

Gemini AI - Google's latest AI model (gemini-2.0-flash)
OpenAI - GPT-4o-mini support
Smart Recommendations - Root cause analysis, fix steps, prevention tips
Beautiful UI - Gradient design with formatted markdown rendering

📊 Comprehensive Reporting

Test History - View all past test runs
Detailed Reports - Step-by-step results with timing and HTTP codes
Order Details - Track created orders, customers, and transactions
Quote Tracking - Monitor cart and quote masked IDs

⚙️ Flexible Configuration

Custom Test Products - Configure SKU, email, timeouts
Multiple Environments - Test different stores/websites
Payment Methods - Auto-detects enabled payment methods
Shipping Methods - Auto-detects enabled shipping carriers

📦 Installation

Via Composer (Recommended)

composer require minsar/magento2-ai-release-guardian
php bin/magento module:enable Minsar_ReleaseGuardian
php bin/magento setup:upgrade
php bin/magento setup:di:compile
php bin/magento cache:flush

Manual Installation

cd /path/to/magento
mkdir -p app/code/Minsar/ReleaseGuardian
# Download from: https://github.com/minsarai/magento2-ai-release-guardian
# Extract to app/code/Minsar/ReleaseGuardian
php bin/magento module:enable Minsar_ReleaseGuardian
php bin/magento setup:upgrade
php bin/magento setup:di:compile
php bin/magento cache:flush

# Add to crontab
* * * * * php /path/to/magento/bin/magento cron:run

🚀 Quick Start

1. Configure Settings

Admin → Stores → Configuration → Minsar → Release Guardian

Testing Settings:

Test Product SKU: 24-MB01 (or your test product)
Test Customer Email: test@example.com
Test Timeout: 5000ms
Default Country: US

AI Settings:

Enable AI Insights: Yes
AI Provider: Gemini or OpenAI
API Key: [Your API key]

2. Run Your First Test

Via Admin:

Admin → Minsar → Release Guardian → Overview → Run Smoke Tests

Via CLI:

php bin/magento guardian:run smoke
php bin/magento guardian:run full

3. View Results

Admin → Minsar → Release Guardian → History

🤖 AI Setup

Get Gemini API Key (Free)

Visit: https://aistudio.google.com/apikey
Click "Create API Key"
Copy key (starts with AIza...)
Add to: Admin → Configuration → AI Settings

Free Tier:

15 requests/minute
1M tokens/day
Perfect for testing!

Get OpenAI API Key

Visit: https://platform.openai.com/api-keys
Create new key
Add to configuration
Cost: ~$0.0001 per analysis

📋 What Gets Tested

Smoke Tests (5 steps)

✅ Homepage - Loads successfully
✅ Search Page - Search functionality works
✅ Product Page - Product details display
✅ Cart Page - Shopping cart accessible
✅ Checkout Page - Checkout process starts

Full Tests (8 steps)

✅ Homepage
✅ Search Page
✅ Product Page
✅ Cart Page
✅ Checkout Page
✅ API: Complete Checkout Flow
- Creates customer account
- Adds product to cart
- Sets billing/shipping address
- Applies payment method
- Places order
✅ Customer Login Page
✅ Customer Account Page

🎯 API Checkout Details

The API test creates a real order with:

✅ Customer Account - New customer with address
✅ Order Placement - Complete checkout flow
✅ Order Increment ID - Track order number
✅ Quote Masked ID - Guest cart tracking
✅ Payment Method - Auto-detected from config
✅ Shipping Method - Auto-detected from config
✅ Grand Total - Order total amount

Example Output:

Order: #000000005
Quote ID: abc123xyz
Email: test_1234567890@example.com
Payment: checkmo
Shipping: flatrate_flatrate
Total: $15.00

🤖 AI Analysis Example

When a test fails, AI provides:

🤖 AI Analysis

Root Cause Analysis
━━━━━━━━━━━━━━━━━━
Issue: Product Page 404 Error
The product is either disabled, deleted, or URL rewrite is missing.

Recommendations
━━━━━━━━━━━━━━━
1. Check product status: Catalog → Products
2. Verify visibility: 'Catalog, Search'
3. Reindex: bin/magento indexer:reindex
4. Clear cache: bin/magento cache:flush

Prevention Tips
━━━━━━━━━━━━━━
- Use dedicated test products
- Keep test products enabled
- Run indexers after changes

Quick Fix Commands
━━━━━━━━━━━━━━━━
php bin/magento indexer:reindex
php bin/magento cache:flush

📊 Admin Interface

Overview Page

Run tests with one click
View latest test results
Real-time progress tracking
Quick status indicators

History Page

All past test runs
Filter by status/profile
Sortable columns
Quick access to reports

Report Page

Detailed step results
AI analysis (if failed)
Order details
Timing information
HTTP status codes

⚙️ Configuration Options

Testing Settings

minsar_guardian/testing/test_sku          - Product SKU for testing
minsar_guardian/testing/test_customer_email - Customer email
minsar_guardian/testing/timeout_ms        - Request timeout
minsar_guardian/testing/default_country   - Default country code

AI Settings

minsar_guardian/ai/enabled                - Enable AI analysis
minsar_guardian/ai/provider               - gemini or openai
minsar_guardian/ai/api_key                - API key (encrypted)
minsar_guardian/ai/timeout_ms             - AI request timeout

🔧 CLI Commands

Run Tests

# Smoke test (5 steps)
php bin/magento guardian:run smoke

# Full test (8 steps)
php bin/magento guardian:run full

# With custom environment
php bin/magento guardian:run full --env="Production"

View Results

# List recent runs
php bin/magento guardian:list

# View specific run
php bin/magento guardian:report <run_id>

🗄️ Database Tables

minsar_guardian_run

Stores test run information:

run_id, profile, status, environment
duration_ms, critical_failures
ai_summary (AI analysis)
started_at, completed_at

minsar_guardian_step

Stores individual step results:

step_id, run_id, step_name
status, url, http_code
duration_ms, error_message
response_data (order details JSON)

🎨 Features Showcase

✅ Implemented

✅ Smoke & Full Testing
✅ API Checkout Flow
✅ Customer Account Creation
✅ Order Placement
✅ AI Analysis (Gemini + OpenAI)
✅ Real-time Progress
✅ Admin UI
✅ CLI Commands
✅ Cron Integration
✅ Detailed Reporting
✅ Order Tracking
✅ Quote Tracking

🚧 Future Enhancements

Trends & Analytics
Performance Metrics
Incident Tracking
Email Notifications
Slack Integration
Multi-store Testing

🐛 Troubleshooting

Tests Always Pass

Check if test product SKU exists
Verify product is enabled and in stock
Ensure product has inventory source assigned (MSI)

API Test Fails

Check MSI configuration
Verify payment methods enabled
Verify shipping methods enabled
Check customer creation permissions

AI Not Working

Verify API key is valid
Check quota limits (free tier)
Test API key at provider website
Check error logs: var/log/system.log

Cron Not Running

Verify crontab is configured
Check cron logs: var/log/cron.log
Run manually: php bin/magento cron:run

📝 Best Practices

Test Product Setup

Create dedicated test product (SKU: test-product)
Keep it enabled and in stock
Assign to inventory source
Set simple price ($10-$20)
Don't delete or modify during tests

Testing Schedule

Smoke Tests: Every 15 minutes
Full Tests: Every hour or before deployments
Manual Tests: Before major releases

AI Usage

Enable for production environments
Review AI recommendations before applying
Use AI insights for documentation
Track recurring issues

🔒 Security

API keys are encrypted in database
Test customers use unique emails
No sensitive data sent to AI
Only error messages analyzed
Follows Magento security best practices

📄 License

MIT License - See LICENSE file for details

🤝 Support

GitHub: https://github.com/minsarai/magento2-ai-release-guardian
Issues: https://github.com/minsarai/magento2-ai-release-guardian/issues
Documentation: /docs folder
Email: saifali.openai@gmail.com
Website: https://minsar.vercel.app

🎉 Credits

Built with ❤️ by Minsar Team

Magento 2.4+
Google Gemini AI
OpenAI GPT-4o-mini
PHP 8.1+

Happy Testing! 🚀

minsarai / magento2-ai-release-guardian

Maintainers

Package info

Statistics

Security