xmlshop / laravel-queue-monitor
Queue Monitoring for Laravel Database Job Queue
Fund package maintenance!
xmlshop
Installs: 1 805
Dependents: 0
Suggesters: 0
Security: 0
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Requires
- php: ^8.0
- ext-json: *
- ext-mbstring: *
- ext-posix: *
- gpressutto5/laravel-slack: ^2.0
- illuminate/database: ^5.5|^6.0|^7.0|^8.0|^9.0
- illuminate/queue: ^5.5|^6.0|^7.0|^8.0|^9.0
- illuminate/support: ^5.5|^6.0|^7.0|^8.0|^9.0
- lorisleiva/cron-translator: ^0.4
- nesbot/carbon: ^2.0
- webpatser/laravel-uuid: ^4.0
Requires (Dev)
- laravel/framework: ^5.5|^6.0|^7.0|^8.0|^9.0
- mockery/mockery: ^1.3.2
- orchestra/testbench: ^3.8|^4.0|^5.0|^6.0
- phpstan/phpstan: ^0.12.99|^1.0
- phpunit/phpunit: ^8.0|^9.0
README
For Laravel framework
This package offers monitoring like Laravel Horizon for database queue.
This package initially forked from Laravel-Queue-Monitor.
Features
- Monitor jobs like Laravel Horizon for any queue
- Handle failing jobs with storing exception
- Monitor job progress
- Get an estimated time remaining for a job
- Store additional data for a job monitoring
Installation
composer require xmlshop/queue-monitor
Configuration
Copy configuration & migration to your project:
php artisan vendor:publish --provider="xmlshop\QueueMonitor\Providers\MonitorProvider" --tag=config --tag=migrations
Migrate the Queue Monitoring table. The table name can be configured in the config file or via the published migration.
php artisan migrate
Scheduler
class Kernel extends ConsoleKernel
{
#...
protected function schedule(Schedule $schedule)
{
#...
$schedule->command('queue-monitor:aggregate-queues-sizes')->everyMinute();
$schedule->command('queue-monitor:clean-up')->dailyAt('01:23');
$schedule->command('queue-monitor:listener')->everyMinute();
$schedule->command('monitor:sync-scheduler')->daily();
$schedule->command('monitor:pid-checker')->everyMinute(); #for each node, if >1
#...
}
}
After the listener automatically will be launched queue-monitor:listener
. It might be disabled in configuration or by command
#php artisan queue-monitor:listener disable {hours} php artisan queue-monitor:listener disable 24 #disables alert-launcher for a day. By default 1 hour php artisan queue-monitor:listener enable #enables that back
Application Error handler have to be replaced at your application Please add custom error handler:
final class Handler extends \Illuminate\Foundation\Exceptions\Handler implements \Illuminate\Contracts\Debug\ExceptionHandler { /** * Render an exception to the console. * * @param \Symfony\Component\Console\Output\OutputInterface $output * @param \Throwable $e * @return void */ final public function renderForConsole($output, Throwable $e) { $command_name = null; foreach (request()->server('argv') as $arg) { if (Str::contains($arg, ':')) { $command_name = $arg; break; } } if (null !== $command_name && class_exists('xmlshop\QueueMonitor\Services\CLIFailureHandler')) { app('xmlshop\QueueMonitor\Services\CLIFailureHandler')->handle($command_name, $e); } parent::renderForConsole($output, $e); } }
And please register that in ApplicationProvider
$this->app->bind(\Illuminate\Contracts\Debug\ExceptionHandler::class, \App\Exceptions\Handler::class);
Alert function
- Listener looks into database in the
x_queue_monitoring_queue_sizes
table and comparing current amount with amount mentioned in fieldalert_threshold
. If exceed - alert. - Listener looks into database in the
x_queue_monitoring
table and comparing several metrics (pending time, execution time, etc.) - You can manage exceptions for each Job. Including ignore alert.
Usage
To monitor a job, simply add the xmlshop\QueueMonitor\Traits\IsMonitored
Trait.
use Illuminate\Bus\Queueable; use Illuminate\Queue\SerializesModels; use Illuminate\Queue\InteractsWithQueue; use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Foundation\Bus\Dispatchable; use xmlshop\QueueMonitor\Traits\IsMonitored; // <--- class ExampleJob implements ShouldQueue { use Dispatchable; use InteractsWithQueue; use Queueable; use SerializesModels; use IsMonitored; // <--- }
Important! You need to implement the Illuminate\Contracts\Queue\ShouldQueue
interface to your job class. Otherwise, Laravel framework will not dispatch any events containing status information for monitoring the job.
UI
You can enable the optional UI routes by calling Route::queueMonitor()
inside your route file, similar to the official ui scaffolding.
Route::prefix('monitor')->group(function () { Route::queueMonitor(); });
Routes
Route | Action |
---|---|
/monitor |
Show the jobs table |
See the configuration files for more information.
Extended usage
Progress
You can set a progress value (0-100) to get an estimation of a job progression.
use Illuminate\Contracts\Queue\ShouldQueue; use xmlshop\QueueMonitor\Traits\IsMonitored; class ExampleJob implements ShouldQueue { use IsMonitored; public function handle() { $this->queueProgress(0); // Do something... $this->queueProgress(50); // Do something... $this->queueProgress(100); } }
Chunk progress
A common scenario for a job is iterating through large collections.
This example job loops through a large amount of users and updates it's progress value with each chunk iteration.
use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Database\Eloquent\Collection; use xmlshop\QueueMonitor\Traits\IsMonitored; class ChunkJob implements ShouldQueue { use IsMonitored; public function handle() { $usersCount = User::count(); $perChunk = 50; User::query() ->chunk($perChunk, function (Collection $users) use ($perChunk, $usersCount) { $this->queueProgressChunk($usersCount‚ $perChunk); foreach ($users as $user) { // ... } }); } }
Progress cooldown
To avoid flooding the database with rapidly repeating update queries, you can set override the progressCooldown
method and specify a length in seconds to wait before each progress update is written to the database. Notice that cooldown will always be ignore for the values 0, 25, 50, 75 and 100.
use Illuminate\Contracts\Queue\ShouldQueue; use xmlshop\QueueMonitor\Traits\IsMonitored; class LazyJob implements ShouldQueue { use IsMonitored; public function progressCooldown(): int { return 10; // Wait 10 seconds between each progress update } }
Custom data
This package also allows setting custom data in array syntax on the monitoring model.
use Illuminate\Contracts\Queue\ShouldQueue; use xmlshop\QueueMonitor\Traits\IsMonitored; class CustomDataJob implements ShouldQueue { use IsMonitored; public function handle() { $this->queueData(['foo' => 'Bar']); // WARNING! This is overriding the monitoring data $this->queueData(['bar' => 'Foo']); // To preserve previous data and merge the given payload, set the $merge parameter true $this->queueData(['bar' => 'Foo'], true); } }
In order to show custom data on UI you need to add this line under config/monitor.php
'ui' => [ ... 'show_custom_data' => true, ... ]
Only keep failed jobs
You can override the keepMonitorOnSuccess()
method to only store failed monitor entries of an executed job. This can be used if you only want to keep failed monitors for jobs that are frequently executed but worth to monitor. Alternatively you can use Laravel's built in failed_jobs
table.
use Illuminate\Contracts\Queue\ShouldQueue; use xmlshop\QueueMonitor\Traits\IsMonitored; class FrequentSucceedingJob implements ShouldQueue { use IsMonitored; public static function keepMonitorOnSuccess(): bool { return false; } }
Retrieve processed Jobs
use xmlshop\QueueMonitor\Models\MonitorQueue; $job = MonitorQueue::query()->first(); // Check the current state of a job $job->isFinished(); $job->hasFailed(); $job->hasSucceeded(); // If the job is still running, get the estimated seconds remaining // Notice: This requires a progress to be set $job->getRemainingSeconds(); $job->getRemainingInterval(); // Carbon\CarbonInterval // Retrieve any data that has been set while execution $job->getData(); // Get the base name of the executed job $job->getBasename();
Model Scopes
use xmlshop\QueueMonitor\Models\MonitorQueue; // Filter by Status MonitorQueue::failed(); MonitorQueue::succeeded(); // Filter by Date MonitorQueue::lastHour(); MonitorQueue::today(); // Chain Scopes MonitorQueue::today()->failed();