README

Stop guessing how many queue workers you need. This package automatically optimizes worker counts based on job type (CPU-intensive, API calls, file downloads) and your server's resources.

Why You Need This

Running 10 workers for everything? You're wasting resources or creating bottlenecks.

Image downloads saturate your network with too many workers
API calls queue up with too few concurrent workers
CPU jobs leave cores idle or cause context-switching overhead

Are you running your application on different machines? Different CPU core configurations, changing network bandwidth? With this package you can optimize it automatically and fine tune it for best results.

Quick Start

composer require adamczykpiotr/laravel-balanced-queues
php artisan vendor:publish --tag="laravel-balanced-queues-config"

Tag your jobs with the right workload type using traits:

use AdamczykPiotr\LaravelBalancedQueues\Traits\HasBalancedQueues;

class ScrapeVideosJob implements ShouldQueue
{
    public function __construct() {
        $this->onHighNetworkBandwidthUsageQueu();
    }
}

class CallApiJob implements ShouldQueue
{
    public function __construct() {
        $this->onHighNetworkRequestUsageQueue();
    }
}

class ProcessBigDataJob implements ShouldQueue
{
    public function __construct() {
        $this->onHighCpuUsageQueue();
    }
}

class ProcessSmallerDataJob implements ShouldQueue
{
    public function __construct() {
        $this->onMediumCpuUsageQueue();
    }
}

Run your queues:

php artisan queue:run-balanced

Done. The package spawns all the queues with adjusted worker counts for each job type.

Available helpers

Helper method	Workload Type	Use Case
`onHighCpuUsageQueue`	CPU_HIGH	Large file processing ~MB/GB
`onMediumCpuUsageQueue`	CPU_MEDIUM	PDF generation, smaller file parsing (~KB/MB)
`onHighNetworkRequestUsageQueue`	NETWORK_HIGH_BANDWIDTH	Large file downloads (~ >10MB)
`onHighNetworkBandwidthUsageQueue`	NETWORK_HIGH_REQUESTS	API calls, webhooks

How It Works

By default, CPU_HIGH usage assumes 100% CPU utilisation for a single job and 25% for CPU_MEDIUM. Jobs on NETWORK_HIGH_BANDWIDTH are best suited for downloading large files that completely saturate network bandwidth whereas NETWORK_HIGH_REQUESTS are in most cases are bottleneck by neither network nor cpu and are offloaded on a queue simply to improve UX.

If the default configuration doesn't fully utilize your machine, adjust the following default configuration in config/balanced-queues.php:

use AdamczykPiotr\LaravelBalancedQueues\Enums\JobWorkloadType;

$CPU_CORES = CpuCoreConfigurationResolver::CPU_CORES;

return [
    'queues' => [
        JobWorkloadType::DEFAULT->value => 1,
        
        JobWorkloadType::CPU_HIGH->value => $CPU_CORES,
        JobWorkloadType::CPU_MEDIUM->value => 4 * $CPU_CORES,
        JobWorkloadType::NETWORK_HIGH_BANDWIDTH->value => 5,
        JobWorkloadType::NETWORK_HIGH_REQUESTS->value => 50,
    ],
];

The default configuration aims to ensure best results across wide range of cases. In order to get full benefits of this approach, it's highly recommended to fine-tune the configuration to your specific use-case. The clue to optimisation is to ensure a single queue can fully saturate the machine:

Dispatching CPU_CORES jobs to CPU_HIGH queue should result in ~100% CPU usage
Dispatching 4 * CPU_CORES jobs to CPU_MEDIUM queue should result in ~100% CPU usage
Dispatching 5 jobs to NETWORK_HIGH_BANDWIDTH queue should result in full network saturation
Dispatching 50 jobs to NETWORK_HIGH_REQUESTS should result in jobs being finished as soon as possible (no real bottleneck)

In case more than one queue is fully saturated, OS scheduler will balance processes CPU time accordingly.

Advanced Configuration

Custom Queue Names

'queues' => [
    'ml-training' => 2 * CPU_CORES,
    'notifications' => 10,
    'web-scraping' => 25,
],

VM / Docker CPU Limits

'defaults' => [
    'cpu_core_count' => 4,  // Override auto-detected core count
],

Commands

# Run queue workers
php artisan queue:run-balanced

# Run in background
php artisan queue:run-balanced --background

# Get path to generated config and run supervisor manually
php artisan queue:run-balanced --path-only

Docker container entrypoint example

#!/bin/sh
set -e

SUPERVISORD_CONFIG_PATH=$(php artisan queue:run-balanced --path-only)
exec /usr/bin/supervisord -n -c "$SUPERVISORD_CONFIG_PATH"

Requirements

PHP 8.3+
Laravel 11.x or 12.x
Supervisor

Testing

composer test

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

License

The MIT License (MIT). Please see License File for more information.

adamczykpiotr / laravel-balanced-queues

Maintainers

Details