README

Introduction

Disqontrol is a user-friendly framework for running background jobs in any language.

Imagine your user has just uploaded a large picture and you want to create four other pictures from it. But the pictures are not needed immediately and you don't want to keep the user waiting for the server response.

Instead of processing the picture right now and delaying the response to the user, you create a background job and respond to the user without waiting for the result. The background job will be picked up by another program that will process the picture without delaying your main application.

There are five steps in this process:

Creating a background job
Storing it
Picking it up from the storage
Processing it
Handling failed jobs

Where does Disqontrol come in?

Ad 1. Disqontrol helps you create a new background job, either from PHP by using the Disqontrol producer, or from a command line. You can also add the job with your own preferred library.

Adding a job in PHP:

$producer->add(new Job('example-pic.png', 'resize-pic'));

Adding a job from the command line:

disqontrol addjob resize-pic '"example-pic.png"'

Ad 2. The job is stored in Disque, a specialized job queue created by the author of Redis.

Ad 3. The job is picked up by a Disqontrol consumer, a long-running process that listens to Disque, picks up jobs coming in and decides who should process them.

Configuration:

queues:
    'resize-pic':
        worker:
            type: isolated_php_worker
            name: 'resize_pic_worker'

Ad 4. The code for processing a job - the worker - is ultimately your responsibility, but Disqontrol provides a helpful scaffolding to write your own workers, so you can just concentrate on the processing itself. Just follow the interface.

The simplest PHP worker for resizing pictures:

class ResizePicWorker implements WorkerInterface
{
    public function process(JobInterface $job)
    {
        $picFileName = $job->getBody();
        // Concrete implementation left out
        return $this->resize($picFileName);
    }
}

Ad 5. It's important to decide what to do with failed jobs. Even though the jobs are running in the background, some of them must be processed, some make only sense for a certain amount of time and some of them are optional. Disqontrol helps you configure failure handling for each job type individually, as well as set a reasonable default behavior.

Features

Disqontrol is written in PHP and uses Disque as its underlying job queue.

With Disqontrol you get the following features:

Process jobs in any programming language
Schedule jobs and run them regularly
Write PHP workers with all the scaffolding already in place
Configure the lifecycle of a job in detail
Handle failures robustly
Process new jobs immediately for development and debugging
Run multiple jobs from one queue in parallel
Switch automatically to the best Disque node

Workers can be called via a command line and can therefore be written in any language. The library also provides convenient scaffolding for workers written in PHP.

The goal of Disqontrol is to be a user-friendly, clean and robust tool.

Disqontrol follows semantic versioning.

Documentation

Why Disque?

There are many general message queues - RabbitMQ, ZeroMQ, IronMQ, Amazon SQS. Redis also has a queue functionality. Some libraries even offer databases as queue backends.

General message queues treat a job just like a message and the job workflow must be simulated in libraries.

Each queue implementation has different features and weaknesses. Queue libraries usually allow you to choose from different backends and try to make up for missing features in the code. This leads to the problem of a common denominator, unnecessary complexity and doesn't allow one to use the features of one queue fully.

That's why we have decided to use just one queue - Disque - written by the developer of Redis.

For the above mentioned reasons, we have no plans to support other queues. Instead we want to fully use all features of Disque.

Roadmap

The following features are planned for the 1.0 release:

Consumers pick up jobs and call workers
Supervisor keeps Consumers alive
Consumers quit gracefully
Isolated and inline PHP workers
Command-line workers
Failing jobs re-enqueued with exponential backoff
Disqontrol has access to a PHP app environment
Scheduled repeated jobs
Synchronous mode
HTTP workers (a job can be processed by an HTTP call)
Autoscaling of consumers ("just" the algorithm is missing, see Disqontrol\Consumer\Autoscale\PredictiveAutoscaling)
Using custom failure strategies (see Disqontrol\Dispatcher\JobDispatcher and Disqontrol\Dispatcher\Failure\FailureStrategyCollection)
The --config parameter (for installations not using the bootstrap file, eg. teams not writing PHP code)

Change log

Please see the CHANGELOG for more information what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Credits

All Contributors
Big thanks to Antirez for Disque
Big thanks to Mariano for Disque-php
Thanks to Martin Patera for kicking off the work

Alternatives

For similar libraries in PHP, have a look at

For libraries in other languages, have a look at

Resque in Ruby
Celery in Python

License

The MIT License (MIT). Please see the License for more information.

disqontrol / disqontrol

Maintainers

Details