ph4r05/laravel-queue-database-ph4

Laravel Database queue driver with optimistic locking

v0.1.6 2020-11-01 15:54 UTC

Requires

• php: ^7.2
• illuminate/database: 6.*
• illuminate/queue: 6.*
• illuminate/support: ^6.2

Requires (Dev)

• ext-json: *
• mockery/mockery: ~1.0
• phpunit/phpunit: ^7.0

MIT

• Dusan Klinec (ph4r05)

README

Laravel database queue implementation using optimistic locking. Increases concurrency, eliminates deadlocks.

Installation

1. Install this package via composer using:

composer require ph4r05/laravel-queue-database-ph4

1. Create job table

php artisan queue_ph4:table
php artisan migrate

1. Queue configuration

<?php
// config/queue.php

return [
    'database_ph4' => [
        'driver' => 'database_ph4',
        'table' => 'jobs_ph4',
        'queue' => 'default',
        'retry_after' => 4,
        'num_workers' => 1,
        'window_strategy' => 1,
    ],
];

• The num_workers should correspond to the number of workers processing jobs in the queue.
• window_strategy.
  • 0 means worker selects one available job for processing. Smaller throughput, job ordering is preserved as with pessimistic locking.
  • 1 means workers will select num_workers next available jobs and picks one at random. Higher throughput with slight job reordering (for more info please refer to the blog)

To use the optimistic locking you can now change your .env:

QUEUE_DRIVER=database_ph4

Usage

Once you completed the configuration you can use Laravel Queue API. If you used other queue drivers you do not need to change anything else. If you do not know how to use Queue API, please refer to the official Laravel documentation: http://laravel.com/docs/queues

Testing

Run the tests with:

vendor/bin/phpunit

Blog post about optimistic locking

https://ph4r05.deadcode.me/blog/2017/12/23/laravel-queues-optimization.html

Benefits:

• No need for explicit transactions. Single query auto-commit transactions are OK.
• No DB level locking, thus no deadlocks. Works also with databases without deadlock detection (older MySQL).
• Job executed exactly once (as opposed to pessimistic default DB locking)
• High throughput.
• Tested with MySQL, PostgreSQL, Sqlite.

Cons:

• Job ordering can be slightly shifted with multiple workers (reordering 0-70 in 10 000 jobs)

Contribution

You can contribute to this package by discovering bugs and opening issues. Please, add to which version of package you create pull request or issue.

Donations

Thank you for all your support!

Monero:

87iMuZKqgBZbxjvjJaieTqLBsW3VKtkiuRXL2arcr8eiL4eK8kFi4QbaXCXGgmNWYp5Linpd9pj5McFZ8SQevkenGuZWMCT

Bitcoin:

17hzavLCXqavzmWEEjtX54VeCJVmbrHZQC