Gearman auto-scaling workers for PHP

0.1 2013-10-11 16:20 UTC


Gearman auto-scaling workers for PHP.

Allows workers to be automatically spawned based on configuration and limited by system load. Workers automatically die when no jobs to process.

This project is at early stages but comments/input welcome from anyone who finds this useful. Get in touch!

To run the examples

  • cd examples
  • ./
  • This will start monitoring the queue and show what workers are registered...
  • php testLongRunningBackgroundJobs.php
  • This will create 100 jobs that sleep a random amount of time...
  • php testLoadLimitingJobs.php
  • This will create CPU intensive jobs to see the workers getting throttled based on CPU load...
  • Tail /tmp/workers.log to see the output of the workers whilst they are running.

Use it in your own project via Composer?

ToDo List

  • Need to better document how to use in your own app
  • Obviously deal with all the various //TODO's in the code...
  • Support a lock file to tell the app when to prevent spawning of any new workers (e.g. when deploying a new version of a host app and you don't want workers running whilst this happens)
  • Add tests to prove the auto-scaling and closing down of workers
  • Try various callbacks to the client and see how they work
  • Try non-background jobs and how clients should deal with these
  • Work out how best to trap error scenarios in the workers (e.g. worker dies, exceptions etc)
  • Whether we should log the submitted jobs somewhere so we can see them (Memcache? SQLIte? Mongo?)
  • RESTful client for submitting jobs?
  • Logging (allow passing in a file handle or filename or MonoLog instance or whatever... Maybe defaults to using MonoLog internally if no logger specified)