ryunosuke / hellowo
simple job worker
Requires
- php: >=7.4
- psr/log: *
Requires (Dev)
- kicken/gearman-php: 1.*
- pda/pheanstalk: 4.*
- php-amqplib/php-amqplib: 3.*
- phpunit/phpunit: 9.*
- ryunosuke/phpunit-extension: 3.*
- symfony/process: 5.*
Suggests
- ext-gearman: Required depending on the driver
- ext-inotify: Required depending on the driver
- ext-mysqli: Required depending on the driver
- ext-pcntl: Required on substantially
- ext-pgsql: Required depending on the driver
- pda/pheanstalk: Required depending on the driver
README
Description
This package is simple job worker.
Install
{ "require": { "ryunosuke/hellowo": "dev-master" } }
This requires pcntl
extension. Also, Windows only works minimally.
Feature
Driver
Driver features:
feature | FileSystem | Gearman | Beanstalk | MySql | PostgreSql |
---|---|---|---|---|---|
simply | very high | high | high | middle | middle |
pull or push(*1) | pull(&inotify) | push | push | pull(&trigger) | pull(&pub/sub) |
multi worker server | optional(*2) | yes | yes | yes | yes |
not lost to sudden death(*3) | yes (ttr) | no | yes (ttr) | yes (kill) | yes (keepalive) |
priority job | yes | yes | yes | yes | yes |
delay job | yes | no | yes | yes | yes |
managed retry | no | no | no | no | no |
unmanaged retry limit | yes | no | no | yes | yes |
clustering | no | no | no | optional(*4) | optional(*4) |
- *1 push is almost real-time, but pull has time lag due to polling
- *2 e.g. NFS
- *3 Except for FileSystem, TCP keepalive can be enabled to some extent
- *4 e.g. Replication, Fabric, NDB
Worker
The recommended process manager is systemd. Also, the basic operation is in series. If you want to run multiple jobs in parallel, you need to launch multiple processes.
SIGALRM is used to implement the timeout, so it cannot be used by the user. When it receives an SIGTERM or SIGINT, it waits for the currently running job until stopping it. Therefore, in some cases, it may take a long time to stop (see TimeoutStopSec of systemd).
Default logging, operation log is written to STDOUT. php error log is written to STDERR.
Operation log can be changed by overriding the logger
option.
php error log uses system default. This can be changed by php.ini or ini_set.
Client
Client is a simple class from which only the request part of the driver is extracted.
You can use send
or sendBulk
for add job.
Demo
- Driver: mysql
- Parallel: 4
- Log: /var/log/hellowo
require root.
ready
sudo su - WORKDIR=/path/to/hellowo
ready driver
cat << 'EOS' > $WORKDIR/driver.php <?php require_once __DIR__ . '/vendor/autoload.php'; return new ryunosuke\hellowo\Driver\MySqlDriver([ 'transport' => [ 'host' => '127.0.0.1', 'username' => 'root', 'password' => 'password', ], // job database.table name 'database' => 'test', 'table' => 't_job', 'waittime' => 2.0, 'waitmode' => 'php', 'sharedFile' => '/tmp/jobs.txt', ]); EOS
ready worker
cat << 'EOS' > $WORKDIR/worker.php <?php $driver = require __DIR__ . '/driver.php'; $worker = new ryunosuke\hellowo\Worker(['driver' => $driver]); $worker->start(function (ryunosuke\hellowo\Message $message) { file_put_contents('/var/log/hellowo/receive.log', "$message\n", FILE_APPEND | LOCK_EX); }); EOS
ready systemd
cat << EOS > /etc/systemd/system/example@.service [Unit] After=network.target PartOf=example.target [Service] Type=simple Environment=SYSTEMD_SERVICE_ID=%i ExecStartPre=/bin/mkdir -p /var/log/hellowo ExecStart=/bin/sh -c 'exec /usr/bin/php $WORKDIR/worker.php 1>/var/log/hellowo/stdout-%i.log 2>/var/log/hellowo/stderr-%i.log' TimeoutStopSec=90s Restart=always [Install] EOS cat << EOS > /etc/systemd/system/example.target [Unit] Wants=example@1.service Wants=example@2.service Wants=example@3.service Wants=example@4.service [Install] WantedBy=multi-user.target EOS systemctl daemon-reload systemctl restart example.target systemctl status example@*
ready client
cat << 'EOS' > $WORKDIR/client.php <?php $driver = require __DIR__ . '/driver.php'; $client = new ryunosuke\hellowo\Client(['driver' => $driver]); $client->sendBulk(array_map(fn($v) => "data-$v", range(1, 99))); EOS php client.php
output log
cat /var/log/hellowo/stdout-*.log
[Y-m-dTH:i:s.v][1045984] ...
[Y-m-dTH:i:s.v][1045984] ...
[Y-m-dTH:i:s.v][1045984] ...
cat -n /var/log/hellowo/receive.log
data-8
data-17
data-15
...
data-98
data-96
data-99
License
MIT
Release
Versioning is romantic versioning(no semantic versioning).
- major: large BC break. e.g. change architecture, package, class etc
- minor: small BC break. e.g. change arguments, return type etc
- patch: no BC break. e.g. fix bug, add optional arguments, code format etc
x.y.z
- API の除去
- protected で不要なメソッドを隠す意図の設計だったが足枷になってきている
1.2.1
- [fixbug] client で文字列以外を send するときに IDE エラーが出る
1.2.0
- [fixbug] gearman に未来 job を登録するとその間無限ループする
- [*feature] job 側にも timeout を持たせる
- [*feature] delay に予定時刻を入れられる機能
- [*feature] 失敗したジョブを保存する機能
- [feature] sendBulk を追加
- [*change] sendJson 廃止
- [*change] notify は Client の責務とする
- [*change] job データは json で包む
- [*change] wait 系のデフォルト waittime を 10 に変更
- [*change] drop deprecated
- [tests] 意味の分からないコードがあったので除去
1.1.10
- [feature] breather イベントを追加
- [change] rework Gearman
- [feature] cancel 実装
- [refactor] 例外の構造変更と UnsupportedException を追加
- [fixbug] sha1 は無駄だし一意ではない
1.1.9
- [feature] prepare は1回で十分
- [fixbug] systemd で起動しない不具合
- [fixbug] EchoLogger の引数漏れ
1.1.8
- [feature] EchoLogger にレベルフィルタを実装
- [feature] ジョブ候補をファイルキャッシュする機能
- [feature] ジョブを分散させるために starttime を追加
- [fixbug] 無駄な行を取りすぎている
- [fixbug] active 側で isStandby が高頻度で呼ばれている
1.1.7
- [fixbug] 並列数が1になることがある不具合
1.1.6
- [feature] start 時に work を与えられる機能
1.1.5
- [feature] PostgreSqlDriver
- [fixbug] 初回実行時に setup されない不具合
- [fixbug] 一部のドライバーにミリ秒を与えるとエラーになる
- [fixbug] heartbeat 秒応答がないホストに対して heartbeat 秒 ping を待機している
- [feature] delay がある場合は notify しても無駄
- [feature] MySql のミリ秒対応
1.1.4
- [fixbug] USR1 以外でも async をキャンセルしていた
- [fixbug] select と sleep で使用されるインデックスが異なる可能性がある
1.1.3
- [feature] JSON を組み込みで実装
- [refactor] API の整理
- [refactor] ext のパスを変更
- [refactor] Listener の階層を Logger と合わせる
1.1.2
- [feature] リトライ回数を実装
- [fixbug] standby 状態で setup して死ぬ不具合を修正
1.1.1
- [feature] added restart trigger
1.1.0
- [*change] select+done+retry -> generator
- [change] log format
- [fixbug] no keeps connecting on server gone away
1.0.1
- [feature] check writable mode
- [fixbug] transaction may not be closed
- [fixbug] fix miss (PROCEDURE -> FUNCTION)
1.0.0
- [feature] added logger logs when Throwable
- [refactor] changed nullable to notnull
0.2.0
- [fixbug] fixed "Commands out of sync" when receive USR1
- [fixbug] deleted IF EXISTS from mysql driver
- [feature] added listen cycle event
0.1.0
- [change] move notify to notifyLocal
0.0.0
- publish