README

Description

This package is simple job worker.

Install

{
    "require": {
        "ryunosuke/hellowo": "dev-master"
    }
}

This requires pcntl extension. Also, Windows only works minimally.

Feature

Driver

Driver features:

feature	FileSystem	Gearman	Beanstalk	MySql	PostgreSql
simply	very high	high	high	middle	middle
pull or push(*1)	pull(&inotify)	push	push	pull(&trigger)	pull(&pub/sub)
multi worker server	optional(*2)	yes	yes	yes	yes
not lost to sudden death(*3)	yes (ttr)	no	yes (ttr)	yes (kill)	yes (keepalive)
priority job	yes	yes	yes	yes	yes
delay job	yes	no	yes	yes	yes
managed retry	no	no	no	no	no
unmanaged retry limit	yes	no	no	yes	yes
clustering	no	no	no	optional(*4)	optional(*4)

*1 push is almost real-time, but pull has time lag due to polling
*2 e.g. NFS
*3 Except for FileSystem, TCP keepalive can be enabled to some extent
*4 e.g. Replication, Fabric, NDB

Worker

The recommended process manager is systemd. Also, the basic operation is in series. If you want to run multiple jobs in parallel, you need to launch multiple processes.

SIGALRM is used to implement the timeout, so it cannot be used by the user. When it receives an SIGTERM or SIGINT, it waits for the currently running job until stopping it. Therefore, in some cases, it may take a long time to stop (see TimeoutStopSec of systemd).

Default logging, operation log is written to STDOUT. php error log is written to STDERR. Operation log can be changed by overriding the logger option. php error log uses system default. This can be changed by php.ini or ini_set.

Client

Client is a simple class from which only the request part of the driver is extracted. You can use send or sendBulk for add job.

Demo

Driver: mysql
Parallel: 4
Log: /var/log/hellowo

require root.

ready

sudo su -
WORKDIR=/path/to/hellowo

ready driver

cat << 'EOS' > $WORKDIR/driver.php
<?php
require_once __DIR__ . '/vendor/autoload.php';
return new ryunosuke\hellowo\Driver\MySqlDriver([
    'transport'  => [
        'host'     => '127.0.0.1',
        'username' => 'root',
        'password' => 'password',
    ],
    // job database.table name
    'database'   => 'test',
    'table'      => 't_job',
    'waittime'   => 2.0,
    'waitmode'   => 'php',
    'sharedFile' => '/tmp/jobs.txt',
]);
EOS

ready worker

cat << 'EOS' > $WORKDIR/worker.php
<?php
$driver = require __DIR__ . '/driver.php';
$worker = new ryunosuke\hellowo\Worker(['driver' => $driver]);
$worker->start(function (ryunosuke\hellowo\Message $message) {
    file_put_contents('/var/log/hellowo/receive.log', "$message\n", FILE_APPEND | LOCK_EX);
});
EOS

ready systemd

cat << EOS > /etc/systemd/system/example@.service
[Unit]
After=network.target
PartOf=example.target

[Service]
Type=simple
Environment=SYSTEMD_SERVICE_ID=%i
ExecStartPre=/bin/mkdir -p /var/log/hellowo
ExecStart=/bin/sh -c 'exec /usr/bin/php $WORKDIR/worker.php 1>/var/log/hellowo/stdout-%i.log 2>/var/log/hellowo/stderr-%i.log'
TimeoutStopSec=90s
Restart=always

[Install]
EOS

cat << EOS > /etc/systemd/system/example.target
[Unit]
Wants=example@1.service
Wants=example@2.service
Wants=example@3.service
Wants=example@4.service

[Install]
WantedBy=multi-user.target
EOS

systemctl daemon-reload
systemctl restart example.target
systemctl status example@*

ready client

cat << 'EOS' > $WORKDIR/client.php
<?php
$driver = require __DIR__ . '/driver.php';
$client = new ryunosuke\hellowo\Client(['driver' => $driver]);
$client->sendBulk(array_map(fn($v) => "data-$v", range(1, 99)));
EOS

php client.php

output log

cat /var/log/hellowo/stdout-*.log
[Y-m-dTH:i:s.v][1045984] ...
[Y-m-dTH:i:s.v][1045984] ...
[Y-m-dTH:i:s.v][1045984] ...

cat -n /var/log/hellowo/receive.log
data-8
data-17
data-15
...
data-98
data-96
data-99

License

MIT

Release

Versioning is romantic versioning(no semantic versioning).

major: large BC break. e.g. change architecture, package, class etc
minor: small BC break. e.g. change arguments, return type etc
patch: no BC break. e.g. fix bug, add optional arguments, code format etc

x.y.z

API の除去
- protected で不要なメソッドを隠す意図の設計だったが足枷になってきている

1.2.3

Merge tag 'v1.1.12'

1.2.2

Merge tag 'v1.1.11'

1.2.1

[fixbug] client で文字列以外を send するときに IDE エラーが出る

1.2.0

[fixbug] gearman に未来 job を登録するとその間無限ループする
[*feature] job 側にも timeout を持たせる
[*feature] delay に予定時刻を入れられる機能
[*feature] 失敗したジョブを保存する機能
[feature] sendBulk を追加
[*change] sendJson 廃止
[*change] notify は Client の責務とする
[*change] job データは json で包む
[*change] wait 系のデフォルト waittime を 10 に変更
[*change] drop deprecated
[tests] 意味の分からないコードがあったので除去

1.1.12

[feature] 絶え間なくジョブが実行され続けたら警告を出す
[feature] 結果の如何にかかわらず job を検知して終了した場合のイベント finish を追加

1.1.11

[change] ブランチ切り替えがしんどいので docker-compose の変更をバックポート
[feature] standup イベントを追加
[fixbug] readonly 状態では SELECT FOR UPDATE を投げられない
[fixbug] エラー時は正常終了ではなく異常終了で systemd の再起動を促す

1.1.10

[feature] breather イベントを追加
[change] rework Gearman
[feature] cancel 実装
[refactor] 例外の構造変更と UnsupportedException を追加
[fixbug] sha1 は無駄だし一意ではない

1.1.9

[feature] prepare は1回で十分
[fixbug] systemd で起動しない不具合
[fixbug] EchoLogger の引数漏れ

1.1.8

[feature] EchoLogger にレベルフィルタを実装
[feature] ジョブ候補をファイルキャッシュする機能
[feature] ジョブを分散させるために starttime を追加
[fixbug] 無駄な行を取りすぎている
[fixbug] active 側で isStandby が高頻度で呼ばれている

1.1.7

[fixbug] 並列数が1になることがある不具合

1.1.6

[feature] start 時に work を与えられる機能

1.1.5

[feature] PostgreSqlDriver
[fixbug] 初回実行時に setup されない不具合
[fixbug] 一部のドライバーにミリ秒を与えるとエラーになる
[fixbug] heartbeat 秒応答がないホストに対して heartbeat 秒 ping を待機している
[feature] delay がある場合は notify しても無駄
[feature] MySql のミリ秒対応

1.1.4

[fixbug] USR1 以外でも async をキャンセルしていた
[fixbug] select と sleep で使用されるインデックスが異なる可能性がある

1.1.3

[feature] JSON を組み込みで実装
[refactor] API の整理
[refactor] ext のパスを変更
[refactor] Listener の階層を Logger と合わせる

1.1.2

[feature] リトライ回数を実装
[fixbug] standby 状態で setup して死ぬ不具合を修正

1.1.1

[feature] added restart trigger

1.1.0

[*change] select+done+retry -> generator
[change] log format
[fixbug] no keeps connecting on server gone away

1.0.1

[feature] check writable mode
[fixbug] transaction may not be closed
[fixbug] fix miss (PROCEDURE -> FUNCTION)

1.0.0

[feature] added logger logs when Throwable
[refactor] changed nullable to notnull

0.2.0

[fixbug] fixed "Commands out of sync" when receive USR1
[fixbug] deleted IF EXISTS from mysql driver
[feature] added listen cycle event

0.1.0

[change] move notify to notifyLocal

0.0.0

publish

ryunosuke / hellowo

Maintainers

Details