geoffroy-aubry / supervisor
Oversee script execution, recording stdout, stderr and exit code with timestamping, and ensure email notifications will be sent (on start, success, warning or error).
Installs: 6 777
Dependents: 1
Suggesters: 0
Security: 0
Stars: 8
Watchers: 2
Forks: 1
Open Issues: 0
Requires
Requires (Dev)
- php: >=5.3.3
- geoffroy-aubry/helpers: 1.*
- phpunit/phpunit: >=3.7
README
Estimated code coverage: 87% (553 of 638 lines).
Oversee script execution, recording stdout
, stderr
and exit code with timestamping,
and ensure email notifications will be sent (on startup, success, warning or error)… plus many other features.
Technologies : Supervisor is in Bash, only unit tests are in PHP.
Supported OS : Debian/Ubuntu Linux, FreeBSD/OS X.
Table of Contents
- Features
- Requirements
- Usage
- Installation
- Copyrights & licensing
- Change log
- Continuous integration
- Git branching model
Features
General
- Oversee script execution, recording
stdout
,stderr
and exit code - Prefix each line of
stdout
with timestamps with hundredths of seconds. - An execution ID uniquely identify each script execution.
Used in:
supervisor.info.log
,supervisor.error.log
,<script>_<exec_id>.info.log
and<script>_<exec_id>.error.log
. - Possibility to block parallel script execution (only on Debian/Ubuntu Linux). Useful if an execution takes too long…
- You can specify a configuration file to load in addition to the default one (overloading).
- Entirely configurable tag system allowing executed scripts to dynamically specify via
stdout
warning, mailto, mail attachment, instigator… - Automatic log archiving mechanism.
- Display a summary of supervisor's activity during last days, including final status per day and per supervised script.
- Handle CSV output, with definition of field separator, field enclosure and number of the field to watch in configuration file.
- Add proper signal handling and interrupts:
SIGHUP
,SIGINT
,SIGQUIT
andSIGTERM
now transmitted to supervised script.
Notifications
- Ensure email notifications will be sent on following events (configurable): startup, success, warning or error. Catch exit code, fatal error, exceptions…
- All mail contains content of
stdout
andstderr
of executed script in Gzip attachment. - Mails are fully customizable. Just provide your file on command line.
- Possibility to inject multiple external parameters on command line into customized emails.
- Supervisor itself is monitoring by another process, sending critical email notifications using an exponential backoff algorithm in minute increments.
Ergonomics
- During supervision and in addition of all log files, Supervisor displays in real time outputs, warning and errors of overseen script, adding timestamp and respecting both colors and indentation. Name of all log files are printed too.
Code quality
- Bash code executed with following directives:
set -o nounset
: treat unset variables and parameters other than the special parameters@
or*
as an error when performing parameter expansion. An error message will be written to the standard error, and a non-interactive shell will exit.set -o pipefail
: The return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully.
- A lot of unit tests (via PHPUnit through a PHP wrapper) covering almost the whole application.
Requirements
-
Bash v4 (2009) and above
-
mutt, to send email notifications (
apt-get install mutt
, orbrew install mutt
) -
Unit tests require PHP >= 5.3.3 and Xdebug
-
On FreeBSD/OS X only, you need to install these GNU tools:
$ brew install coreutils $ brew install gnu-sed $ brew install gawk
Tested on Debian/Ubuntu Linux and FreeBSD/OS X.
Usage
Help on command prompt
Displayed by:
$ supervisor.sh [-h|--help]
Text version
Description
Oversee script execution, recording stdout, stderr and exit code with timestamping,
and ensure email notifications will be sent (on start, success, warning or error).
Usage
supervisor [OPTION]… <script-path> [<script-parameters>]
supervisor [-c <conf-file>] --archive=<min-days>
supervisor [-c <conf-file>] --monitor
supervisor [-c <conf-file>] --summarize
Options
--archive=<min-days>
Archive in Gzip supervisor's logs older than <min-days>.
-c <conf-file>, --conf=<conf-file>
Specify a configuration file to load in addition to the default one,
namely '/conf/supervisor-dist.sh'.
--customized-mails=<file>
Path to a Bash script customizing sent mails by redefining some of
the sendMailOn[Init|Success|Warning|Error]() functions.
See --param option.
--exec-id=<string>
Allow to force execution id, used in mails and name of logs.
By default: YYYYMMDDHHIISS_XXXXX, where X are random digits.
--extra-param-mode=[only-value|with-name|none]
Two extra parameters are added to the end of <script-parameters>: a unique
execution ID and name of file recording script's stderr. with-name mode prefixes
values with '--exec-id=' and '--error-log-file=' respectively.
By default: only-value.
-h, --help
Display this help.
--mail-instigator=<email>
Specify who executed the supervisor.
--mail-to=<email>
Add a new recipient's email address. Multiple --mail-to options may be used.
--monitor
Check whether supervisor's error log file is empty. If not, then send critical
email notifications using an exponential backoff algorithm in minute increments.
Typically called every minute with a cron job:
* * * * * <user> /path/to/supervisor.sh --conf=<conf-file> --monitor
-p <key>=<value>, --param=<key>=<value>
Allow to inject multiple external parameters into customized emails.
Assign the value <value> to the Bash variable $EXT_<key>.
See --customized-mails option.
--summarize=<max-nb-days>
Display a summary of supervisor's activity during last <max-nb-days> days,
including final status per day and per supervised script.
Also send this summary by email.
<script-path>
Executable script to oversee.
<script-parameters>
Optional oversaw script's parameters.
Exit status
0 if and only if no error
65 Missing script name!
66 Script '…' not found!
67 Script '…' is not executable!
68 Exit code changed from 0 to 68 due to errors.
69 Another instance of '…' is still running with supervisor!
71 Customized mails file not found: '…'
72 Invalid Mutt command: '…'
xxx Any code not null returned by user script
Exit status
List of exit status:
0
If and only if no error65
Missing script name!66
Script '…' not found!67
Script '…' is not executable!68
Exit code changed from 0 to 68 due to errors.69
Another instance of '…' is still running with supervisor!71
Customized mails file not found: '…'72
Invalid Mutt command: '…'- Any code not null returned by user script
Successful execution
An execution is successful if and only if stderr
is empty, exit status is 0
and there are no warnings.
A short example, here in Bash but no matter the language:
#!/usr/bin/env bash echo Title: echo -e '\033[0;30m┆\033[0m \033[1;32mgreen level 1' echo -e '\033[0;30m┆\033[0m \033[0;30m┆\033[0m \033[1;33myellow level 2' echo 'END'
Output:
If SUPERVISOR_MAIL_SEND_ON_STARTUP
is set to 1
in configuration file:
Note that Supervisor add two parameters in addition of script parameters:
- a unique execution ID
- name of file recording script's
stderr
If SUPERVISOR_MAIL_SEND_ON_SUCCESS
is set to 1
in configuration file:
Content of supervisor.info.log.20131006223705_01765.gz
attachment:
2013-10-06 22:37:05 58cs;20131006223705_01765;/usr/local/lib/supervisor/tests/resources/bash_colored_simple.sh;START
2013-10-06 22:37:05 78cs;20131006223705_01765;/usr/local/lib/supervisor/tests/resources/bash_colored_simple.sh;OK
Content of bash_colored_simple.sh.20131006223705_01765.info.log.gz
attachment:
2013-10-06 22:37:05 58cs;[SUPERVISOR] START
2013-10-06 22:37:05 74cs;Title:
2013-10-06 22:37:05 75cs;┆ green level 1
2013-10-06 22:37:05 76cs;┆ ┆ yellow level 2
2013-10-06 22:37:05 77cs;END
2013-10-06 22:37:05 78cs;[SUPERVISOR] OK
Note that colors are stripped but indentation is kept.
Errors
An execution end in error status if either stderr
is not empty or if exit status is different from 0
.
PHP fatal error
We will oversee a PHP script throwing a fatal error:
<?php undefined_fct();
Output:
If SUPERVISOR_MAIL_SEND_ON_ERROR
is set to 1
in configuration file:
Content of supervisor.info.log.20131006223703_01618.gz
attachment:
2013-10-06 22:37:03 42cs;20131006223703_01618;/usr/local/lib/supervisor/tests/resources/php_fatal_error.php;START
2013-10-06 22:37:03 59cs;20131006223703_01618;/usr/local/lib/supervisor/tests/resources/php_fatal_error.php;ERROR
Content of php_fatal_error.php.20131006223703_01618.info.log.gz
attachment:
2013-10-06 22:37:03 42cs;[SUPERVISOR] START
2013-10-06 22:37:03 59cs;[SUPERVISOR] ERROR
Content of php_fatal_error.php.20131006223703_01618.error.log.gz
attachment:
PHP Fatal error: Call to undefined function undefined_fct() in /usr/local/lib/supervisor/tests/resources/php_fatal_error.php on line 4
PHP Stack trace:
PHP 1. {main}() /usr/local/lib/supervisor/tests/resources/php_fatal_error.php:0
[SUPERVISOR] Exit code not null: 255
Other errors
Similarly, scripts provided below end with an error status.
It should be noted that when stderr
is not-empty while exit status is 0
,
then Supervisor change exit status from 0
to 68
(see Exit status).
PHP notice
Content of tests/resources/php_notice.php
:
<?php $a = $b;
Result in php_notice.php.[…].error.log.gz
attachment:
PHP Notice: Undefined variable: b in /usr/local/lib/supervisor/tests/resources/php_notice.php on line 4
PHP Stack trace:
PHP 1. {main}() /usr/local/lib/supervisor/tests/resources/php_notice.php:0
[SUPERVISOR] Exit code changed from 0 to 68 due to errors.
PHP exception
Content of tests/resources/php_exception.php
:
<?php throw new RuntimeException("It's an error!\n", 42);
Result in php_exception.php.[…].error.log.gz
attachment:
PHP Fatal error: Uncaught exception 'RuntimeException' with message 'It's an error!
' in /usr/local/lib/supervisor/tests/resources/php_exception.php:4
Stack trace:
#0 {main}
thrown in /usr/local/lib/supervisor/tests/resources/php_exception.php on line 4
[SUPERVISOR] Exit code not null: 255
PHP stderr
Content of tests/resources/php_stderr.php
:
<?php // file_put_contents('php://stderr', "It's an error!\n"); // or: fwrite(STDERR, "It's an error!\n");
Result in php_stderr.php.[…].error.log.gz
attachment:
It's an error!
[SUPERVISOR] Exit code changed from 0 to 68 due to errors.
Exit not null
Content of tests/resources/bash_exit_not_null.sh
:
#!/usr/bin/env bash exit 42
Result in bash_exit_not_null.sh.[…].error.log.gz
attachment:
[SUPERVISOR] Exit code not null: 42
Tags
Tags allow user scripts to send some information to supervisor.
Tags are strings between square brackets —for example: [warning]
— used in output of supervised scripts.
Tags must be at the beginning of a line or only preceded by tabulations —defined by SUPERVISOR_LOG_TABULATION
— or spaces.
A tag concerns all the line.
Warnings
If a successful executed script contains warning tags in its output, then its status will be changed from Success to Warning.
Default configuration: SUPERVISOR_WARNING_TAG='[WARNING]'
.
If SUPERVISOR_MAIL_SEND_ON_WARNING
is set to 1
in configuration file:
Please note that warnings are listed with 2 lines of context.
Debug traces
By default, messages preceded by a debug tag are hidden in Supervisor's output (but still present in <script>_<exec_id>.info.log
).
Set SUPERVISOR_SHOW_DEBUG_MSG
to 1
to display them.
Default configuration:
SUPERVISOR_DEBUG_TAG='[DEBUG]'
SUPERVISOR_SHOW_DEBUG_MSG=0
E-mail related tags
Default configuration:
SUPERVISOR_MAILTO_TAG='[MAILTO]'
SUPERVISOR_MAIL_ATTACHMENT_TAG='[MAIL_ATTACHMENT]'
TO DOCUMENT
Locks
Sometimes there's the need to ensure that a script is only executed one time.
In order to achieve this, set SUPERVISOR_LOCK_SCRIPT
to 1 in your configuration file.
Only available on Debian/Ubuntu Linux, otherwise leave 0.
# Lock script against parallel run (0|1)
SUPERVISOR_LOCK_SCRIPT=1
Example of output:
Example of sent mail:
Customized mails
TO DOCUMENT
Supervisor supervised
TO DOCUMENT
Archiving
Archive in Gzip format all supervisor's logs older than <min-days>
.
Monitoring
Check whether supervisor's error log file is empty. If not, then send critical
email notifications using an exponential backoff algorithm in minute increments.
Files supervisor.info.log
and supervisor.error.log
will be attached in Gzip format.
Typically called every minute with a cron job:
* * * * * <user> /path/to/supervisor.sh --conf=<conf-file> --monitor
Mail generated through unit tests:
Summary
Display a summary of supervisor's activity during last <max-nb-days>
days,
including final status per day and per supervised script.
Also send this summary by email to SUPERVISOR_MAIL_TO
list.
Installation
-
Move to the directory where you wish to store the source and clone the repository:
$ git clone https://github.com/geoffroy-aubry/Supervisor.git
-
You should be on
stable
branch. If not, switch your clone to that branch:$ cd Supervisor && git checkout stable
-
Create your own configuration file and adapt it:
$ cp conf/supervisor-dist.sh conf/supervisor.sh
Note: file
conf/supervisor-dist.sh
is always loaded. Then optional configuration file specified with option--conf
is loaded (overwriting). But if no option--conf
is found, thenconf/supervisor.sh
is loaded if exists. -
If you have to supervise scripts producing CSV output, then install Awk CSV parser component:
-
Manually:
-
See Installation section of Awk CSV parser's
README.md
. -
Then update
SUPERVISOR_CSV_PARSER
constant in/conf/supervisor.sh
:SUPERVISOR_CSV_PARSER="/path/to/csv-parser.awk"
-
-
Via composer (require PHP >= 5.3.3):
$ composer install # or $ php composer.phar install
See http://getcomposer.org/doc/00-intro.md#installation-nix for more information to install composer.
-
-
You can create a symlink to
supervisor.sh
:$ sudo ln -s /path/to/src/supervisor.sh /usr/local/bin/supervisor
It's ready for use:
$ supervisor --help
Copyrights & licensing
Licensed under the GNU Lesser General Public License v3 (LGPL version 3). See LICENSE file for details.
Change log
See CHANGELOG file for details.
Continuous integration
Estimated code coverage: 87% (553 of 638 lines).
Require PHP >= 5.3.3 and Xdebug.
Create your own configuration file and adapt it:
$ cp conf/phpunit-dist.php conf/phpunit.php
Launch unit tests with PHPUnit:
$ vendor/bin/phpunit --configuration conf/phpunit-dist.xml
On FreeBSD/OS X you must exclude tests about exclusive execution (SUPERVISOR_LOCK_SCRIPT
):
$ vendor/bin/phpunit --configuration conf/phpunit-dist.xml
Git branching model
The git branching model used for development is the one described and assisted
by twgit
tool: https://github.com/Twenga/twgit.