geoffroy-aubry/supervisor

Oversee script execution, recording stdout, stderr and exit code with timestamping, and ensure email notifications will be sent (on start, success, warning or error).

Installs: 1 115

Dependents: 1

Stars: 5

Watchers: 1

Forks: 1

Language: PHP

v1.8.1 2015-05-04 22:58 UTC

README

Latest stable version Build Status  Estimated code coverage: 87% (553 of 638 lines).

Oversee script execution, recording stdout, stderr and exit code with timestamping, and ensure email notifications will be sent (on startup, success, warning or error)… plus many other features.

Technologies : Supervisor is in Bash, only unit tests are in PHP.

Supported OS : Debian/Ubuntu Linux, FreeBSD/OS X.

Table of Contents

Features

General

  • Oversee script execution, recording stdout, stderr and exit code
  • Prefix each line of stdout with timestamps with hundredths of seconds.
  • An execution ID uniquely identify each script execution. Used in: supervisor.info.log, supervisor.error.log, <script>_<exec_id>.info.log and <script>_<exec_id>.error.log.
  • Possibility to block parallel script execution (only on Debian/Ubuntu Linux). Useful if an execution takes too long…
  • You can specify a configuration file to load in addition to the default one (overloading).
  • Entirely configurable tag system allowing executed scripts to dynamically specify via stdout warning, mailto, mail attachment, instigator…
  • Automatic log archiving mechanism.
  • Display a summary of supervisor's activity during last days, including final status per day and per supervised script.
  • Handle CSV output, with definition of field separator, field enclosure and number of the field to watch in configuration file.
  • Add proper signal handling and interrupts: SIGHUP, SIGINT, SIGQUIT and SIGTERM now transmitted to supervised script.

Notifications

  • Ensure email notifications will be sent on following events (configurable): startup, success, warning or error. Catch exit code, fatal error, exceptions…
  • All mail contains content of stdout and stderr of executed script in Gzip attachment.
  • Mails are fully customizable. Just provide your file on command line.
  • Possibility to inject multiple external parameters on command line into customized emails.
  • Supervisor itself is monitoring by another process, sending critical email notifications using an exponential backoff algorithm in minute increments.

Ergonomics

  • During supervision and in addition of all log files, Supervisor displays in real time outputs, warning and errors of overseen script, adding timestamp and respecting both colors and indentation. Name of all log files are printed too.

Code quality

  • Bash code executed with following directives:
    • set -o nounset: treat unset variables and parameters other than the special parameters @ or * as an error when performing parameter expansion. An error message will be written to the standard error, and a non-interactive shell will exit.
    • set -o pipefail: The return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully.
  • A lot of unit tests (via PHPUnit through a PHP wrapper) covering almost the whole application.

Requirements

  • Bash v4 (2009) and above
  • mutt, to send email notifications (apt-get install mutt, or brew install mutt)
  • Unit tests require PHP >= 5.3.3 and Xdebug
  • On FreeBSD/OS X only, you need to install these GNU tools:

      $ brew install coreutils
      $ brew install gnu-sed
      $ brew install gawk

Tested on Debian/Ubuntu Linux and FreeBSD/OS X.

Usage

Help on command prompt

Displayed by:

$ supervisor.sh [-h|--help]

Help on command prompt

Text version
Description
    Oversee script execution, recording stdout, stderr and exit code with timestamping,
    and ensure email notifications will be sent (on start, success, warning or error).

Usage
    supervisor [OPTION]… <script-path> [<script-parameters>]
    supervisor [-c <conf-file>] --archive=<min-days>
    supervisor [-c <conf-file>] --monitor
    supervisor [-c <conf-file>] --summarize

Options
    --archive=<min-days>
        Archive in Gzip supervisor's logs older than <min-days>.

    -c <conf-file>, --conf=<conf-file>
        Specify a configuration file to load in addition to the default one,
        namely '/conf/supervisor-dist.sh'.

    --customized-mails=<file>
        Path to a Bash script customizing sent mails by redefining some of
        the sendMailOn[Init|Success|Warning|Error]() functions.
        See --param option.

    --exec-id=<string>
        Allow to force execution id, used in mails and name of logs.
        By default: YYYYMMDDHHIISS_XXXXX, where X are random digits.

    --extra-param-mode=[only-value|with-name|none]
        Two extra parameters are added to the end of <script-parameters>: a unique
        execution ID and name of file recording script's stderr. with-name mode prefixes
        values with '--exec-id=' and '--error-log-file=' respectively.
        By default: only-value.

    -h, --help
        Display this help.

    --mail-instigator=<email>
        Specify who executed the supervisor.

    --mail-to=<email>
        Add a new recipient's email address. Multiple --mail-to options may be used.

    --monitor
        Check whether supervisor's error log file is empty. If not, then send critical
        email notifications using an exponential backoff algorithm in minute increments.
        Typically called every minute with a cron job:
            * * * * * <user> /path/to/supervisor.sh --conf=<conf-file> --monitor

    -p <key>=<value>, --param=<key>=<value>
        Allow to inject multiple external parameters into customized emails.
        Assign the value <value> to the Bash variable $EXT_<key>.
        See --customized-mails option.

    --summarize=<max-nb-days>
        Display a summary of supervisor's activity during last <max-nb-days> days,
        including final status per day and per supervised script.
        Also send this summary by email.

    <script-path>
        Executable script to oversee.

    <script-parameters>
        Optional oversaw script's parameters.

Exit status
      0 if and only if no error
     65 Missing script name!
     66 Script '…' not found!
     67 Script '…' is not executable!
     68 Exit code changed from 0 to 68 due to errors.
     69 Another instance of '…' is still running with supervisor!
     71 Customized mails file not found: '…'
     72 Invalid Mutt command: '…'
    xxx Any code not null returned by user script

Exit status

List of exit status:

  • 0 If and only if no error
  • 65 Missing script name!
  • 66 Script '…' not found!
  • 67 Script '…' is not executable!
  • 68 Exit code changed from 0 to 68 due to errors.
  • 69 Another instance of '…' is still running with supervisor!
  • 71 Customized mails file not found: '…'
  • 72 Invalid Mutt command: '…'
  • Any code not null returned by user script

Successful execution

An execution is successful if and only if stderr is empty, exit status is 0 and there are no warnings.

A short example, here in Bash but no matter the language:

#!/usr/bin/env bash

echo Title:
echo -e '\033[0;30m┆\033[0m   \033[1;32mgreen level 1'
echo -e '\033[0;30m┆\033[0m   \033[0;30m┆\033[0m   \033[1;33myellow level 2'
echo 'END'

Output:

Successful output

If SUPERVISOR_MAIL_SEND_ON_STARTUP is set to 1 in configuration file:

Mail on startup

Note that Supervisor add two parameters in addition of script parameters:

  • a unique execution ID
  • name of file recording script's stderr

If SUPERVISOR_MAIL_SEND_ON_SUCCESS is set to 1 in configuration file:

Mail on success

Content of supervisor.info.log.20131006223705_01765.gz attachment:

2013-10-06 22:37:05 58cs;20131006223705_01765;/usr/local/lib/supervisor/tests/resources/bash_colored_simple.sh;START
2013-10-06 22:37:05 78cs;20131006223705_01765;/usr/local/lib/supervisor/tests/resources/bash_colored_simple.sh;OK

Content of bash_colored_simple.sh.20131006223705_01765.info.log.gz attachment:

2013-10-06 22:37:05 58cs;[SUPERVISOR] START
2013-10-06 22:37:05 74cs;Title:
2013-10-06 22:37:05 75cs;┆   green level 1
2013-10-06 22:37:05 76cs;┆   ┆   yellow level 2
2013-10-06 22:37:05 77cs;END
2013-10-06 22:37:05 78cs;[SUPERVISOR] OK

Note that colors are stripped but indentation is kept.

Errors

An execution end in error status if either stderr is not empty or if exit status is different from 0.

PHP fatal error

We will oversee a PHP script throwing a fatal error:

<?php

undefined_fct();

Output:

Error output

If SUPERVISOR_MAIL_SEND_ON_ERROR is set to 1 in configuration file:

Mail on error

Content of supervisor.info.log.20131006223703_01618.gz attachment:

2013-10-06 22:37:03 42cs;20131006223703_01618;/usr/local/lib/supervisor/tests/resources/php_fatal_error.php;START
2013-10-06 22:37:03 59cs;20131006223703_01618;/usr/local/lib/supervisor/tests/resources/php_fatal_error.php;ERROR

Content of php_fatal_error.php.20131006223703_01618.info.log.gz attachment:

2013-10-06 22:37:03 42cs;[SUPERVISOR] START
2013-10-06 22:37:03 59cs;[SUPERVISOR] ERROR

Content of php_fatal_error.php.20131006223703_01618.error.log.gz attachment:

PHP Fatal error:  Call to undefined function undefined_fct() in /usr/local/lib/supervisor/tests/resources/php_fatal_error.php on line 4
PHP Stack trace:
PHP   1. {main}() /usr/local/lib/supervisor/tests/resources/php_fatal_error.php:0
[SUPERVISOR] Exit code not null: 255
Other errors

Similarly, scripts provided below end with an error status.

It should be noted that when stderr is not-empty while exit status is 0, then Supervisor change exit status from 0 to 68 (see Exit status).

PHP notice

Content of tests/resources/php_notice.php:

<?php

$a = $b;

Result in php_notice.php.[…].error.log.gz attachment:

PHP Notice:  Undefined variable: b in /usr/local/lib/supervisor/tests/resources/php_notice.php on line 4
PHP Stack trace:
PHP   1. {main}() /usr/local/lib/supervisor/tests/resources/php_notice.php:0
[SUPERVISOR] Exit code changed from 0 to 68 due to errors.
PHP exception

Content of tests/resources/php_exception.php:

<?php

throw new RuntimeException("It's an error!\n", 42);

Result in php_exception.php.[…].error.log.gz attachment:

PHP Fatal error:  Uncaught exception 'RuntimeException' with message 'It's an error!
' in /usr/local/lib/supervisor/tests/resources/php_exception.php:4
Stack trace:
#0 {main}
  thrown in /usr/local/lib/supervisor/tests/resources/php_exception.php on line 4
[SUPERVISOR] Exit code not null: 255
PHP stderr

Content of tests/resources/php_stderr.php:

<?php

// file_put_contents('php://stderr', "It's an error!\n");
// or:
fwrite(STDERR, "It's an error!\n");

Result in php_stderr.php.[…].error.log.gz attachment:

It's an error!
[SUPERVISOR] Exit code changed from 0 to 68 due to errors.
Exit not null

Content of tests/resources/bash_exit_not_null.sh:

#!/usr/bin/env bash

exit 42

Result in bash_exit_not_null.sh.[…].error.log.gz attachment:

[SUPERVISOR] Exit code not null: 42

Tags

Tags allow user scripts to send some information to supervisor.

Tags are strings between square brackets —for example: [warning]— used in output of supervised scripts. Tags must be at the beginning of a line or only preceded by tabulations —defined by SUPERVISOR_LOG_TABULATION— or spaces. A tag concerns all the line.

Warnings

If a successful executed script contains warning tags in its output, then its status will be changed from Success to Warning.

Default configuration: SUPERVISOR_WARNING_TAG='[WARNING]'.

If SUPERVISOR_MAIL_SEND_ON_WARNING is set to 1 in configuration file:

Mail on error

Please note that warnings are listed with 2 lines of context.

Debug traces

By default, messages preceded by a debug tag are hidden in Supervisor's output (but still present in <script>_<exec_id>.info.log). Set SUPERVISOR_SHOW_DEBUG_MSG to 1 to display them.

Default configuration:

SUPERVISOR_DEBUG_TAG='[DEBUG]'
SUPERVISOR_SHOW_DEBUG_MSG=0
E-mail related tags

Default configuration:

SUPERVISOR_MAILTO_TAG='[MAILTO]'
SUPERVISOR_MAIL_ATTACHMENT_TAG='[MAIL_ATTACHMENT]'

TO DOCUMENT

Locks

Sometimes there's the need to ensure that a script is only executed one time. In order to achieve this, set SUPERVISOR_LOCK_SCRIPT to 1 in your configuration file. Only available on Debian/Ubuntu Linux, otherwise leave 0.

# Lock script against parallel run (0|1)
SUPERVISOR_LOCK_SCRIPT=1

Example of output:

Example of lock

Example of sent mail:

Example of sent e-mail

Customized mails

TO DOCUMENT

Supervisor supervised

TO DOCUMENT

Archiving

Archive in Gzip format all supervisor's logs older than <min-days>.

Archiving

Monitoring

Check whether supervisor's error log file is empty. If not, then send critical email notifications using an exponential backoff algorithm in minute increments. Files supervisor.info.log and supervisor.error.log will be attached in Gzip format.

Typically called every minute with a cron job:

* * * * * <user> /path/to/supervisor.sh --conf=<conf-file> --monitor

Mail generated through unit tests:

Critical error

Summary

Display a summary of supervisor's activity during last <max-nb-days> days, including final status per day and per supervised script. Also send this summary by email to SUPERVISOR_MAIL_TO list.

Summary

Installation

  1. Move to the directory where you wish to store the source and clone the repository:

    $ git clone https://github.com/geoffroy-aubry/Supervisor.git
  2. You should be on stable branch. If not, switch your clone to that branch:

    $ cd Supervisor && git checkout stable
  3. Create your own configuration file and adapt it:

    $ cp conf/supervisor-dist.sh conf/supervisor.sh

    Note: file conf/supervisor-dist.sh is always loaded. Then optional configuration file specified with option --conf is loaded (overwriting). But if no option --conf is found, then conf/supervisor.sh is loaded if exists.

  4. If you have to supervise scripts producing CSV output, then install Awk CSV parser component:

  5. You can create a symlink to supervisor.sh:

    $ sudo ln -s /path/to/src/supervisor.sh /usr/local/bin/supervisor

It's ready for use:

$ supervisor --help

Copyrights & licensing

Licensed under the GNU Lesser General Public License v3 (LGPL version 3). See LICENSE file for details.

Change log

See CHANGELOG file for details.

Continuous integration

Build Status  Estimated code coverage: 87% (553 of 638 lines).

Require PHP >= 5.3.3 and Xdebug.

Create your own configuration file and adapt it:

$ cp conf/phpunit-dist.php conf/phpunit.php

Launch unit tests with PHPUnit:

$ vendor/bin/phpunit --configuration conf/phpunit-dist.xml

On FreeBSD/OS X you must exclude tests about exclusive execution (SUPERVISOR_LOCK_SCRIPT):

$ vendor/bin/phpunit --configuration conf/phpunit-dist.xml

Git branching model

The git branching model used for development is the one described and assisted by twgit tool: https://github.com/Twenga/twgit.