README

make ci answers "is the code sound". This package answers the other question, the one nothing used to ask: is what runs on this server complete?

It carries the deployment commands that used to be copied from one project to the next:

Command	What it does
`install:doctor`	Diagnoses the deployment: environment, dependencies, database, queue, scheduler, worker, assets
`install:cron`	Renders and installs the `/etc/cron.d` entry running the scheduler
`install:supervisor`	Renders and installs the Supervisor program running the queue worker
`install:logrotate`	Renders and installs the rotation of the logs Laravel does not rotate itself
`install:fail2ban`	Renders and installs the filter and jails of the scanner trap
`ops:security-audit`	Runs `composer audit` and `npm audit`, and mails new advisories
`ops:queue`	Shows the pending and failed jobs of a database queue, and acts on one pending job

Installation

composer require caprel/laravel-ops

The commands are available right away. Two things are worth publishing:

# The deployment templates, into deploy/ of the project. Version them there.
php artisan vendor:publish --tag=ops-deploy

# The configuration, only when a path or a repair command has to change.
php artisan vendor:publish --tag=ops-config

Publish config/ops.php whole: the merge with the defaults of the package happens on the top-level keys only, so a file holding a partial cron section replaces the whole section instead of completing it.

The templates are templates

Everything under deploy/ carries {{APP_PATH}}, {{PHP}}, {{USER}}, {{LOG_DIR}}, {{APP_SLUG}}: the install:* commands render them with the values of the running deployment. Never copy one of those files into /etc with cp, and never hand-edit the generated file:

Supervisor refuses a program whose stdout_logfile directory does not exist, and it refuses the WHOLE file with it (CANT_REREAD), taking every other program declared there down;
logrotate and fail2ban say nothing at all, and simply never rotate nor ban;
a hand-edited file in /etc is reverted by the next deploy.

The fail2ban jails name their action explicitly (name=<slug>-hp, name=<slug>-hpi). Left to the default, fail2ban names the iptables chain f2b-<jail name>, iptables refuses a name over 28 characters, and the jail logs Ban <ip> for every scanner while dropping no packet: any slug over 7 characters broke the instant jail. A project published before the fix carries a template without these lines; install:fail2ban now refuses it. Copy the two action lines of stubs/deploy/fail2ban/jail.d/honeypot.conf into the project's deploy/fail2ban/jail.d/honeypot.conf and re-run install:fail2ban --force.

On a server where the jails already ran, restart them, never reload:

fail2ban-client restart <slug>-honeypot
fail2ban-client restart <slug>-honeypot-instant
fail2ban-client get <slug>-honeypot-instant actions   # must name ...-<slug>-hpi

A reload keeps only the actions the jail already had: one under a new name is created, then removed at the end of the same reload as obsolete (the log says Flush ticket(s) with <action>). The jail is left with no action at all, get <jail> actions answers No actions, bans are still counted and nothing reaches the firewall, without a single error. The chain itself only appears at the first ban, so iptables -S | grep f2b- is empty until then.

Every command accepts --print (render to stdout, write nothing), --root (prefix every target, to inspect a whole tree), --force (replace a target whose content differs) and --user / --group / --php.

Diagnosing

php artisan install:doctor          # readable table
php artisan install:doctor --json   # same content for a probe

Three statuses, and a return code that follows: OK, WARN (return code 0), FAIL (non-zero, so the command plugs into external monitoring as it is).

Every line that is not OK carries the command that repairs it, and that command is the one of the project: make migrate rather than php artisan migrate when the Makefile wraps the rights and the --force. A project that deploys differently overrides ops.fix_commands and the doctor stops naming a target that does not exist there.

Checks that concern the server keep quiet when the server is not there: no /etc/cron.d on this machine means the cron check passes with a mention, because a diagnosis that shouts on a workstation stops being read anywhere.

Reading the queue

Laravel acts on whole queues and on failed jobs (queue:retry, queue:forget, queue:flush, queue:pause, queue:resume, queue:clear) but has no way to show what is waiting: Horizon does it for Redis alone, and every application of the park runs the database driver.

php artisan ops:queue                         # every queue by state, then the backlog by job class
php artisan ops:queue --list --queue=mail     # one line per pending job
php artisan ops:queue --job=SendInvoice       # only the classes containing this fragment
php artisan ops:queue --failed                # failed jobs, most recent first, first line of the error
php artisan ops:queue --show=123              # one pending job and its payload
php artisan ops:queue --show-failed=<uuid>    # one failed job, its payload and its trace
php artisan ops:queue --delete=123            # asks first, --force to skip
php artisan ops:queue --run-now=123           # a delayed or stalled job, available at once

A pending job is in one of four states:

State	Meaning
waiting	ready, no worker has taken it yet
delayed	dispatched or released with a delay, not ready yet
running	reserved by a worker within `retry_after`
stalled	reserved longer ago than `retry_after`: its worker died or hangs. The next worker takes it again on its own

A running job is never deleted nor moved: that would not stop it, and its worker would then fail on its own bookkeeping.

A payload is shown as stored and never unserialised: the command it carries may restore models from the database on wake-up, or be encrypted. It holds the arguments of the job, which are often personal data (a recipient, a document title): who may read it is the caller's decision, and the admin screen of caprel/laravel-admin restricts it.

The reading lives in Caprel\Ops\Queue\QueueInspector, which that admin screen uses too: one reading of the tables, whoever asks.

Cron or Supervisor

The scheduler runs from one of two places, never both:

// config/ops.php
'scheduler' => env('OPS_SCHEDULER', 'cron'),   // 'cron' | 'supervisor'

Left on cron, the doctor asks for the /etc/cron.d entry and install:cron installs it. Set to supervisor, the doctor asks instead for a schedule:work program pointing at this checkout, and install:cron refuses to run: installing the entry alongside would make every scheduled task run twice, which on a task that writes is an incident rather than waste.

What does NOT depend on the key: the single-scheduler check. Declaring one topology does not stop the other from being installed by hand, and that pair is precisely what must never exist, so it is reported either way.

Two things to move along with the key when choosing supervisor:

ops.cron.log points at the stdout_logfile of the scheduler program, which is the file whose freshness proves the scheduler still runs;
ops.cron.max_age_minutes comes from the most frequent task of the project. schedule:run writes a line every minute even with nothing due, but schedule:work writes only when a task actually runs, so the default of five minutes describes cron and nothing else.

Dependency audit

ops:security-audit runs both audits, and mails the result to ops.security.recipient (OPS_SECURITY_EMAIL), falling back to the address the application sends its mail from. The same list of advisories is reported once, not every morning: a daily mail that never changes stops being read within a week, and the day a critical advisory appears in it nobody notices.

Schedule it in routes/console.php:

Schedule::command('ops:security-audit')->dailyAt('06:00');

The hash of the last report sits in the default cache store. A deployment that clears the cache forgets it, and the next morning mails the same list again: name a store the deployment keeps (a file store outside storage/framework, for instance) in ops.security.cache_store (OPS_SECURITY_CACHE_STORE).

An application with its own mail layout points the views at its own, rather than publishing those of the package:

'security' => [
    'views' => [
        'html' => 'emails.security-audit',
        'text' => 'emails.security-audit-text',
    ],
],

Tests

composer ci        # pint + phpstan + phpunit

The suite runs every command against a throwaway deployment built in a temporary directory, whose deploy/ holds the stubs this package publishes: what the tests cover is what a project gets.

The CI runners have no logrotate, fail2ban nor supervisor, so the generated files are read back by tests/Support/SystemFiles.php, which applies the rules of each format: an unknown logrotate directive, a cron.d line without its user field, an INI key declared twice or a failregex missing the trap's levels fail the suite instead of being skipped in silence on the server.

On top of composer ci, the pipeline validates composer.json, runs composer audit, and runs the tests a third time against the lowest versions the constraints allow (composer update --prefer-lowest).

Licence

AGPL-3.0-or-later, see LICENSE.md.

The services that install this package are published under the same licence. Keeping the package free is what lets anyone redeploy one of them without a piece missing.

caprel / laravel-ops

Package info

Security

Statistics