README

Scheduled, model-routed log analysis. Once per window (default 24 h) it reads one service's logs from Loki or Datadog, groups errors into fingerprinted clusters, decides in plain code which ones deserve attention, asks the model provider (OpenAI or Gemini) to judge only those, and posts one digest to Slack, Telegram or Discord. It is read-only: it never acts on a service, opens tickets, or stores raw log lines.

How it works

flowchart LR
  host["Host trigger<br/>cron, CronJob or Laravel schedule"] --> run["watcher run"]
  run --> src[("Log backend<br/>Loki, Datadog or pod events")]
  src --> red["Reduce<br/>fingerprint, counters,<br/>mute and watch rules"]
  red --> sel{"Select<br/>deterministic, no model"}
  sel -- "nothing qualifies" --> quiet["Zero model calls"]
  sel --> tri["Triage<br/>cheap model, one batch"]
  tri --> ana["Analyze<br/>mid model, top M"]
  ana --> esc["Escalate<br/>strong model, once per finding"]
  esc --> dd["Dedupe and digest"]
  ana --> dd
  dd --> out[("Slack, Telegram<br/>or Discord")]
  red <-. "fingerprints, verdicts, run records" .-> st[("StateStore<br/>Redis, PDO or file")]

Every feature has an on/off flag in the config and a WATCHER_* environment variable, so one package can run in int, prod and dev with different settings; the master switch enabled (WATCHER_ENABLED) defaults to off and environment (WATCHER_ENVIRONMENT) prefixes the state keys (see docs/configuration.md and docs/operations.md). Redaction has no switch.

Also: event sinks (log, file, http, Prometheus textfile) and a Grafana dashboard so results can be read without a chat channel (driver none; docs/observability.md), check, status and run --dry-run / --json for operating it, a heartbeat for staleness alerts, fleet mode for several services, baselines and similarity grouping to cut noise, and per-feature flags with an environment label. An hourly health check (watcher:health) reads a host app's health snapshot from its Redis and alerts at once when a service's version changes; it is off by default (Host health check).

Log hygiene (spec 006): vendor/bin/watcher hygiene --config=... reports which log call sites write personal data, which are noisy or badly shaped, which mute/watch rules the history justifies, from counts and masked templates only (see docs/operations.md and docs/privacy.md).

Several runs a day (spec 014): schedule.every (PT1H to PT12H) or a list of schedule.run_at times gives contiguous windows, a high-water mark so nothing is reported twice, catch-up of missed windows after an outage (schedule.catch_up).

Kubernetes events (spec 008) were removed in 0.7.0: the kubernetes and events sections and source.driver: kubernetes fail at load. Export cluster events to your log backend and read them through loki or datadog.

Read-only MCP server (spec 015): watcher mcp over stdio lets an assistant read status, the latest report, one finding, the hygiene report and an optional confirmed dry run, and --setup helps to write and validate a first config. It never writes to a monitored system. The optional actions group (off unless named in mcp.groups) runs the digest, the health check and a test message with parameters: a sending action needs confirm: true, a digest is a dry run unless dry_run: false, and each action is rate limited and audited. See docs/how-it-works.md.

Master switch: when enabled is not the boolean true the package does nothing at all (no source, state, model or notifier access), in every entry point, even with a broken or partial config.

Nothing new to report means zero model calls. The first run only records a baseline. Growth can be measured against the previous window (default), the average of the last N windows, or the median of the same weekday (baseline.mode).

At a glance

Area	Supported today
Log sources	Loki, Datadog, Kubernetes pod events (read-only queries only)
Model providers	Anthropic, OpenAI, Gemini; each step (triage, analyze, escalate) can use a different one
Notifications	Slack, Telegram, Discord, or `none`; event sinks for `log`, `file`, `http` and a Prometheus textfile (observability)
State	Redis (default), any PDO database, or a file (single runner only)
Schedule	Once at `schedule.run_at`, every `PT1H` to `PT12H`, or a list of times; catch-up after an outage
Several services	One fleet file; each service keeps its own window, lock and digest; a consumer's finding can point at its origin
Operations	`check`, `status`, `run --dry-run`, `state:get`, `hygiene`, a read-only MCP server with optional actions (run, health check, test message)
Privacy	Allowlisted fields only, redaction in code, no raw line persisted; the model never chooses its input
Unreleased	Certificate and secret expiry probes (`probes`), see CHANGELOG

How a run works

The host may trigger the command more often than once a window. Only the trigger inside the due hour does work; the others exit 0 and do nothing.

sequenceDiagram
  autonumber
  participant H as Host trigger
  participant W as watcher
  participant S as StateStore
  participant L as Log backend
  participant M as Model provider
  participant N as Notifier
  H->>W: run
  alt not due, or this window already has a run record
    W-->>H: exit 0, nothing done
  else due
    W->>S: take lock
    W->>L: read window (queries only)
    W->>W: reduce, redact, select
    opt candidates exist
      W->>M: triage, analyze, escalate (allowlisted input)
      M-->>W: findings, checked against the schema
    end
    W->>S: snapshot, run record, reported markers
    W->>N: digest (only when something is reportable)
    W-->>H: exit 0 (failure: exit 1 and a notice)
  end

The first run of a service only records a baseline (baselined): it sends no analysis, and the next run compares against it.

Example digest

Shortened from the golden test fixture (tests/Fixtures/golden/); a live digest will differ.

🔎 svc · stg · 2026-10-08 06:00 → 2026-10-09 06:00 (UTC)
2 findings

🟠 error (1)
• Problem <ID> — 5×  `922a8ce78cec`
  Evidence: seen repeatedly; dependency payments shows 6 errors
  Hypothesis: a dependency may be failing  Next: inspect the dependency

🟡 warn (1)
• Problem <ID> — 8× (new)  `9ab3237802d9`
  Evidence: seen repeatedly; dependency payments shows 6 errors
  Hypothesis: a dependency may be failing  Next: inspect the dependency

1 muted
📈 info ×6 vs previous window

Every hypothesis is labelled as one. A finding's level is never lowered by the model.

Requirements

PHP ^8.2. Support matrix: CI tests PHP 8.2 to 8.5. PHP 8.2 security support ends in December 2026; use 8.3 or newer in production. PHP 8.1 support was dropped in 0.1.2.
Optional: ext-redis or predis/predis (Redis state, the default driver), ext-zlib (compresses large Redis values), ext-pdo plus a driver (PDO state).
A notification channel (Slack incoming webhook, Telegram bot token and chat id, or Discord webhook), an OpenAI or Gemini API key, read-only credentials for Loki or Datadog.
Laravel is optional (illuminate/support, illuminate/console ^10 or ^11; CI tests the oldest and the newest versions Composer resolves, on top of the locked set); the core has no framework dependency. Laravel 10 no longer receives security fixes upstream, prefer 11. Lumen is unmaintained upstream: the adapter has no Lumen-specific test, so treat Lumen as best effort.

Installation

Status: pre-1.0. The API may change between minor versions (see CHANGELOG.md).

composer require everrest/watcher

Laravel and Lumen: the service provider is auto-discovered (extra.laravel.providers); in Lumen register Watcher\Laravel\WatcherServiceProvider by hand in bootstrap/app.php. Without a framework use vendor/bin/watcher.

Quick start

Copy config/watcher.example.php into your service and fill in the environment variables it reads. It returns a plain PHP array. Set enabled to true (or WATCHER_ENABLED=1): the master switch is off by default and a disabled run reports disabled and touches nothing, even if the rest of the config is broken or half-filled (only check validates it then).

Check the setup (read-only; posts nothing), preview a run, then run it:

vendor/bin/watcher check --config=path/to/watcher.php        # source, provider auth, models, state, webhooks
vendor/bin/watcher run --config=path/to/watcher.php --dry-run --estimate   # no model call: groups per step
vendor/bin/watcher run --config=path/to/watcher.php --dry-run              # real model calls, digest printed, nothing posted
vendor/bin/watcher run --config=path/to/watcher.php          # add --now=2026-10-23T06:00:00Z to run "as of" an instant
vendor/bin/watcher status --config=path/to/watcher.php       # is the job alive? exit 0 only when ok

Command	Options
`watcher run`	`--config=<file>` `[--now=<ISO-8601>]` `[--dry-run [--estimate]]` `[--json]`
`watcher check`	`--config=<file>` `[--post-test]` `[--json]` (`--post-test` posts one labelled message per destination)
`watcher status`	`--config=<file>` `[--json]`
`watcher state:get <key>`	`--config=<file>`; read-only debug view, `<key>` is `heartbeat`, `last-snapshot`, `snapshot:<window end>` or `run:<window end>` (strings redacted, cut to 200 characters; exit 1 when nothing is stored)

With --json, stdout is exactly one JSON line and every human line goes to stderr. A dry run never posts, writes state. Laravel/Lumen: watcher:run {--now=} {--dry-run} {--estimate} {--json}, watcher:check {--post-test} {--json}, watcher:status {--json}, watcher:state-get {key}, watcher:health (hourly host health check, see below).

Lumen/Laravel: the service provider (Watcher\Laravel\WatcherServiceProvider) registers the watcher:run command and merges config/watcher.php, which reads WATCHER_* environment variables. Laravel auto-discovers the provider; in Lumen register it with $app->register(Watcher\Laravel\WatcherServiceProvider::class) in bootstrap/app.php. Laravel can publish the config with php artisan vendor:publish --tag=watcher-config. Then schedule it hourly:
```
$schedule->command('watcher:run')->hourly();
```

Documentation

The README is the overview; the detail lives in docs/ (the index). Internal planning documents in docs/ (audit, review, brief, roadmap) and the publishing and development guides are excluded from the Composer archive via .gitattributes; they remain in the repository.

Document	What it covers
docs/README.md	Index of every document in `docs/`
docs/smoke-test.md	First live run, step by step
docs/how-it-works.md	A map: where the watcher runs, where its configuration lives, what runs when, and the traps
docs/operations.md	Running it: scheduling, alerting, dry run, heartbeat, fleet, troubleshooting
docs/configuration.md	Every key and environment variable, feature flags
docs/privacy.md	What leaves the service
docs/development.md	Tests, coverage, hooks, supported versions
docs/publishing.md, docs/repository-settings.md	Release process and GitHub settings
SECURITY.md, CHANGELOG.md	Reporting vulnerabilities; release history
docs/roadmap.md, docs/brief.md	Planned specs; the original brief
docs/examples/	Configs: single service, Loki, fleet
.specify/memory/constitution.md	Principles every change must respect

Architecture and decisions for the run workflow live in spec 001: plan, research, data model and the contracts (configuration, PHP interfaces, analyzer output schema, Slack messages).

Specs

Development is spec-driven. Status is from docs/roadmap.md and the changelog.

Spec	Topic	Status
001	Scheduled log watch	Implemented (live run pending)
002	Operability tools	Implemented
003	Fleet and correlation	Implemented
004	Baselines and templates	Implemented (backend patterns deferred)
005	Incident workflow	Phases 1–3 implemented (known issues, severity routing, ticket drafts); weekly report deferred
006	Log hygiene	Implemented
007	Code-aware analysis	Spec written, blocked on a constitution amendment
008	Kubernetes events	Removed in 0.7.0
009	Metrics and Sentry	Not released (branch `feat/metrics-e009-030`)
010	API context	Implemented, off by default
011	More notifiers (Telegram, Discord)	Implemented
012	Feature flags and environments	Implemented (changelog)
013	Observability sink	Implemented (changelog)
014	Schedule periods	Phases 1–2 implemented (periods, catch-up, DST, caps); same-slot baselines and roll-up open
015	Read-only MCP server, onboarding and Q&A	Implemented (read-only tools, setup and Q&A); docs and some tests open

Configuration

config/watcher.example.php is the complete plain-PHP reference: every key with its default and a one-line comment. config/watcher.php is the same surface for Lumen/Laravel, driven by environment variables. The contract is specs/001-scheduled-log-watch/contracts/configuration.md.

Configuration is validated once, before any network call. A bad value throws InvalidConfig naming the key (never the value), and the process exits with code 2. Unknown top-level keys are rejected. Required: service, schedule.run_at, a source driver with its credentials, a key for every provider the steps use (analysis.api_keys), all three analysis.models, a state driver with its settings, notify.digest.webhook.

Log sources

Use read-only credentials. The package only issues query requests (enforced by tests/Unit/ArchitectureTest.php).

Driver	Requests	Credential
`loki`	`GET /loki/api/v1/query`, `GET /loki/api/v1/query_range`	Bearer token or basic auth with query-only rights; optional `X-Scope-OrgID` tenant. `https` is required; plain `http` only with `allow_insecure: true` and a loopback, private-network (RFC 1918) or cluster-local host (see docs/operations.md). `url` may be a Loki base URL or a Grafana datasource proxy URL.
`datadog`	`POST /api/v2/logs/analytics/aggregate`, `POST /api/v2/logs/events/search` (read-only queries sent as POST)	API key plus an application key limited to the `logs_read_data` scope. Set `site` (e.g. `datadoghq.eu`).

source.stream_tags selects which logs belong to the service. Only levels at or above source.read_min_level are read as events; lower levels are counted in totals only.

State

State holds fingerprints, counters, cached verdicts and run records. Every item has an expiry (state.retention, default 30 days; notice markers 2 days, noise/known verdicts 3 days, heartbeat records 3x retention; see docs/privacy.md).

Driver	Notes
`redis`	The default. Needs `ext-redis` or `predis/predis` (`state.redis.client`: `auto`, `phpredis`, `predis`). Pass `host/port/password/database`, or inject a connected client as `state.redis.instance`. Every key has a TTL; values over `state.redis.compress_min_bytes` (512) are zlib-compressed.
`pdo`	Any PDO DSN. The table is never created implicitly: run `Watcher\State\PdoStateStore::createTableSql($driver, $table)` once (sqlite, mysql, pgsql) in a migration.
`file`	Single runner only. No cross-process guarantees; use it for one host or development. In Kubernetes a pod-local directory loses history (`watcher check` warns).

Scheduling

The host triggers the command at least hourly. The package decides whether a trigger is the scheduled one: a run is due from schedule.run_at (in schedule.timezone) for one hour; with schedule.every or a list of run_at times it is due at each boundary. Outside that hour the trigger exits 0 at no cost. Inside it, a lock (schedule.lock_ttl) and a per-window run record make the run idempotent: a second trigger in the same window does nothing, and a failed run leaves no record so the next trigger retries.

Host health check

watcher:health is separate from the daily run: no run gate and no digest. Once per hour it reads the host app's health snapshot from the host's Redis, one JSON object {"<service>": {"success": bool, "version": string|null, "checked_at": ISO-8601}}, and sends an immediate notice when a service's version changes. Once a day, at health_check.summary_at (09:00 by default), it also sends a summary of every service. Both go out only on health_check.summary_days (weekdays by default) and are judged in health_check.timezone (UTC by default; set Europe/Berlin for CET). Only those four fields are kept, one copy per service under health:<watcher service>:<snapshot service> in the watcher's Redis. A null or Version not defined version is unknown and never alerts; the first sighting stores silently. A failed read or send exits this command with 1 and writes to the host log; the daily run and the digest are not affected.

Off by default. Enable it with health_check.enabled (the watcher's enabled must be on too); the host must schedule the command hourly:

$schedule->command('watcher:health')->hourly();

Keys, the optional up/down transitions (health_check.notify_transitions) and the rest are in docs/configuration.md.

Privacy

Enforced in code, not by the model. The model never redacts and never chooses its own input.

Only allowlisted fields are sent: level, normalised message template (numbers, ids, emails, tokens, hosts become placeholders such as <N>, <ID>, <EMAIL>), error type, at most 5 stack frames, counts and timestamps.
Clusters smaller than redaction.min_group_events_for_text (default 3) send no message text, because small clusters are the ones most likely to hold a single person's data.
Fail closed per field or event: a field that cannot be scanned is dropped, and a line the rules cannot scan becomes <UNSCANNABLE> (it never fails the window). A shape no rule recognises is not an error and passes; see the limits in docs/privacy.md. Add your own patterns with redaction.extra_patterns.
Model output is scrubbed again before it is cached or shown.
The model cannot lower an error-or-higher group: its findings are never shown below the group's level, and a cached verdict below that level is not reused (see docs/privacy.md).
No raw log line is persisted. Field values used by watch_rules and mute are matched in memory and dropped.

What leaves the service: (1) to the model provider (OpenAI or Gemini), the compact payload above plus service_context (up to 1000 chars, written by you); (2) to the notification channel (Slack, Telegram or Discord), the digest (titles, evidence, hypotheses, counts, short fingerprints) and failure notices; (3) queries to your log backend. Nothing else is sent anywhere.

flowchart TB
  raw["Raw log lines<br/>stay in the backend"] -->|read-only query| allow["Allowlist<br/>level, template, error type,<br/>5 frames, counts, timestamps"]
  allow --> red["Redaction<br/>emails, ids, tokens, hosts become placeholders"]
  red -->|a field that cannot be scanned is dropped| model["Model provider<br/>compact payload only"]
  model --> scrub["Output scrubbed again<br/>before cache or display"]
  scrub --> digest["Digest<br/>titles, evidence, counts, short fingerprints"]
  digest --> chan["Notification channel"]
  red -.->|groups under 3 events| text["No message text sent"]

Model calls

Work is routed by tier and each tier has its own model id and input limit (no ids in code):

triage (cheap model): one batched call over the selected candidates; verdict per fingerprint.
analyze (mid model): only groups triage marked interesting, at most selection.top_m.
escalate (strong model): only for error-or-worse findings with confidence below analysis.confidence_threshold, once per finding, at most analysis.escalation_cap per run.

Selection is deterministic: new fingerprints, groups that grew by selection.growth_factor, level at or above selection.min_level, watch-rule matches, plus the top selection.top_n; the rest collapse to one summary line. No candidates means zero model calls; unchanged groups reuse cached verdicts. Inputs over max_input_tokens drop the lowest-priority groups and the digest says so.

Noise control

mute: by fingerprint or matcher (template, error_type, field), optionally until a date. Muted groups never reach a model; the digest shows a count.
watch_rules: named matchers that raise a matching group's priority; only the rule name leaves the service. Plain-literal template rules and RE2-safe error_type rules are also sent to Loki as search filters (Datadog gets only plain-word literals; one pattern it cannot express switches its server-side filter off), so a matching line below read_min_level is still read; other rules (non-literal templates, field) are matched only on the lines that were read.
Dedupe: a group already reported is omitted unless it grew again.

Failures and exit codes

Code	Meaning
0	Success, including "not due", "already done", "locked", baseline and quiet runs
1	Run failed (source unavailable, analysis failed, delivery failed, internal error)
2	Invalid configuration, bad arguments or bad `--now` (`--estimate` without `--dry-run`)

watcher check exits 0 when no check failed, 1 otherwise (an invalid config is a failed config check). watcher status exits 0 when ok, 1 when degraded, stale or failed, 3 when unknown (no record, expired record or unreadable store). No source sets degraded in 0.7.0.

Failures are never silent: a source, analysis or internal failure posts a short notice to notify.notice (or the digest destination) and returns 1. If the notice channel itself fails, the host logger (if you pass one to Watcher::fromConfig()) and the exit code carry it. Notices state a reason and never include secrets or values.

Digest

One message per service per run: a header (service, environment, window), findings grouped by severity (critical, error, warn, info), then a line for minor/low-confidence findings, remaining groups, muted count, truncation notices. A finding shows title, event count and trend (new or up from N), a 12-character fingerprint, evidence, a hypothesis (always labelled as such), a suggested next step, a Check: line saying what to look at to confirm or refute the hypothesis (never reproduction steps) and, when useful, a Related: line naming other groups or declared dependencies to look at next. analysis.advice.enabled=false turns the last two off and saves their output tokens.

Several services (fleet)

One job can watch several services, including ones that cannot host the package (no CLI, no scheduler): list them in a fleet file and run watcher fleet:run --config=<file|dir>. Each service keeps its own window, run record, lock and digest and runs in isolation (one failing service never stops another; the exit code is the worst). A fleet can add a one combined digest (digest: combined), and correlation: when a consumer's findings reference a service that has a new or grown finding in the same window, the consumer's own finding is annotated likely caused by the <origin> finding <fingerprint> (hypothesis) and one additional message names the origin. It is decided in code from names, placeholders and times; it makes no model call and withholds nothing. See docs/examples/watcher-fleet.php, operations and the configuration reference. notify.days is set per service, so each service in a fleet can have its own send days.

Known limits

The Grafana Explore URL shape and the Datadog response shapes are not verified against live systems; sources are tested against recorded fixtures only.
The request formats of the built-in OpenAI and Gemini clients were tested against recorded fixtures only, not the live APIs.
At most source.group_cap (5000) distinct problem groups are held per run; on a day with more distinct messages the rest is grouped as one "other" group per level and the digest says so. Watch-rule matches are exempt up to twice the cap.
Cluster templates group by word similarity, not exact shape (grouping.similarity); two different problems with very similar wording can still share a template, and fingerprints do not depend on it. A name identical in every event of a cluster is not detectable (see docs/privacy.md).
Grouping is bounded: above grouping.message_cap distinct messages, or when the comparison budget runs out, the affected groups show no message text and the digest says so. Backend-provided patterns (grouping.backend_patterns) are deferred and rejected for now.
baseline.mode rolling and weekday need enough retained history (state.retention); with too little the run falls back to the previous window and the digest says so.
Field values never leave the service, so field matchers can match but cannot be explained to the model.
Dependencies are read with the service's own log source: dependencies.<name>.source (a different backend per dependency) is rejected as not available yet. A dependency's lines are read before triage, so a day on which triage marks everything noise can still have spent the dependency reads (totals are always one query per dependency).

threshold again or on the next day.
Loki's server-side filter works per physical line, so stack-trace lines are not returned and frames come from the head line of a record; a record that straddles a Loki page boundary stays split.
Only Slack, Telegram, Discord, OpenAI, Gemini, Loki, Datadog and the three state drivers exist. There is no remediation and no ticketing.
Metrics and Sentry (spec 009) and the weekly report (spec 005) are not released yet.
Same-slot baselines for several runs a day and the daily roll-up digest (spec 014) are not released yet.

Development

composer install
composer test    # ParaTest (parallel), no network; composer test:serial = plain PHPUnit
composer stan    # PHPStan
composer style   # Pint (--test)

Development is spec-driven: .specify/memory/constitution.md (principles), docs/brief.md (origin), specs/001-scheduled-log-watch/ (spec, plan, contracts, quickstart, tasks). A behaviour change starts with a failing test; a change to what leaves the service extends the privacy corpus (tests/Fixtures/privacy). tests/Unit/ArchitectureTest.php guards: no model ids, no consumer names, read-only sources, no framework in the core.

everrest / watcher

Package info

Security

Statistics