README

Metric, span and span-tag probes on a running PHP service, added, changed and removed from Sentinel without a deploy.

PHP pays every cost on every request in every worker: nothing survives from one request to the next, so a client that polls, parses or exports on the request path multiplies that work by the request rate. This client is built around that fact, and the numbers below were measured.

How it works

The request path does almost nothing. bootstrap/prepend.php runs first on every request. It does four getenv calls and one APCu read of the pool's hot record ([poll after, armed probes]). When nothing is armed and no poll is due, it returns without loading a class.
Method probes (Class::method) use the OpenTelemetry extension's hook() (pecl install opentelemetry). The extension calls the hooks around the method; the client registers hooks only for armed probes.
Probe points work without any extension. Probe points are named sites you declare once: Probes::point('checkout.total', ['cart' => $cart]). A point no probe targets costs one isset.
No network on the request path, ever. One worker of the pool is elected with an APCu lease. Once per poll interval, after its response has been sent (fastcgi_finish_request), it drains the telemetry spool, polls the probe set, reports statuses and exports OTLP.
- Each call has a 2 s time limit, and the whole run has a 4 s limit.
- A failing backend is retried with exponential backoff, up to 5 minutes. Every worker honours it, because the schedule is in shared memory.
- The backend's host name is resolved once every 5 minutes, not on every call.
Requests only append. A request whose probe fired appends one line to a local spool file at shutdown. The spool is capped at 4 MiB, and it is drained on every run whether or not the export succeeds, so a dead collector never grows it.
Two budgets, both enforced across the pool:
- Rate: each probe evaluates at most 50 times a second for the whole pool (an APCu counter). Calls over the limit are rejected before any capture.
- Overhead:
  - Within a request, probes may spend at most 2% of the request's time, with a 0.2 ms floor. Probes are skipped for the rest of the request once it's spent.
  - Across requests, a probe that took more than 2% of the time of the requests it ran in (over at least 20 requests in a window) is switched off. Its status turns to error with the measured share, until the probe is saved again.
Capture runs no application code.
- Objects are read with get_mangled_object_vars(): no __get, __toString, __debugInfo, getters, ArrayAccess or iteration of user Traversables.
- Uninitialised lazy objects (PHP 8.4 lazy objects, Doctrine and ProxyManager proxies) are shown by type and never initialised, so a probe never issues a query.
- Values are bounded: depth 2, 20 entries, 256 characters, 30 variables, and a 2,000-node budget per capture.
- Values are redacted by name (password, token, session, …, plus the probe's own keywords).
- A full sample (every variable, the return value and the method's source) is taken at most once every 10 s per probe for the pool. Other evaluations copy only the variables the probe reads.
The client never fails the application. Every hook and the shutdown work are wrapped. A probe that fails reports error with the reason instead.
The agent's own HTTP stays out of your traces. It talks over a raw stream_socket_client, not curl or Guzzle, so OpenTelemetry's instrumentation never sees it.

Install

composer require sentinel/probes

php.ini (or the pool's php_admin_value):

auto_prepend_file = /app/vendor/sentinel/probes/bootstrap/prepend.php
; optional, php-fpm: compile the client once at pool start (see Overhead)
opcache.preload = /app/vendor/sentinel/probes/bootstrap/preload.php
opcache.preload_user = www-data

The pool's environment:

env[SENTINEL_PROBES_URL] = https://ingest.example.com/otlp/v1/probes
env[SENTINEL_API_KEY] = <ingest key>
env[SENTINEL_SERVICE] = checkout          ; or OTEL_SERVICE_NAME
env[SENTINEL_ENV] = production
env[SENTINEL_VERSION] = 1.4.2             ; optional
env[SENTINEL_OTLP_ENDPOINT] = http://otel-collector:4318   ; or OTEL_EXPORTER_OTLP_ENDPOINT
env[SENTINEL_PROBES_POLL_SECONDS] = 5     ; optional

Recommended: ext-apcu (the pool-wide state, rate limits and sampling; without it a small file in the temp directory, and rate limits per request), ext-opentelemetry (method probes) and open-telemetry/sdk (span probes join your traces; span-tag probes need a span to tag).

Overhead

Measured with tests/fpm/run.sh:

Setup: php-fpm 8.3 with 8 static workers, behind nginx, with opcache.
App: a request that does 1 ms of work.
Load: ab -n 40000 -c 8.
Machine: Intel Core Ultra 7 155H.
Runs: three; the table shows the mean, and runs varied ±4%.

Scenario	req/s	vs. no client	peak memory
no client	5,584	—	549 KB
client installed, no probes	5,526	−1%	+2–3 KB
3 probes on a method every request calls	5,256	−6%	+13–20 KB
the same, with `opcache.preload`	5,416	−3%	+21 KB
backend answering 503 / unreachable	5,651 / 5,446	noise	+1 KB

The probes scenario is the worst case: three probes on a method that every request calls, 5,500 times a second, so most calls are rejected by the rate limit. The memory figure is per request, released at its end.

The pool polled once per interval, not once per request: 2–3 polls in an 8 s run at a 5 s interval. A backend answering 503 was retried with backoff.

Per call (tests/bench.php, CLI with opcache and APCu):

Path	µs
the prepend, nothing armed (loads no class)	0.4
booting the client (only when a probe is armed or a poll is due)	2.3–2.5
`Probes::point` with no probe on it	0.03
boot + a probe call over its rate limit	5–7
boot + a probe evaluated (median)	4–5
boot + a probe evaluated, p99 (includes the 10-second full sample)	50–80

Continuous profiling

Set SENTINEL_PROFILING_ENABLED=true and, with the excimer extension and APCu, the client also profiles CPU. Profiles go to Sentinel once a minute per pool and appear on the service's Profiling tab.

Setting	Default	Meaning
`SENTINEL_PROFILING_URL`	beside `SENTINEL_PROBES_URL`	`…/otlp/v1/profiles/pprof`
`SENTINEL_PROFILING_REQUESTS_PER_SECOND`	`10`	requests profiled per second, per pool
`SENTINEL_PROFILING_INTERVAL_MS`	`10`	sampling interval inside a profiled request
`SENTINEL_PROFILING_MAX_OVERHEAD`	`0.01`	the most of the pool's request time it may use
`SENTINEL_PROFILING_CLI`	off	also profile long-running CLI workers

How it stays cheap:

A few requests, not all. The prepend counts requests in APCu (one increment) and profiles only the first 10 each second in the pool. Every other request pays the counter and nothing else. The profile is scaled to all requests.
After the response. A profiled request's samples are folded after fastcgi_finish_request, so the visitor never waits for them.
A budget it enforces on itself. The pool meters the profiler's work against its own request time. Over 1%, profiling switches off for 10 minutes, and logs why, once.
Bounded. 1,000 samples and 64 stacks per request, 5,000 stacks per pool a minute, and one worker uploads per minute.

Per request (tests/ProfilerTest.php microbench, opcache and APCu):

Path	µs
profiling off	0.05
on, request not selected	0.45
on, request profiled (10/s a pool)	33

Under load (tests/fpm/profile.sh, kernel CPU per request of the fpm workers), profiling on and off differed by less than the run-to-run noise (±5% between two identical runs).

What runs where

Runtime	Status
php-fpm 8.1+	Supported; measured above.
Apache `mod_php`	Supported. There is no `fastcgi_finish_request`, so the one request per interval that runs the flusher sends its response after it: at most the 4 s run limit, and no wait when the backend is down because of backoff.
CLI scripts	Off unless `SENTINEL_PROBES_CLI=1`: a script exits when it is done, and a poll at its shutdown would hold it open.
Long-running workers (queues, RoadRunner, Swoole, FrankenPHP worker mode)	Call `Agent::tick()` between jobs. It spools, flushes when due and re-arms. One limit: the OpenTelemetry extension hooks a method only if the hook exists before the worker first calls it, so a probe added later on a method the worker has already run takes effect at the next worker restart. Probe points have no such limit.
Line probes	Not supported. They need a Zend extension that hooks individual opcodes. A probe on a line reports `error` and says so.

Without APCu the probe set lives in a small file that the page cache keeps hot. Rate limits then apply per request, not per pool. Install APCu for production.

Tests

The tests run in Docker; no Composer packages are needed.

php tests/PatternTest.php       # 3,038 regex verdicts, shared with every client
php tests/ConformanceTest.php   # the expression cases, shared with every client
php -d apc.enable_cli=1 tests/StoreTest.php && php -d apc.enabled=0 tests/StoreTest.php
sh tests/integration/run.sh     # end to end against a fake backend
sh tests/fpm/run.sh             # the overhead table (php-fpm + nginx + ab)
php -d apc.enable_cli=1 -d opcache.enable_cli=1 tests/bench.php

The end-to-end test covers:

method and point probes, with condition, metric, span and span-tag types;
statuses, including the refusals of line probes and newer expression versions;
samples: parameters, object fields read without __get, redaction, return value, duration and source;
the OTLP export: a gauge, a count as a delta sum, and one span per call with error status on a throw;
one poll per interval for the pool.

sentinel / probes

Package info

Security

Statistics