README

mltstephane/data-hub is the autonomous collector used by Laravel applications to deliver logs, queue lifecycle events, and custom metrics to Hub. Laravel 12 and 13 are supported and tested. Laravel 11 remains a source-compatibility target only. The package requires PHP 8.2 or newer; each Laravel release's own PHP requirement still applies.

The collector is designed for at-least-once delivery. Collection failures, a full disk, queue failures, and an unavailable Hub are contained inside the package and never interrupt application work.

Installation

composer require mltstephane/data-hub
php artisan vendor:publish --tag=data-hub-config

Laravel package discovery registers the provider and the Metrics facade. Configure the application without committing its token:

DATA_HUB_ENABLED=true
DATA_HUB_ENDPOINT=https://hub.example.test
DATA_HUB_TOKEN=replace-with-the-ingestion-token

If the package is disabled, or if either endpoint or token is absent, it performs no collection, scheduling, dispatch, or disk write. Restart long-running workers after changing configuration.

The default spool is storage/framework/data-hub. Its path, event/byte limits, flush threshold, batch size (at most 100), queue delay, timeouts, retry policy, gzip, log level, and scheduler cron are configurable in config/data-hub.php.

The endpoint must be HTTPS without userinfo, query, or fragment. Port 443 is the default; any alternate port must appear exactly in transport.allowed_ports. transport.allowed_hosts can restrict public endpoints to exact hostnames. DNS A and AAAA answers are resolved and validated before every attempt, and all answers must be public. Private Hub deployments require an exact hostname or IP in transport.allowed_private_hosts; wildcards are rejected. Validated addresses are pinned with cURL CURLOPT_RESOLVE, preserving the endpoint hostname for Host, SNI, and certificate verification. If pinning is unavailable, transport fails closed. Redirects remain disabled and the bearer token is attached only after validation.

As of 2026-07-28, fresh Laravel 11/Testbench 9 installation and CI validation are blocked because every available Laravel 11 release is covered by active Composer security advisories. The package keeps constraints that target Laravel 11 source compatibility, but neither source compatibility nor runtime support can currently be guaranteed by an installable test job. CI therefore treats Laravel 11 as an explicit advisory probe rather than a supported validation job; it becomes a real audit and test job only if secure dependency resolution succeeds. Neither local tooling nor CI disables or ignores Composer's advisory policy.

Metrics API

use MltStephane\DataHub\Facades\Metrics;

Metrics::increment('orders.completed', by: 1, tags: ['region' => 'eu'], unit: 'order');
Metrics::gauge('queue.depth', 12, ['queue' => 'default']);
Metrics::observe('checkout.duration', 183.4, ['route' => 'checkout'], 'ms');

The exact public methods are:

Metrics::increment(string $name, int|float $by = 1, array $tags = [], ?string $unit = null): void;
Metrics::gauge(string $name, int|float $value, array $tags = [], ?string $unit = null): void;
Metrics::observe(string $name, int|float $value, array $tags = [], ?string $unit = null): void;

Counters may be positive or negative. Names, units, tag counts, keys, scalar values, and finite numeric values are validated. Invalid input is ignored and counted in the local diagnostic file; it never throws into client code.

Wire contract v1

The collector sends POST {endpoint}/api/v1/batches with Accept: application/json, JSON (optionally gzip-compressed), and the configured bearer token.

{
  "schema_version": 1,
  "batch_id": "stable UUID",
  "sent_at": "UTC ISO8601",
  "events": [
    {
      "event_id": "stable UUID",
      "type": "log|job|metric",
      "occurred_at": "UTC ISO8601",
      "payload": {}
    }
  ]
}

Each batch contains 1 to 100 events. Log payloads contain only level, message, environment, nullable channel, and a context restricted to request_id, trace_id, exception_class, url_path, method, status_code, job_name, and queue. environment is the explicitly required scalar application-environment label used by Logs Viewer; the collector never enumerates or reads arbitrary environment variables. Job payloads contain only logical_id, job_class, connection, queue, status, attempt, optional timestamps/duration, and cleaned exception class/message. Raw queue payloads are never requested, serialized, stored, or transmitted. Metric payloads contain only name, metric_type (counter, gauge, or histogram), finite value, nullable unit, and bounded scalar tags.

Hub must answer with:

{
  "batch_id": "the submitted batch UUID",
  "duplicate": false,
  "acknowledged_event_ids": ["explicitly persisted event UUID"]
}

Only the exact intersection of submitted and acknowledged event IDs is removed. A pre-existing valid manifest keeps its batch ID while its event set is unchanged. Without one, the collector derives a UUID-shaped version-5 hash from schema_version and the ordered event IDs, so a failed manifest write or crash before acknowledgement reconstructs the same batch ID. After a partial acknowledgement changes the remaining event set, the next batch receives the corresponding new deterministic ID. A crash after sending but before applying the acknowledgement therefore causes a safe duplicate delivery.

Logs and queue jobs

The package listens to MessageLogged; it never replaces or writes to Laravel log channels. Laravel does not guarantee a channel on that event. An event/context channel is accepted only when listed in logs.allowed_channels; otherwise the trusted configured logs.default_channel is used, or null. Channel exclusions therefore cannot infer an unavailable channel. Context is reduced to configured keys from the closed v1 allowlist.

Compatible Laravel queue processing, processed, retry/exception, and failed events emit started, succeeded, retrying, and failed. The collector never calls payload(), resolveName(), getRawBody(), or unserialize. Consequently, job_class is deliberately the safe queue wrapper/event-job class, not the application job's business class. Provider IDs may be strings or integers and become stable opaque logical IDs namespaced by connection and queue. Internal marker jobs and the dedicated local-affinity queue pair are excluded to prevent telemetry loops.

Flush and scheduling

php artisan hub:flush
php artisan hub:flush -v

The command always exits successfully, even when Hub is unavailable; unacknowledged data remains buffered. Verbose output contains counts/status codes owned by the package, never endpoint, token, payload, response body, or exception text.

The provider schedules hub:flush with the configured cron on every node. It intentionally uses no shared scheduler mutex: the spool is node-local and its non-blocking local flock prevents concurrent flushes for that spool. Run Laravel's scheduler on every collector node.

Automatic queue dispatch is disabled by default because a generic worker may execute on a different node from the local spool. It is enabled only when all four settings are explicit and valid:

DATA_HUB_DISPATCH_ASYNC=true
DATA_HUB_DISPATCH_LOCAL_AFFINITY=true
DATA_HUB_DISPATCH_CONNECTION=database
DATA_HUB_DISPATCH_QUEUE=hub-local

That connection/queue must be operationally pinned to workers on the same node and must not use the sync driver. Invalid opt-in configuration is diagnosed and never falls back to another queue. Without this opt-in, threshold and budget remainders wait for the next local scheduler run. No HTTP request runs in a business request or application job.

Flushes are bounded by both max_batches_per_flush and flush_time_budget_seconds (clamped below the internal job's 60-second timeout). HTTP 408, 429, network errors, and 5xx responses retain the active batch and use bounded exponential backoff/jitter. A valid Retry-After seconds value or HTTP date is honored up to retry_after_max_seconds. If the deadline is exhausted before a request or while deciding a retry/backoff, the explicit budget_exhausted result reaches the flush job; a remainder is redispatched only through valid local-affinity opt-in, otherwise it waits for the next local scheduler run.

Redaction and safety

Redaction runs before every spool write and again when an event is restored from disk. Collection and restoration use the same closed schema; unknown payload keys cause quarantine. Strings are first normalized as real Unicode text with NFKC when intl is available, or a conservative combining-mark/fullwidth fallback, then stripped of complete or incomplete ANSI sequences, controls, bidirectional characters, and CR/LF-forging content so those characters cannot fragment secret markers. Configured Authorization/Bearer, API-key, password, token, cookie, secret, and canary patterns are then redacted, followed by a second normalization and redaction pass before bounding and HTML escaping. Log messages and exception messages retain their documented hard size limits. Pattern redaction cannot identify an arbitrary unlabelled secret in free text; those fields therefore remain a residual disclosure risk, while configured sensitive fields and canary/labelled-secret forms receive the stronger handling. Headers, cookies, request bodies, arbitrary environment variables, and raw job payloads are never collected.

Saturation and diagnostics

The file spool is bounded by both event count and bytes. Priority is fixed as follows:

critical/error logs and failed jobs;
queue lifecycle events;
other logs and metrics.

The event and byte limits count pending and inflight files together. Hard internal ceilings additionally constrain configured counts, bytes, quarantine, response bodies, individual artifacts, text, and redaction patterns. When full, the oldest lower-priority pending event is evicted first. If none exists, the new event is dropped with a saturation diagnostic. Victims are durably journaled before deletion; if the journal cannot be written, the new event is dropped and no victim is removed. Once deletion starts, recovery completes that controlled commit without destructively rolling back the durable new event. Claimed events are atomically renamed into the pre-created inflight/ directory while the spool lock is held; the filename carries the batch ID and event ID. Inflight files remain in the total budget but are excluded from eviction, remain authoritative after transport failure or restart, and are removed only by exact acknowledgement or durable quarantine. A partial acknowledgement renames every remainder under its new deterministic batch ID before retry. A persistent monotone sequence orders new pending events deterministically even when timestamps are identical. Legacy v1 event files written before sequence and legacy state files written before next_sequence are read lazily and moved into inflight without content rewriting. Legacy events always sort before sequenced events and are ordered by created_at then filename/event ID. A pre-existing legacy manifest keeps its batch ID and event IDs. If a new manifest cannot be written, inflight filenames reconstruct its deterministic identity and transport still proceeds. Atomic writes use exclusive random temporary files, verify device/inode identity, loop over short writes, require complete content, flush and file-level fsync, verify chmod, rename, and attempt directory fsync; rename/unlink transitions likewise sync affected parent directories. Failure to obtain required durability is diagnosed and leaves dirty recovery state where applicable. File locks protect event files, inflight reservation, manifests, acknowledgement, diagnostics, and concurrent flushes; append retries contention three times within a bounded 10 ms budget.

Pending and inflight files are the source of truth. The state cache persists separate pending and inflight event/byte counters plus their totals. A healthy append reads this valid state cache in O(1): it writes dirty before publishing the event, then writes event and state, and clears dirty only after durable state success. It does not scan or decode the pending directory. Dirty, absent/invalid state, explicit claim, explicit stats, and saturation enforcement trigger bounded reconciliation scans. Exact acknowledged inflight files are unlinked first to release space, then a minimal native dirty marker, the remaining inflight identity, and state cache are persisted best-effort. A crash before dirty may temporarily leave an overcount, but cannot reuse next_sequence; claim, stats, saturation, or later recovery reconciles it. Dirty recovery returns reconstructed in-memory statistics when state persistence fails and retains dirty until a later state write succeeds.

Local diagnostics.json contains only bounded counters, a domain-level last_failure (sanitize_failed, buffer_failed, dispatch_failed, or transport_failed), a more specific package-owned last_error, and a timestamp. It never contains a token, endpoint, payload, response body, or raw exception message. The package never uses Laravel logging for its own diagnostics.

Invalid/truncated manifests and event files are moved to the bounded local storage/framework/data-hub/quarantine directory and valid remaining events are immediately rebuilt under a new batch UUID. HTTP 401/403 keeps the active batch blocked because every event is affected by authentication. HTTP 400/409/413/415/422 quarantines the rejected batch's manifest and event artifacts, then permits later batches to progress. Quarantine limits are quarantine.max_files and quarantine.max_bytes, with an effective minimum of 101 artifacts and one full buffer plus 1 MiB so a complete v1 batch and manifest fit. The oldest artifact is deleted only when required by those bounds, with an explicit quarantine_evictions diagnostic—never silently.

Quarantine artifacts may contain the already-sanitized event data and must retain spool-level filesystem protections. Recovery is deliberately operator-controlled: inspect artifacts offline, correct the endpoint/contract cause, restore only validated event JSON files to the node's events/ directory, remove the small state.json so it is rebuilt, then run php artisan hub:flush. Never paste quarantine content into logs or command output.

The spool path must be an absolute dedicated path without traversal or glob metacharacters. Existing components, root, files, and quarantine entries are checked with lstat/fstat; symlinks and unstable inode/device identities are rejected, regular reads are streamed with hard byte limits, and the owned root is forced to mode 0700. Portable PHP does not expose openat plus O_NOFOLLOW, so these before/after checks reduce but cannot eliminate TOCTOU against a privileged local filesystem attacker. Such an attacker is already outside the package's trust boundary.

For persistent delivery, monitor write permissions, quarantine growth, and free space for each node's spool; run Laravel's scheduler on every node; and alert on increasing diagnostics or saturation. Fixing Hub connectivity and running hub:flush retries retained active data.

Package development

composer validate --strict
composer run test:unit
composer run test:feature
composer run test
composer run lint:check
composer run lint
composer run quality

These commands are package-local and do not require or integrate with the Hub web application.

mltstephane / data-hub

Maintainers

Package info

Statistics

Security