README

A pinned, reproducible Drupal site with known ground-truth state, used as the target AI agents are evaluated against. You point an agent at a real Drupal, let it build, and grade what it actually built: the saved config, the executed view, the persisted Canvas tree. Not what the model says it did.

Project page: drupal.org/project/ai_proving_ground.

Why this exists

Most AI-agent evals for Drupal run in an empty folder. They grade the text the model produces, so a plausible answer scores well even when the build it describes would not work. The teams building Drupal's eval tooling (ai_eval, ai_bench, the Inspect AI runner, the behavioral skill evals) all hit the same missing piece: a shared, resettable site with known contents that graders can check against. This project is that piece.

Three properties make a site usable as an eval target, and they are the whole design:

Known state. Every content item and config object the graders rely on is pinned in a fixture: fixed UUIDs, recorded fields, declared counts. A doctor command verifies the live site still matches, and says exactly what drifted when it does not.
Cheap reset. Runs mutate the site; the next run needs the same starting point. Provisioning captures a baseline (a database dump plus a git ref) keyed by a content hash of the exact fixture set, and reset restores it in seconds. Same set, same baseline, every time.
One grader, both directions. The droost_state grader from ai_eval_droost (the Droost-backed grader companion to ai_eval, not yet published on drupal.org) reads live state through Droost's read and validate tools. The ai_eval runtime calls it in-process (inside-in). An external harness such as Inspect AI calls the same grader over drush through ddev ai-eval-grade and gets an every_eval_ever 0.2.2 envelope back (outside-in). The scores are comparable because the grading code is identical.

How it fits together

The bed is not one fixed site. It is a base recipe (Canvas plus the eval stack) and a library of fixture recipes. Each fixture seeds one coherent content and config set and declares what it owns: a set of config name prefixes and a content-UUID namespace. A target names the fixtures it needs, pinned as name@version; the bed composes them and refuses to compose fixtures whose ownership claims overlap.

recipe.yml            # BASE recipe: Canvas + eval stack, no target content
fixtures/
  fixture_kb_docs/    # a fixture = recipe + config + content + manifest fragment
    recipe.yml
    config/           # pinned config objects (exported, not hand-written)
    content/          # pinned content, fixed UUIDs (core default-content format)
    manifest/fragment.yml   # what this fixture OWNS + machine-written facts
  fixture_canvas_home/
manifest/             # example of the unioned manifest the doctor reads
ddev/
  commands/           # the five ddev commands (see below)
  agents/             # run adapters; the shipped one drives nothing itself
scripts/              # hash, fixture-export, manifest union, doctor, task, grade
skills/               # operator skills: extend the bed and run evals
docs/                 # design amendments and live verification evidence

Two fixtures ship today. kb_docs is a small knowledge base: the kb_doc content type, four fields, and 12 published nodes whose sections and titles the graded tasks depend on. canvas_home is the site's front page as a Canvas page entity plus its two page regions. Together they back the first graded dataset (a Views build task and a Canvas build task, in ai_eval_droost's droost_build_tasks dataset).

Getting started

The outer bootstrap is delegated to one_line_installer: it stands up DDEV, Drupal CMS (which includes Canvas), and an agent container, unattended. This exact sequence has been run end to end; the evidence is in docs/disposable-bed-proof.md.

# 1. A fresh Drupal CMS site with droost, fully unattended.
bash <(curl -fsSL https://aibp.drupalstarforge.ai/install.sh) \
  --yolo --name my-bed --extras droost
cd my-bed

# 2. The eval stack. Until ai_eval_droost ships a release, copy or
#    composer-require ai_eval and ai_eval_droost into web/modules/contrib/.

# 3. This recipe and its commands.
#    Copy the repository to recipes/ai_proving_ground, then:
mkdir -p .ddev/commands/web .ddev/commands/host .ddev/agents
cp recipes/ai_proving_ground/ddev/commands/web/ai-eval-* .ddev/commands/web/
cp recipes/ai_proving_ground/ddev/commands/host/ai-eval-* .ddev/commands/host/
cp recipes/ai_proving_ground/ddev/agents/* .ddev/agents/
chmod +x .ddev/commands/host/ai-eval-* .ddev/agents/*

# 4. Baselines need a git tree. The installer does not create one:
# Drupal CMS ignores /recipes; baselines need the whole directory tracked.
sed -i '/^\/recipes$/d' .gitignore
git init -b main && git add -A && git commit -m "bed initial state"

# 5. Provision: base recipe + fixtures + baseline capture.
ddev ai-eval-provision kb_docs canvas_home

# 6. Verify, then use.
ddev ai-eval-doctor kb_docs canvas_home
ddev ai-eval-grade dv01_view_build

From there the loop is: reset, let the agent act, grade, reset. The middle step has a paved path, ddev ai-eval-run <task>, described under Running an agent below; nothing stops an external harness from driving the bed itself.

Commands

Commands take the fixture set as arguments (default kb_docs canvas_home).

Command	Job
`ddev ai-eval-provision [fixture...]`	Apply the base recipe and the named fixtures, then capture baseline B for that set: a DB dump plus a git tag, keyed by the content hash of base version + sorted `fixture@version` list. No-op when that hash already has both halves of its pair.
`ddev ai-eval-reset [fixture...]`	Restore baseline B for the set: scoped git tree (never `vendor/` or `settings.local.php`), then the DB dump, then `drush cr`. Derives the same hash; never falls back to a naive key.
`ddev ai-eval-run <task-id> [--agent=<name>] [--dataset=<ref>] [--model=<id>]`	Resolve a task to its prompt, hand it to an agent adapter, and record the attempt in a run manifest. Never grades. See Running an agent below.
`ddev ai-eval-grade <task-id> [dataset] [model-id] [--run=<run-id>]`	Grade one task against live state with the `droost_state` grader and print an every_eval_ever 0.2.2 envelope on stdout, then store it so the verdict appears in ai_eval's results UI (`AI_PG_NO_IMPORT=1` prints without storing; see Where verdicts go below). Exit 0 means a verdict was emitted, pass and fail alike; exit 3 means the question has no droost spec so nothing was graded (stderr carries `APG_GRADE_SKIP`); other non-zero means the grade could not run. Read the exit-code caveat below before relying on 3. The optional model-id stamps `model_info` (outside-in, only the driving harness knows which model acted).
`ddev ai-eval-doctor [fixture...]`	Assert live state matches the union of the set's manifest fragments (installed modules, entity counts, every declared UUID resolving with the declared label) and that no two fixtures claim the same config prefix or UUID namespace. JSON verdict, exit 0 when healthy.

Bed git requirements

Reset only restores files git tracks in the baseline ref under config/sync, web/modules/custom, and recipes. A stock Drupal project must remove or override the /recipes ignore rule, then add and commit the recipe files before provisioning. Absent or empty paths are allowed. A populated path with no files tracked in the baseline is refused with APG_PROVISION_UNTRACKED or APG_RESET_UNTRACKED; the diagnostic identifies the path and matching ignore rule.

Set AI_PG_ALLOW_UNTRACKED=1 inside the web container to accept DB-only restoration for untracked paths. This prints a warning and does not disable grader checks. Provision records grader.<set>.sha256 and grader.<set>.files beside the DB dump. The recipe and the Drupal-resolved ai_eval (including ai_eval_droost) and droost module trees are fingerprinted, excluding .git and .ai-proving-ground state/run output. Contrib grader code is checked, not restored. Reset fails with APG_RESET_GRADER_DRIFT and changed paths if the restored bed still differs; doctor reports a failing grader_tree check. Do not grade that bed.

Older baselines without a fingerprint remain usable: reset warns with APG_RESET_GRADER_UNRECORDED, and doctor reports a non-failing "not recorded, re-provision to record" note. To record one, remove the matching dump and baseline tag as described by provision, then provision again from the accepted clean tree.

Code upgrades

The fixture-set hash stays unchanged across code upgrades. Each new baseline records versions.<set>.json beside its dump: Drupal core and PHP versions, modules from the base and selected fixture recipes, and ai_eval, ai_eval_droost, and droost. Module code identity uses the containing Composer package name, version, and source reference when available, then the module's info.yml version, then unknown. Core module code versions are covered by the core entry. Installed database schema versions are recorded for all these modules for diagnosis, but are excluded from code comparisons.

On an existing baseline, provision refuses code differences with APG_PROVISION_STALE_BASELINE and lists recorded and live versions. AI_PG_ALLOW_STALE_BASELINE=1 warns and preserves the existing no-op; it does not refresh the dump or manifest. Older baselines without a manifest warn with APG_PROVISION_VERSIONS_UNRECORDED. Doctor's code_versions check fails on code differences; missing manifests are a non-failing "not recorded, re-provision to record" detail.

After restoring the database and rebuilding caches, reset warns about code differences with APG_RESET_CODE_DRIFT. It checks pending database updates and fails with APG_RESET_PENDING_UPDATES, listing modules and updates, if the restored database is behind the code. It never runs updates automatically. Do not grade a bed after a failed reset.

After upgrading code, bring the disposable bed's database up to date explicitly with ddev drush updb -y, and verify the intended fixture state. Remove the matching dump and baseline tag named in provision's recapture instruction, then run ddev ai-eval-provision with the same fixtures to capture a new baseline and manifest. This reapplies recipes; if the existing state cannot safely be reapplied, rebuild the disposable bed first. Run ddev ai-eval-doctor afterward. Keep a backup of the old baseline pair if you need to return to the old code.

Where verdicts go

ddev ai-eval-grade prints its envelope on stdout and also stores it, so an outside-in verdict shows up at /admin/config/ai/ai-eval/results beside the runs ai_eval drove itself. That is the comparison the grading seam exists to make: same grader, both directions, one screen. The grade already runs inside Drupal, so it hands the envelope straight to ai_eval's importer with no trip through a file.

Storing can never change the outcome. It happens after stdout is finished, it reports only on stderr, and it swallows its own failures: a storage problem must not masquerade as a grading failure, nor turn an emitted verdict into a non-verdict. If the envelope cannot be stored, the verdict on stdout still stands and stderr reports APG_GRADE_NOT_STORED <reason>. This includes importer exceptions, rejections, and conflicts. Exit 0 still means a verdict was emitted. An intentional skip or absent importer does not emit this token.

ddev ai-eval-grade dv01_view_build --run="$RUN_ID"  # grade a recorded attempt
ddev ai-eval-grade dv01_view_build              # print and store
AI_PG_NO_IMPORT=1 ddev ai-eval-grade dv01_view_build   # print only

Set AI_PG_NO_IMPORT=1 when an external harness owns persistence, or when you are grading repeatedly and do not want the rows.

Two things worth knowing before you rely on it:

Storing needs a new enough ai_eval. The importer arrived in ai_eval issue #3594739. On an older ai_eval the grade prints as it always did and says once on stderr that it could not store. The base recipe does not pin a version that guarantees it, so this is a normal state rather than an error.
Run identity controls deduplication. With --run, the evaluation ID ends in the run ID. An identical re-grade is already stored. A changed measurement for that ID is an import conflict. Without --run, the ID ends in a timestamp and four random bytes encoded as hex.

--run=<run-id> can appear anywhere in the arguments. Grade reads run.json from ${AI_PG_BASELINE_DIR:-<approot>/.ai-proving-ground}/runs/<run-id>/. It uses the recorded model when the model argument is empty. A conflicting model, a different task, an invalid run ID, or missing or unreadable run metadata emits APG_GRADE_RUN_INVALID <reason> on stderr. No envelope is emitted and the wrapper exits 1.

The envelope records apg_run_id in source_metadata.additional_details. Grade also saves it as grade.json in that run directory using a temporary file and rename. Re-grading replaces that file. A failed save emits APG_GRADE_NOT_SAVED <reason> on stderr but does not fail the grade. The run directory, including run.json and grade.json, survives ai-eval-reset. Reset restores the database and only config/sync, web/modules/custom, and recipes.

Check hashes in the envelope

source_metadata.additional_details records:

dataset_path: the FileSource-selected file, relative to the Drupal app root when inside it, otherwise absolute.
dataset_sha256: SHA-256 of that file's bytes.
task_sha256: SHA-256 of the matching questions[] entry parsed from those same bytes. Recursively sort associative-array keys, preserve list order, then JSON-encode with JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_PRESERVE_ZERO_FRACTION | JSON_THROW_ON_ERROR.

eval_library.additional_details records:

grader_class_file: the resolved droost_state class file, relative to the app root when inside it, otherwise absolute.
grader_class_sha256: SHA-256 of that class file.
grader_tree_sha256: the grade-time fingerprint from the shared grader-tree helper, covering the recipe and installed ai_eval/droost trees.
hash_algorithm: sha256.

Verify file hashes with sha256sum <dataset file> and sha256sum <class file>. Recompute the canonical task hash to compare one check across dataset edits. Compare grader_tree_sha256 with the bed's .ai-proving-ground/grader.<set>.sha256 (or the escrow's copy). Provision must have recorded the same code; installing new grader code changes that fingerprint. If dataset resolution, parsing, or the loaded task's ID/droost cross-check fails, all three dataset keys are omitted and dataset_provenance says unresolved: <reason>. A tree failure omits its hash and sets grader_tree; a class-file failure sets grader_class. Failures warn on stderr with APG_GRADE_DATASET_UNRESOLVED or APG_GRADE_GRADER_UNRESOLVED and do not change the verdict.

These hashes do not authenticate the envelope: it is agent-writable after the fact, so the harness must keep the stdout it captured (see the escrow section). grader_tree_sha256 is computed by code inside the bed and is only as trustworthy as an ai-eval-verify run before grading.

Running an agent

ddev ai-eval-run <task> closes the gap between reset and grade. It resolves the task through the same dataset loader ai-eval-grade uses, so run and grade can never disagree about which task is which, writes a run directory under .ai-proving-ground/runs/, and hands the prompt to an adapter. It never grades: a failed attempt and a failed grade are different facts.

Two properties are non-negotiable for an eval bed, and the command enforces both rather than asking you to remember them.

Vanilla by default. The shipped adapter, manual, drives nothing. It prints the prompt and waits for you to run whatever agent you are actually evaluating. A bed that shipped a tuned agent, skill set or rule set would be grading its own configuration; tuning is what this bed measures. Selecting a real runtime is always explicit (--agent, or AI_PG_AGENT). With no terminal attached, manual records no attempt and exits non-zero rather than claim one happened.

Bed-scoped credentials. A bed runs untrusted attempts by definition, so an agent running against it must not hold the host's credentials. Before running an adapter that needs credentials, the command inspects the bed's own DDEV config for host-wide credential sharing and refuses with APG_RUN_CRED_REFUSED when it finds any. The preflight detects volume sources outside the project (home paths, variable expansions unless known to resolve inside the project, and absolute host paths), host Docker socket mounts, and credential-shaped environment names in config*.yaml, .ddev/.env, and .ddev/.env.* (excluding .env.example). It reports names and source paths, never secret values. AI_PG_ALLOW_HOST_CREDS=1 remains the deliberate override. This is not hypothetical: a bed built by one_line_installer with the ddev-assistant-claude add-on bind-mounts the whole host ~/.claude, credentials included, into the web container.

An adapter is any executable in .ddev/agents/ (or the recipe's ddev/agents/). It declares its credential needs in a header line the same way a ddev command declares its description:

#!/usr/bin/env bash
## Description: what this adapter drives
## Credentials: none | bed-scoped | host-shared

A missing or unrecognised declaration reads as unknown and is treated as needing credentials. Fail closed: a bed that guesses in the permissive direction is the exact failure this project exists to catch. The adapter receives the run request by environment and file, never on stdin, so an interactive agent CLI keeps the terminal:

Variable	Meaning
`APG_RUN_DIR`	the run directory; write anything worth keeping here
`APG_PROMPT_FILE`	the task prompt, verbatim
`APG_REQUEST_FILE`	the full run request as JSON
`APG_TASK_ID`, `APG_DATASET`, `APG_MODEL`	what is being attempted
`APG_APPROOT`, `APG_DDEV_PROJECT`	the bed to act on

Exit 0 from an adapter means an attempt completed. Anything else is recorded as no attempt.

.ai-proving-ground/runs/<run-id>/run.json is the authoritative record of a run: task, dataset, adapter, model, timestamps, adapter exit code, whether an attempt happened, the prompt's sha256, and what the credential preflight found. It sits beside the baselines, which ai-eval-reset does not restore, so the record of an attempt survives the reset that follows it. Read it rather than parsing terminal output, and cite it from an envelope when a run needs provenance.

Versioning rule: bump a fixture's version: in its fragment whenever its exported content changes, and the base version under extra: ai_proving_ground: version in recipe.yml whenever the base changes. The hash does the rest: a bumped version is a new baseline, an unchanged set reuses the cached one.

Fixtures are exported, not hand-written

scripts/fixture-export.php runs on a source site and captures what the fragment declares: the owned config objects (exact names must exist, trailing-dot prefixes expand) and the content matched by the fragment's export.content selectors, in core's DefaultContent format with fixed UUIDs. It then regenerates the fragment's machine-written facts (counts, per-node records). Three keys stay hand-maintained: version, owns, and export. Ground truth is keyed on UUID, never on nid: node IDs are serials assigned at import time and they do not survive re-provisioning. The first disposable-bed run proved this the hard way; see finding 4 in docs/disposable-bed-proof.md.

To add a fixture or a graded task, read skills/extending-the-proving-ground/SKILL.md. It is written as an agent skill, so both people and coding agents can follow it.

Your own tests

Keep your fixtures in your project's recipes/fixture_<name>/, tracked in that project's git so they survive APG Composer updates and reset restores them. Alternatively, set AI_PG_FIXTURE_PATH to colon-separated directories that contain fixture_<name>/ folders. Relative entries are relative to the app root (DDEV_APPROOT, normally /var/www/html), not the Drupal docroot. APG commands run inside the web container, so set it there: ddev config --web-environment-add=AI_PG_FIXTURE_PATH=my-fixtures and ddev restart. Most projects do not need it: recipes/fixture_<name>/ works without any setting.

Lookup searches those entries, then the project's recipes/, then APG's shipped fixtures/. A hit requires recipe.yml. Two distinct directories with the same fixture name fail with APG_FIXTURE_AMBIGUOUS; use a new name for a variant. Symlinks to the same real directory count as one copy.

Export with scripts/fixture-export.php as described in the fixture guide. For a new name it creates an empty scaffold in the first AI_PG_FIXTURE_PATH entry, or in the project's recipes/ by default. Edit its recipe install list, ownership and export selectors, then re-export from the live source site. Existing fixtures export back to their resolved directory.

Put your tasks in a dataset YAML in your own enabled module's datasets/ folder (AI Eval 1.0.0-beta4 or later), or configure ai_eval.settings: dataset_path. Track the dataset in your project's git. Pick each test's fixture set with command arguments, AI_PG_FIXTURES, or an AI Eval target's pins via drush ai-eval:target-fixtures <target> --names. Custom search roots outside reset's tracked paths are not restored by reset.

Target binding

AI Eval 1.0.0-beta2 and later ship EvalTarget fixtures pins such as kb_docs@2 and drush ai-eval:target-fixtures <target>. The command prints name@version pins; add --names to pass the short names to APG:

ddev ai-eval-provision $(ddev drush ai-eval:target-fixtures my_target --names)

The target records the required versions; --names only selects the set, so compare the pins with the fixtures' manifest versions when updating them.

Status

Honest state as of 2026-07-18: the loop works and has been exercised twice over. Every command above ran green on the source site and again on a fresh disposable bed built by one_line_installer on Drupal 11.4.4, where the first real run surfaced five defects (all fixed here; the UUID assertion fix shipped in AI Eval 1.0.0-beta3). Grade envelopes validate against the reference every_eval_ever models. Not yet done: the drupal.org registration (gates the final name), the bug/fix rotation fixture, and the integrity layer for unattended agent runs (baseline escrow plus tamper-evident grading; the evaluation that scoped it is docs/milestone-8-evaluation.md).

Verification evidence lives in docs/: milestone-3.md (baseline cache), milestone-6.md (grading seam), disposable-bed-proof.md (the full loop), milestone-8-evaluation.md (isolation options compared), and experiments/2026-08-13-view-discrimination/ (the worked A/B proof that a config-valid build can still be functionally wrong and only execution catches it, with envelopes, provenance hashes, and a copy-paste reproduce script that works on released versions, ai_eval 1.0.0-beta3 or later).

Implementation notes for contributors

DDEV collapses custom-command exit codes. ddev <command> returns 1 for every non-zero exit a custom command makes, host and web alike; the real code survives only as text on stderr (error=exit status 3). Verified on DDEV v1.25.1. ddev exec is faithful by contrast. So the documented codes above are what the scripts return, not what a caller of ddev ... observes: every distinguishable outcome also emits a stable stderr token (APG_GRADE_SKIP, APG_RUN_CRED_REFUSED, and so on), and a harness that needs a machine-readable result should read run.json or the grade envelope rather than an exit code.
All scripts run through `drush php:script --script-path=/scripts.php -- `; arguments arrive in `$extra`. Success paths fall through without `exit(0)`, because drush reports any explicit `exit()` as abnormal termination, and it collapses every explicit `exit(N)` to code 1. A script that needs to signal more than pass/fail emits a stable stderr token instead; the grade wrapper maps `APG_GRADE_SKIP` back to exit 3 outside the drush layer.
Recipe validation checks the install list against Drupal's cached extension list, so provision rebuilds caches before applying recipes.
On eCryptfs hosts, the DDEV bind mount can serve stale views of in-place file rewrites across the host/container boundary. Scripts here write new-file-then-rename. If an edited file looks stale on the other side, force a fresh inode: cp file .tmp && mv -f .tmp file.

License

GPL-2.0-or-later.

What reset restores

State	Reset behavior
Database	Restored from the baseline dump
Tracked paths	Restored from the baseline git ref
Public files	Restored from `files.<set>.tar.gz` to the recorded app-relative public directory
Private files	Not restored; reset prints `APG_RESET_PRIVATE_FILES_NOT_RESTORED <path>` when configured
Contrib/vendor/core	Checked, not restored: grader fingerprint and version manifest

Provision resolves public:// through Drupal and records the archive, a .json path/checksum sidecar, and a .list path listing beside the database dump. Top-level css/, js/, php/, and styles/ are regenerable derivatives: they are excluded from capture and doctor, and left untouched during reset. Doctor's public_files check compares paths, not file contents. Symlinks are archived as links and never followed during deletion. Stop concurrent writers before capturing or resetting a baseline.

Older baselines without a snapshot print APG_RESET_FILES_UNRECORDED and skip file restore; doctor reports a non-failing note. Provision's existing-baseline no-op does not backfill a snapshot. Recapture the baseline to enable file restore. Unsafe destinations or changed archives are refused before deleting public files.

Agents that run the evals

The operator agent drives provision, reset, run, verify, and grade from outside the tested project. The agent under test works inside that project.

Install skills/running-proving-ground-evals/ for running and grading attempts. Install skills/extending-the-proving-ground/ for adding fixtures and tasks. Install both only on the operator side, in the skills folder of the operator's own workspace outside the tested project. Do not use ~/.claude/skills/ when the agent under test is Claude Code on the same host account: user-level skills load in every project. Never install them in the tested project's .claude/skills/ or .agents/skills/, or give them to the agent under test. They describe the checks and where expected answers live.

Unattended runs: escrow and verify

Provision a trusted bed, then save its baseline on the host:

ddev ai-eval-provision kb_docs canvas_home
ddev ai-eval-escrow kb_docs canvas_home

The host commands require Bash 4+, Git, jq, and GNU coreutils/findutils. Escrow writes to ${AI_PG_ESCROW_DIR:-$HOME/.ai-proving-ground-escrow}/<project>/<set-key>/. It captures the dump, available sidecars (missing ones produce APG_ESCROW_SKIPPED), a baseline-tag bundle, metadata, and a verifier copy. The host verifier captures its own judging-tree hashes, including .ddev/commands/ and .ddev/agents/. Other .ddev/ files are not fingerprinted. An existing set requires explicit --force to replace it. Only do this after reinstalling or re-provisioning trusted code, never to accept an attempt's drift.

Escrow setup calls ddev exec drush php:script ... hash.php with the selected fixtures, just as provision does. This uses the actual YAML parser instead of parsing human-readable provision output. Subsequent verification must not run that PHP: it selects the set from trusted escrow metadata using sorted fixture names (arguments, AI_PG_FIXTURES, or kb_docs canvas_home). If several keys match, selection fails closed with APG_ESCROW_NONE; use the known absolute escrow verifier path. AI_PG_BASELINE_REF can narrow selection. Baseline directory and ref overrides must match provision at escrow time. Container paths starting with /var/www/html in AI_PG_BASELINE_DIR are translated to the host project. The saved destinations are used for subsequent repairs.

For each attempt, the command contract is:

ddev ai-eval-verify kb_docs canvas_home --restore
ddev ai-eval-reset kb_docs canvas_home
# Run the agent and wait for it and its child processes to stop.
ddev ai-eval-run <task-id> # Creates .ai-proving-ground/runs/<run-id>/.
ddev ai-eval-verify kb_docs canvas_home
# Only on successful verification:
ddev ai-eval-grade <task-id> --run=<run-id>

A nonzero verify means do not grade; record TAMPERED and preserve stderr as evidence. Baseline-file and tag differences produce APG_ESCROW_MISMATCH; --restore repairs those and reports APG_ESCROW_RESTORED. Judging-code changes, including added and removed files, produce APG_TAMPERED <path> <change> and always fail, even with --restore. Code is never auto-restored. A successful check ends with APG_ESCROW_OK.

The ddev dispatcher and its host wrapper are themselves agent-writable. The wrapper executes the escrow verifier, but an attempt can replace the wrapper before dispatch. Therefore an unattended harness must save the absolute escrow path returned by setup in its own trusted state and invoke that verifier directly for both checks above, using a trusted host Bash executable:

/usr/bin/bash "$ESCROW_SET_DIR/escrow-verify.sh" "$BED_APPROOT" "$ESCROW_SET_DIR" --restore
# Reset, run agent, stop agent, then:
/usr/bin/bash "$ESCROW_SET_DIR/escrow-verify.sh" "$BED_APPROOT" "$ESCROW_SET_DIR"

Keep harness code, these paths, and its environment outside the project. The agent must be stopped before verification and remain stopped through grading; this is a point-in-time check, not protection against concurrent writes.

Escrow refuses APG_ESCROW_UNSAFE if its resolved directory is inside the project or under a recognized bind source in .ddev/docker-compose.*.yaml. It recognizes conventional short volume syntax and long source: syntax, absolute paths, relative paths based on .ddev, ${HOME}, $HOME, and ~. It does not implement a general YAML or Compose evaluator: YAML aliases, inline mapping syntax, generated Compose overrides, and environment-file interpolation are not resolved. Unresolved dollar substitutions in recognized sources fail closed. The operator must inspect effective mounts for other syntax and mounts configured elsewhere. The escrow is only as safe as the host account's own mounts and permissions. Use a directory outside every container mount.

This does not hide ground truth: the agent can still read it. The agent can change the database, which is the graded deliverable. Grade envelopes and runs/<id>/grade.json remain inside the project and can be rewritten after the fact. A harness must retain the grade stdout it captured itself in trusted storage, together with verification evidence. Hashes are integrity evidence relative to the trusted escrow, not signatures against a compromised host.

Run the host-only regression suite with bash tests/escrow.test.sh. It creates a temporary Git bed and puts a stub ddev executable on PATH that returns a fixed fixture key only for the setup exec drush php:script call. No ddev, Docker, or Drush process is launched by the suite.

drupal / ai_proving_ground

Package info

Security

Statistics