README

A pinned, reproducible Drupal site with known ground-truth state, used as the target AI agents are evaluated against. You point an agent at a real Drupal, let it build, and grade what it actually built: the saved config, the executed view, the persisted Canvas tree. Not what the model says it did.

Project page: drupal.org/project/ai_proving_ground.

Why this exists

Most AI-agent evals for Drupal run in an empty folder. They grade the text the model produces, so a plausible answer scores well even when the build it describes would not work. The teams building Drupal's eval tooling (ai_eval, ai_bench, the Inspect AI runner, the behavioral skill evals) all hit the same missing piece: a shared, resettable site with known contents that graders can check against. This project is that piece.

Three properties make a site usable as an eval target, and they are the whole design:

Known state. Every content item and config object the graders rely on is pinned in a fixture: fixed UUIDs, recorded fields, declared counts. A doctor command verifies the live site still matches, and says exactly what drifted when it does not.
Cheap reset. Runs mutate the site; the next run needs the same starting point. Provisioning captures a baseline (a database dump plus a git ref) keyed by a content hash of the exact fixture set, and reset restores it in seconds. Same set, same baseline, every time.
One grader, both directions. The droost_state grader from ai_eval_droost (the Droost-backed grader companion to ai_eval, not yet published on drupal.org) reads live state through Droost's read and validate tools. The ai_eval runtime calls it in-process (inside-in). An external harness such as Inspect AI calls the same grader over drush through ddev ai-eval-grade and gets an every_eval_ever 0.2.2 envelope back (outside-in). The scores are comparable because the grading code is identical.

How it fits together

The bed is not one fixed site. It is a base recipe (Canvas plus the eval stack) and a library of fixture recipes. Each fixture seeds one coherent content and config set and declares what it owns: a set of config name prefixes and a content-UUID namespace. A target names the fixtures it needs, pinned as name@version; the bed composes them and refuses to compose fixtures whose ownership claims overlap.

recipe.yml            # BASE recipe: Canvas + eval stack, no target content
fixtures/
  fixture_kb_docs/    # a fixture = recipe + config + content + manifest fragment
    recipe.yml
    config/           # pinned config objects (exported, not hand-written)
    content/          # pinned content, fixed UUIDs (core default-content format)
    manifest/fragment.yml   # what this fixture OWNS + machine-written facts
  fixture_canvas_home/
manifest/             # example of the unioned manifest the doctor reads
ddev/
  commands/           # the five ddev commands (see below)
  agents/             # run adapters; the shipped one drives nothing itself
scripts/              # hash, fixture-export, manifest union, doctor, task, grade
skills/               # agent skill: how to extend the bed
docs/                 # design amendments and live verification evidence

Two fixtures ship today. kb_docs is a small knowledge base: the kb_doc content type, four fields, and 12 published nodes whose sections and titles the graded tasks depend on. canvas_home is the site's front page as a Canvas page entity plus its two page regions. Together they back the first graded dataset (a Views build task and a Canvas build task, in ai_eval_droost's droost_build_tasks dataset).

Getting started

The outer bootstrap is delegated to one_line_installer: it stands up DDEV, Drupal CMS (which includes Canvas), and an agent container, unattended. This exact sequence has been run end to end; the evidence is in docs/disposable-bed-proof.md.

# 1. A fresh Drupal CMS site with droost, fully unattended.
bash <(curl -fsSL https://aibp.drupalstarforge.ai/install.sh) \
  --yolo --name my-bed --extras droost
cd my-bed

# 2. The eval stack. Until ai_eval_droost ships a release, copy or
#    composer-require ai_eval and ai_eval_droost into web/modules/contrib/.

# 3. This recipe and its commands.
#    Copy the repository to recipes/ai_proving_ground, then:
mkdir -p .ddev/commands/web .ddev/commands/host .ddev/agents
cp recipes/ai_proving_ground/ddev/commands/web/ai-eval-* .ddev/commands/web/
cp recipes/ai_proving_ground/ddev/commands/host/ai-eval-* .ddev/commands/host/
cp recipes/ai_proving_ground/ddev/agents/* .ddev/agents/
chmod +x .ddev/commands/host/ai-eval-* .ddev/agents/*

# 4. Baselines need a git tree. The installer does not create one:
git init -b main && git add -A && git commit -m "bed initial state"

# 5. Provision: base recipe + fixtures + baseline capture.
ddev ai-eval-provision kb_docs canvas_home

# 6. Verify, then use.
ddev ai-eval-doctor kb_docs canvas_home
ddev ai-eval-grade dv01_view_build

From there the loop is: reset, let the agent act, grade, reset. The middle step has a paved path, ddev ai-eval-run <task>, described under Running an agent below; nothing stops an external harness from driving the bed itself.

Commands

Commands take the fixture set as arguments (default kb_docs canvas_home).

Command	Job
`ddev ai-eval-provision [fixture...]`	Apply the base recipe and the named fixtures, then capture baseline B for that set: a DB dump plus a git tag, keyed by the content hash of base version + sorted `fixture@version` list. No-op when that hash already has both halves of its pair.
`ddev ai-eval-reset [fixture...]`	Restore baseline B for the set: scoped git tree (never `vendor/` or `settings.local.php`), then the DB dump, then `drush cr`. Derives the same hash; never falls back to a naive key.
`ddev ai-eval-run <task-id> [--agent=<name>] [--dataset=<ref>] [--model=<id>]`	Resolve a task to its prompt, hand it to an agent adapter, and record the attempt in a run manifest. Never grades. See Running an agent below.
`ddev ai-eval-grade <task-id> [dataset] [model-id]`	Grade one task against live state with the `droost_state` grader and print an every_eval_ever 0.2.2 envelope on stdout, then store it so the verdict appears in ai_eval's results UI (`AI_PG_NO_IMPORT=1` prints without storing; see Where verdicts go below). Exit 0 means a verdict was emitted, pass and fail alike; exit 3 means the question has no droost spec so nothing was graded (stderr carries `APG_GRADE_SKIP`); other non-zero means the grade could not run. Read the exit-code caveat below before relying on 3. The optional model-id stamps `model_info` (outside-in, only the driving harness knows which model acted).
`ddev ai-eval-doctor [fixture...]`	Assert live state matches the union of the set's manifest fragments (installed modules, entity counts, every declared UUID resolving with the declared label) and that no two fixtures claim the same config prefix or UUID namespace. JSON verdict, exit 0 when healthy.

Where verdicts go

ddev ai-eval-grade prints its envelope on stdout and also stores it, so an outside-in verdict shows up at /admin/config/ai/ai-eval/results beside the runs ai_eval drove itself. That is the comparison the grading seam exists to make: same grader, both directions, one screen. The grade already runs inside Drupal, so it hands the envelope straight to ai_eval's importer with no trip through a file.

Storing can never change the outcome. It happens after stdout is finished, it reports only on stderr, and it swallows its own failures: a storage problem must not masquerade as a grading failure, nor turn an emitted verdict into a non-verdict. If the envelope cannot be stored, the verdict on stdout still stands and stderr says why.

ddev ai-eval-grade dv01_view_build              # print and store
AI_PG_NO_IMPORT=1 ddev ai-eval-grade dv01_view_build   # print only

Set AI_PG_NO_IMPORT=1 when an external harness owns persistence, or when you are grading repeatedly and do not want the rows.

Two things worth knowing before you rely on it:

Storing needs a new enough ai_eval. The importer arrived in ai_eval issue #3594739. On an older ai_eval the grade prints as it always did and says once on stderr that it could not store. The base recipe does not pin a version that guarantees it, so this is a normal state rather than an error.
Every grade is its own row. An envelope's evaluation_id carries a timestamp, so grading twice really is two measurements and the importer's deduplication will not collapse them. Re-importing the same envelope is a safe no-op; grading the same unchanged state ten times is ten rows.

Running an agent

ddev ai-eval-run <task> closes the gap between reset and grade. It resolves the task through the same dataset loader ai-eval-grade uses, so run and grade can never disagree about which task is which, writes a run directory under .ai-proving-ground/runs/, and hands the prompt to an adapter. It never grades: a failed attempt and a failed grade are different facts.

Two properties are non-negotiable for an eval bed, and the command enforces both rather than asking you to remember them.

Vanilla by default. The shipped adapter, manual, drives nothing. It prints the prompt and waits for you to run whatever agent you are actually evaluating. A bed that shipped a tuned agent, skill set or rule set would be grading its own configuration; tuning is what this bed measures. Selecting a real runtime is always explicit (--agent, or AI_PG_AGENT). With no terminal attached, manual records no attempt and exits non-zero rather than claim one happened.

Bed-scoped credentials. A bed runs untrusted attempts by definition, so an agent running against it must not hold the host's credentials. Before running an adapter that needs credentials, the command inspects the bed's own DDEV config for host-wide credential sharing (host home directories bind- mounted into the container, credential-shaped names in web_environment) and refuses with APG_RUN_CRED_REFUSED when it finds any. It reports the names it matched, never a value. AI_PG_ALLOW_HOST_CREDS=1 overrides it for a bed you accept as disposable. This is not hypothetical: a bed built by one_line_installer with the ddev-assistant-claude add-on bind-mounts the whole host ~/.claude, credentials included, into the web container.

An adapter is any executable in .ddev/agents/ (or the recipe's ddev/agents/). It declares its credential needs in a header line the same way a ddev command declares its description:

#!/usr/bin/env bash
## Description: what this adapter drives
## Credentials: none | bed-scoped | host-shared

A missing or unrecognised declaration reads as unknown and is treated as needing credentials. Fail closed: a bed that guesses in the permissive direction is the exact failure this project exists to catch. The adapter receives the run request by environment and file, never on stdin, so an interactive agent CLI keeps the terminal:

Variable	Meaning
`APG_RUN_DIR`	the run directory; write anything worth keeping here
`APG_PROMPT_FILE`	the task prompt, verbatim
`APG_REQUEST_FILE`	the full run request as JSON
`APG_TASK_ID`, `APG_DATASET`, `APG_MODEL`	what is being attempted
`APG_APPROOT`, `APG_DDEV_PROJECT`	the bed to act on

Exit 0 from an adapter means an attempt completed. Anything else is recorded as no attempt.

.ai-proving-ground/runs/<run-id>/run.json is the authoritative record of a run: task, dataset, adapter, model, timestamps, adapter exit code, whether an attempt happened, the prompt's sha256, and what the credential preflight found. It sits beside the baselines, which ai-eval-reset does not restore, so the record of an attempt survives the reset that follows it. Read it rather than parsing terminal output, and cite it from an envelope when a run needs provenance.

Versioning rule: bump a fixture's version: in its fragment whenever its exported content changes, and the base version under extra: ai_proving_ground: version in recipe.yml whenever the base changes. The hash does the rest: a bumped version is a new baseline, an unchanged set reuses the cached one.

Fixtures are exported, not hand-written

scripts/fixture-export.php runs on a source site and captures what the fragment declares: the owned config objects (exact names must exist, trailing-dot prefixes expand) and the content matched by the fragment's export.content selectors, in core's DefaultContent format with fixed UUIDs. It then regenerates the fragment's machine-written facts (counts, per-node records). Three keys stay hand-maintained: version, owns, and export. Ground truth is keyed on UUID, never on nid: node IDs are serials assigned at import time and they do not survive re-provisioning. The first disposable-bed run proved this the hard way; see finding 4 in docs/disposable-bed-proof.md.

To add a fixture or a graded task, read skills/extending-the-proving-ground/SKILL.md. It is written as an agent skill, so both people and coding agents can follow it.

Target binding

The eval target owns its environment binding: an ai_eval EvalTarget carries fixtures pins like kb_docs@2, and drush ai-eval:target-fixtures <target> prints them for the provision and reset commands to consume. That change is implemented and tested against ai_eval but not yet upstream; it is pending as an issue on the ai_eval queue. Until it lands, pass the fixture set on the command line.

Status

Honest state as of 2026-07-18: the loop works and has been exercised twice over. Every command above ran green on the source site and again on a fresh disposable bed built by one_line_installer on Drupal 11.4.4, where the first real run surfaced five defects (all fixed here, one pending upstream in ai_eval_droost). Grade envelopes validate against the reference every_eval_ever models. Not yet done: the drupal.org registration (gates the final name), the EvalTarget binding landing upstream, the bug/fix rotation fixture, and the integrity layer for unattended agent runs (baseline escrow plus tamper-evident grading; the evaluation that scoped it is docs/milestone-8-evaluation.md).

Verification evidence lives in docs/: milestone-3.md (baseline cache), milestone-6.md (grading seam), disposable-bed-proof.md (the full loop), milestone-8-evaluation.md (isolation options compared), and experiments/2026-08-13-view-discrimination/ (the worked A/B proof that a config-valid build can still be functionally wrong and only execution catches it, with envelopes, provenance hashes, and a copy-paste reproduce script that works on released versions, ai_eval 1.0.0-beta3 or later).

Implementation notes for contributors

DDEV collapses custom-command exit codes. ddev <command> returns 1 for every non-zero exit a custom command makes, host and web alike; the real code survives only as text on stderr (error=exit status 3). Verified on DDEV v1.25.1. ddev exec is faithful by contrast. So the documented codes above are what the scripts return, not what a caller of ddev ... observes: every distinguishable outcome also emits a stable stderr token (APG_GRADE_SKIP, APG_RUN_CRED_REFUSED, and so on), and a harness that needs a machine-readable result should read run.json or the grade envelope rather than an exit code.
All scripts run through `drush php:script --script-path=/scripts.php -- `; arguments arrive in `$extra`. Success paths fall through without `exit(0)`, because drush reports any explicit `exit()` as abnormal termination, and it collapses every explicit `exit(N)` to code 1. A script that needs to signal more than pass/fail emits a stable stderr token instead; the grade wrapper maps `APG_GRADE_SKIP` back to exit 3 outside the drush layer.
Recipe validation checks the install list against Drupal's cached extension list, so provision rebuilds caches before applying recipes.
On eCryptfs hosts, the DDEV bind mount can serve stale views of in-place file rewrites across the host/container boundary. Scripts here write new-file-then-rename. If an edited file looks stale on the other side, force a fresh inode: cp file .tmp && mv -f .tmp file.

License

GPL-2.0-or-later.

drupal / ai_proving_ground

Maintainers

Package info

Statistics

Security