README

Warning

Pre-release and experimental. The public API, cassette format, and request-matching logic are subject to change. Feedback is welcome via Issues and Pull Requests.

Deterministic evaluations, semantic assertions, and regression testing for LLM pipelines in PHPUnit—without flaky tests or recurring API costs.

The first time a test runs, LLM calls hit the real provider API and are recorded to a local JSON "cassette". Subsequent runs replay the recorded responses offline in milliseconds. You can assert on execution traces (tool calls, token counts), test for prompt regressions, and run LLM-as-a-judge checks whose verdicts are also cached for zero-cost CI runs.

Works out of the box with any PSR-18 HTTP client (Guzzle, Symfony HttpClient, openai-php/client, Anthropic SDK, Laravel Http, Prism). Compatible with OpenAI, Anthropic, Gemini, Ollama, and AWS Bedrock (streaming included).

Requirements: PHP 8.4+ and PHPUnit 12.5 or 13.

What You Can Test

1. Agent Tool Calling & Execution Traces

Verify that autonomous agents follow required policies, call tools in the correct order, and pass expected arguments:

$answer = $agent->handle('My order #404 arrived broken. Can I get my money back?');

self::assertToolCallOrder(['lookup_order', 'issue_refund']);
self::assertToolCalledTimes('issue_refund', 1, ['order_id' => 404]);
self::assertToolNotCalled('delete_account');

2. Hallucination & Semantic Grading (LLM-as-a-Judge)

For RAG pipelines, ensure output is strictly grounded in retrieved context. The judge model evaluates the answer once, and the evaluation score is committed with the cassette:

$docs = $retriever->search('How long do refunds take?');
$answer = $bot->answer('How long do refunds take?', $docs);

self::assertNoHallucination(['The customer asked how long refunds take.', ...$docs], $answer);
self::assertMatchesRubric('Gives a concrete number of days and stays polite', $answer);

3. Dataset Benchmarking & Schema Validation

Run evaluations across JSONL/CSV test suites and require an overall pass-rate threshold rather than fragile absolute matches:

self::assertPassRate(Dataset::load(__DIR__ . '/datasets/reviews.jsonl'), 0.8, function (array $case): void {
    $json = $this->classifier->classify($case['review']);

    self::assertMatchesJsonSchema(self::SCHEMA, $json);
    self::assertSame($case['expected'], json_decode($json, true)['sentiment']);
});

4. Catching Prompt Regressions in Pull Requests

In CI, tests run strictly in replay mode. Any prompt modification alters the semantic hash, failing the test with a readable diff:

No recorded agent_call with semantic_hash 3f2a… in cassette.
Cassette: tests/fixtures/ai-cassettes/App_Tests_RefundAgentTest_testRefund.json

Diff against closest recorded interaction (- recorded, + current):
-  "content": "You are a support agent."
+  "content": "You are a friendly support agent."

Re-record with: AI_EVAL_MODE=record vendor/bin/phpunit --filter 'RefundAgentTest::testRefund\b'

5. Token & Latency Budgets

Prevent prompt bloat and runaway loops before hitting production:

self::assertTokenUsageLessThan(2000); // Total prompt + completion tokens
self::assertLatencyLessThan(15_000);  // Milliseconds recorded during initial run

6. Sampling Non-Deterministic Behaviors

For non-deterministic tasks, record multiple runs and enforce an acceptable passing ratio:

self::assertSampled(runs: 5, minPass: 4, testLogic: function (): void {
    $this->agent->handle('My order #404 arrived broken.');
    self::assertToolCalled('issue_refund');
});

Quick Start

Installation

composer require --dev axitdev/phpunit-ai-evals

Note: Requires a PSR-17 factory implementation. If your application does not have one, install composer require --dev nyholm/psr7.

1. Register the PHPUnit Extension

Add the extension to phpunit.xml:

<extensions>
    <bootstrap class="Axitdev\PhpUnitAiEvals\PhpUnit\AiEvalsExtension">
        <parameter name="cassettesPath" value="tests/fixtures/ai-cassettes"/>
    </bootstrap>
</extensions>

2. Wrap Your Client & Write Tests

use Axitdev\PhpUnitAiEvals\Assertions\AiAssertionsTrait;
use PHPUnit\Framework\TestCase;

final class RefundAgentTest extends TestCase
{
    use AiAssertionsTrait;

    public function testRefundsADamagedOrder(): void
    {
        // Wrap the standard PSR-18 client used by your agent/SDK
        $agent = new RefundAgent($this->wrapHttpClient(new \GuzzleHttp\Client()));

        $answer = $agent->handle('Refund order #404, it arrived broken');

        self::assertToolCalled('issue_refund', ['order_id' => 404]);
        self::assertMatchesRubric('Confirms the refund politely', $answer);
    }
}

3. Record and Replay

Run tests with provider API keys configured. The run generates a cassette at tests/fixtures/ai-cassettes/<Class>_<method>.json:

# Subsequent runs replay offline:
vendor/bin/phpunit --filter testRefundsADamagedOrder

# Re-record when prompts or models change:
AI_EVAL_MODE=record vendor/bin/phpunit --filter testRefundsADamagedOrder

# Clean up unused cassettes:
AI_EVAL_PRUNE=1 vendor/bin/phpunit

Each run displays an evaluation summary:

AI Evals Summary:
  Interactions: 14 total (12 VCR hits, 2 live calls)
  Tokens Used:  4,210 (Prompt: 3,400 / Completion: 810 / Reasoning: 120)
  Judge Evals:  8 passed (Avg Score: 0.94)

Working examples are available in the examples repository.

Documentation

FAQ — Common questions and troubleshooting.
Integrations — Guzzle, Symfony, openai-php, Anthropic SDK, Laravel Http, Prism, and local models.
Assertions Reference — Full assertion list, datasets, and probabilistic sampling.
Configuration — Modes, cassette storage, judge setup, reports, and parallel test execution.
How It Works — Request hashing, cassette formats, secret redaction, streaming, and provider support.

Development

composer install
composer test        # Run PHPUnit test suite
composer phpstan     # Run PHPStan (level max)
composer cs          # Check code style with PHP CS Fixer (composer cs-fix to apply)
composer infection   # Run mutation tests (requires pcov or Xdebug)

Live provider tests are located in tests/Live and are run against actual APIs. See CONTRIBUTING.md and SECURITY.md.

License

MIT. See LICENSE.

axitdev / phpunit-ai-evals

Package info

Security

Statistics