adachsoft / ai-integration-tester

v0.1.0 2026-10-27 10:05 UTC

Requires

• php: ^8.3
• adachsoft/ai-integration: ^0.6
• adachsoft/collection: ^3.0
• adachsoft/console-io: ^0.2
• phpunit/phpunit: ^12.4
• vlucas/phpdotenv: ^5.6

Requires (Dev)

• adachsoft/php-code-style: ^0.7
• friendsofphp/php-cs-fixer: ^3.65
• phpstan/phpstan: ^2.1
• rector/rector: ^2.0
• symplify/phpstan-rules: ^14.9

MIT

• Arkadiusz Adach

• testing
• phpunit
• ai
• integration-testing
• openai
• llm
• deepseek
• ai-testing

Published on Packagist.org on 2026-10-27 09:09 UTC

README

ai-integration-tester verifies AI provider integrations through reusable conformance scenarios. It exercises chat responses, tool calls, typed scenario results, and provider-specific behavior through a small CLI application.

Requirements

• PHP 8.3 or later
• Composer
• API credentials for each provider used by production conformance tests or live examples

Installation

Install the package as a development dependency:

composer require --dev adachsoft/ai-integration-tester

The repository includes a typed configuration file at .ai-test.php. It registers the built-in OpenAI and Deepseek profiles, the default provider, and the tools used by the conformance scenarios. Configuration errors are reported instead of silently falling back to defaults.

Provider credentials

The default environment reader loads a project-root .env file when needed. Set the credentials for the providers you intend to use:

OPENAI_API_KEY=your-openai-api-key
DEEPSEEK_API_KEY=your-deepseek-api-key

The built-in profiles use gpt-4o-mini for OpenAI and deepseek-chat for Deepseek. Production conformance tests make real provider requests and require valid credentials; they do not skip when a key is missing.

CLI

List registered scenarios without connecting to a provider:

php bin/ai-test scenario:list

Run a scenario with a selected provider:

php bin/ai-test scenario:run tool_calling_conformance --provider=openai
php bin/ai-test scenario:run tool_calling_conformance --provider=deepseek

Validate a provider by running the registered smoke scenario, or inspect the configured tool contracts:

php bin/ai-test provider:validate --provider=openai
php bin/ai-test tool:contract

The default provider is configured in .ai-test.php; --provider=<id> overrides it for a command. Scenario and provider setup is performed when a command needs it, so listing scenarios does not require API credentials.

Examples

The scripts in examples/ invoke the same CLI and return its exit code:

php examples/conversation_openai.php
php examples/conversation_deepseek.php
php examples/time_and_file_conversation.php           # defaults to Deepseek
php examples/time_and_file_conversation.php openai
php examples/time_and_file_conversation.php deepseek

The time-and-file example runs the time_and_file_conversation scenario, which exercises the current-time and virtual-file tools. Configure the corresponding provider key in .env before running a live example.

Tests and quality checks

Run the local unit suite and the separate live-provider suite:

vendor/bin/phpunit --testsuite unit
vendor/bin/phpunit --testsuite production

The production suite contacts both OpenAI and Deepseek and requires both API keys. Additional project checks are:

vendor/bin/phpstan analyse
composer cs:check

Project layout

• src/ — provider profiles, configuration, CLI, simulators, conformance scenarios, assertions, and typed results.
• tests/Unit/ — isolated unit tests.
• tests/Production/ — live provider conformance tests.
• examples/ — runnable CLI examples.