adachsoft / ai-integration-tester
A PHP toolkit for testing AI provider integrations with reusable conformance scenarios.
Requires
- php: ^8.3
- adachsoft/ai-conversation-model: ^0.2
- adachsoft/ai-integration: ^0.10
- adachsoft/collection: ^3.0
- adachsoft/console-io: ^0.2
- guzzlehttp/guzzle: ^7.9
- phpunit/phpunit: ^12.4 || ^13.0
- vlucas/phpdotenv: ^5.6
Requires (Dev)
- adachsoft/php-code-style: ^0.7
- friendsofphp/php-cs-fixer: ^3.65
- phpstan/phpstan: ^2.1
- rector/rector: ^2.0
- symplify/phpstan-rules: ^14.9
Suggests
- ext-gd: Required to generate in-memory JPEG and PNG test images.
Provides
None
Conflicts
None
Replaces
None
README
AI Integration Tester is a provider-independent PHP toolkit for running reusable conformance scenarios against AI integrations. It includes a core scenario runner, tool-calling scenarios, optional multimodal scenarios, and a CLI.
Requirements
- PHP 8.3 or later
- Composer
adachsoft/ai-integration^0.10adachsoft/ai-conversation-model^0.2ext-gdis optional and is needed only for scenarios that generate test images
Installation
Install the toolkit as a development dependency in the project whose provider integration you want to test:
composer require --dev adachsoft/ai-integration-tester
The CLI reads .ai-test.php from the current working directory. The repository's example configuration registers the built-in OpenAI and DeepSeek profiles. Credentials can be supplied through environment variables or a project-root .env file.
Architecture
The root Composer package is the single Packagist distribution. The packages/ai-integration-tester-cli and packages/ai-integration-tester-multimodal source trees are included in that distribution and loaded through the root Composer autoloader.
The core in src/ provides provider profiles, simulation, scenario registration, tool execution, assertions, and results. Its service-provider interface (SPI) contains the extension contracts:
AdachSoft\AiIntegrationTester\Spi\Provider\ProviderProfileInterface,ProviderCapabilityEnum, andCapabilityModelDtodescribe provider configuration, declared capabilities, and capability-specific models.AdachSoft\AiIntegrationTester\Spi\Scenario\ScenarioInterfaceandScenarioPackInterfacedefine scenarios and packs of scenarios.AdachSoft\AiIntegrationTester\Spi\Tool\TestToolInterfacedefines executable test tools.AdachSoft\AiIntegrationTester\Spi\Assertion\AssertionInterfacedefines scenario assertions.
The CLI package provides configuration, commands, console I/O, and reports. The multimodal package provides image, file-ID, and audio scenarios. ScenarioPackCollection::getRequiredTools() collects the unique tools required by the registered packs. The CLI registers ToolCallingScenarioPack and MultimodalScenarioPack without a file uploader, so its default listing includes image-message, image-tool, and audio scenarios; the file-ID scenario requires an explicitly configured uploader.
Provider credentials and models
Set the API keys for the providers you use:
OPENAI_API_KEY=your-openai-api-key
DEEPSEEK_API_KEY=your-deepseek-api-key
The built-in profiles use these model defaults and environment overrides:
| Provider capability | Environment variable | Default model |
|---|---|---|
| OpenAI text and tool calling | OPENAI_MODEL | gpt-4o-mini |
| DeepSeek text and tool calling | DEEPSEEK_MODEL | deepseek-flash |
| OpenAI audio output | OPENAI_AUDIO_MODEL | gpt-audio-1.5 |
Override a model by setting its corresponding environment variable. Provider profiles can expose additional models through getCapabilityModel(). Scenarios can request a model for a capability through their session configuration; the scenario catalog lists each requested model capability. If a profile does not provide a model for the requested capability, AgentSimulator uses the profile's default model. Provider model names may be changed or retired; check the provider's current model availability when a request fails.
Writing a provider conformance test
Extend AbstractProviderConformanceTestCase, provide a profile for the provider under test, and return the scenario packs to run. The base test case skips scenarios whose required capabilities are not declared by the profile before sending requests to the provider.
use AdachSoft\AiIntegrationTester\Conformance\AbstractProviderConformanceTestCase;
use AdachSoft\AiIntegrationTester\Scenario\Pack\Collection\ScenarioPackCollection;
use AdachSoft\AiIntegrationTester\Scenario\Pack\ToolCallingScenarioPack;
use AdachSoft\AiIntegrationTester\Spi\Provider\ProviderProfileInterface;
use AdachSoft\AiIntegrationTesterMultimodal\MultimodalScenarioPack;
final class MyProviderConformanceTest extends AbstractProviderConformanceTestCase
{
protected function getProviderProfile(): ProviderProfileInterface
{
return new MyProviderProfile();
}
protected static function getScenarioPacks(): ScenarioPackCollection
{
return new ScenarioPackCollection([
new ToolCallingScenarioPack(),
new MultimodalScenarioPack(),
]);
}
}
A custom profile can extend AbstractProviderProfile and override capability reporting, parameter adaptation, and capability-specific model selection. For example, these overrides advertise the provider's capabilities, force temperature to 1.0, and select an audio model:
use AdachSoft\AiIntegrationTester\Provider\AbstractProviderProfile;
use AdachSoft\AiIntegrationTester\Simulator\Dto\GenerationParametersDto;
use AdachSoft\AiIntegrationTester\Spi\Provider\CapabilityModelDto;
use AdachSoft\AiIntegrationTester\Spi\Provider\Collection\ProviderCapabilityEnumCollection;
use AdachSoft\AiIntegrationTester\Spi\Provider\ProviderCapabilityEnum;
use AdachSoft\AiIntegrationTester\ValueObject\ModelIdVo;
final class MyProviderProfile extends AbstractProviderProfile
{
public function getCapabilities(): ProviderCapabilityEnumCollection
{
return new ProviderCapabilityEnumCollection([
ProviderCapabilityEnum::Text,
ProviderCapabilityEnum::ToolCalling,
ProviderCapabilityEnum::VisionInput,
ProviderCapabilityEnum::AudioOutput,
]);
}
public function adaptParameters(?GenerationParametersDto $parameters): ?GenerationParametersDto
{
return new GenerationParametersDto(
temperature: 1.0,
maxTokens: $parameters?->maxTokens,
customParameters: $parameters?->customParameters ?? [],
);
}
public function getCapabilityModel(ProviderCapabilityEnum $capability): ?CapabilityModelDto
{
if ($capability !== ProviderCapabilityEnum::AudioOutput) {
return null;
}
return new CapabilityModelDto(
envName: 'OPENAI_AUDIO_MODEL',
defaultModelId: new ModelIdVo('gpt-audio-1.5'),
);
}
}
This excerpt shows capability-related overrides only. A concrete provider profile must also implement its provider identity, API-key environment name, default model, and facade-builder configuration required by ProviderProfileInterface.
To add file-ID image conformance, pass a FileUploaderInterface implementation to MultimodalScenarioPack. The package includes DeepseekFileUploader, which can be created with DeepseekFileUploader::create($deepseekApiKey). Without an uploader, the pack omits file_id_image.
Scenario catalog
The built-in scenarios are grouped into tool_calling (ToolCallingScenarioPack) and multimodal (MultimodalScenarioPack). Capability names below are the ProviderCapabilityEnum values.
| Scenario | Pack | Required capabilities | Model capability | What it checks |
|---|---|---|---|---|
smoke_chat | tool_calling | text | — | The provider returns non-empty assistant text without tools. |
tool_calling_conformance | tool_calling | text, tool_calling | — | The provider calls the sum tool and returns the correct answer. |
multi_step_agent | tool_calling | text, tool_calling | — | The provider uses a configured tool before answering. |
time_and_file | tool_calling | text, tool_calling | — | The provider uses current_time and read_file in a two-step conversation. |
multiple_choices | tool_calling | text, multiple_choices | multiple_choices | The provider returns two non-empty completion choices without tools. |
parallel_tool_calls | tool_calling | text, tool_calling | — | The provider calls the sum tool in parallel and reports both results. |
images_in_user_message | multimodal | text, vision_input | vision_input | The provider reads two images included in one user message. |
tool_returns_image | multimodal | text, tool_calling, tool_result_media | tool_result_media | The provider reads an image returned by the read_image tool. |
audio_round_trip | multimodal | text, audio_output | audio_output | An audio response can be referenced by a follow-up request. |
file_id_image | multimodal | text, vision_input, file_id | file_id | The provider reads an image uploaded and referenced by file ID. |
A dash in the model-capability column means the scenario uses the profile's default model. file_id_image is added only when a MultimodalScenarioPack receives a file uploader. The default CLI registers the multimodal pack without one.
Custom scenarios, tools, and packs
Add a scenario by implementing ScenarioInterface, group it in a ScenarioPackInterface, and implement TestToolInterface for any tools it needs. Implement AssertionInterface for reusable result checks. Register your pack alongside the built-in packs in a ScenarioPackCollection; its getRequiredTools() method aggregates the unique tools for the configured packs. These extension points let provider projects add coverage without changing the toolkit core.
Usage
Use the CLI to list scenarios or run conformance checks against a configured provider.
CLI
List scenarios without making a provider request, run a selected scenario, validate a provider, or inspect tool contracts:
vendor/bin/ai-test scenario:list
vendor/bin/ai-test scenario:run tool_calling_conformance --provider=openai
vendor/bin/ai-test scenario:run images_in_user_message --provider=openai
vendor/bin/ai-test provider:validate --provider=openai
vendor/bin/ai-test tool:contract
Use --provider=<id> to override the default provider configured in .ai-test.php. The file-ID scenario is not listed by the default CLI configuration because it has no uploader configured.
Tests
The default PHPUnit suite is the unit suite. The separate production suite makes requests to real provider APIs and may incur charges:
vendor/bin/phpunit
vendor/bin/phpunit --testsuite production
Use the production suite only when you intentionally want to run live provider conformance tests and have configured the required API keys.
Upgrade from 0.4
The 0.5 layout moves public extension points and CLI implementation into dedicated namespaces and packages.
| 0.4 type or layout | 0.5 migration |
|---|---|
ProviderProfileInterface | AdachSoft\AiIntegrationTester\Spi\Provider\ProviderProfileInterface |
ScenarioInterface | AdachSoft\AiIntegrationTester\Spi\Scenario\ScenarioInterface |
TestToolInterface | AdachSoft\AiIntegrationTester\Spi\Tool\TestToolInterface |
AssertionInterface | AdachSoft\AiIntegrationTester\Spi\Assertion\AssertionInterface |
Cli\, Config\, Report\, and Logging\ classes | Use the AdachSoft\AiIntegrationTesterCli\ namespace; their source is in packages/ai-integration-tester-cli/. |
ConformanceScenarioProvider | Use ToolCallingScenarioPack and register scenario packs through ScenarioPackCollection. |
Local ObservedToolCallDto, Simulator\Dto\ToolResultDto, and Tool\Dto\ToolDefinitionDto | The old local DTOs are no longer part of this package. Use the shared tool and content DTOs from adachsoft/ai-conversation-model where applicable. |
Support\ DTOs | Use the shared content DTOs and collections from adachsoft/ai-conversation-model, such as ContentPartCollection, TextContentPartDto, and MediaContentPartDto. |
ProviderFactory | Configure a provider profile and use the provider registry and FacadeFactory. |
The public contracts also changed:
TestToolInterface::execute()returnsContentPartCollection.AgentResponseDto::$completionusesAdachSoft\AiIntegration\Contract\ToolCalling\Dto\ChatCompletionDto.AgentSimulatorInterface::sendUserMessage()accepts either a string orContentPartCollection.GenerationParametersDto::$temperatureis nullable;nullomits the temperature parameter from the request.- Provider profiles declare capabilities with
getCapabilities(), select capability-specific models withgetCapabilityModel(), and adapt request parameters withadaptParameters().
Update imports, custom tools, and provider profiles to these contracts. The production suite is for real provider APIs; run the unit suite with the default PHPUnit command during migration.