Search by

adachsoft / ai-integration-tester

Arkadiusz Adach

A PHP toolkit for testing AI provider integrations with reusable conformance scenarios.

Package info

gitlab.com/a.adach/ai-integration-tester

Issues

pkg:composer/adachsoft/ai-integration-tester

Statistics

Installs: 28

Dependents: 8

Suggesters: 0

Stars: 0

v0.5.2 2026-11-12 19:34 UTC

This package is auto-updated.

Last update: 2026-11-12 18:36:46 UTC


README

AI Integration Tester is a provider-independent PHP toolkit for running reusable conformance scenarios against AI integrations. It includes a core scenario runner, tool-calling scenarios, optional multimodal scenarios, and a CLI.

Requirements

  • PHP 8.3 or later
  • Composer
  • adachsoft/ai-integration ^0.10
  • adachsoft/ai-conversation-model ^0.2
  • ext-gd is optional and is needed only for scenarios that generate test images

Installation

Install the toolkit as a development dependency in the project whose provider integration you want to test:

composer require --dev adachsoft/ai-integration-tester

The CLI reads .ai-test.php from the current working directory. The repository's example configuration registers the built-in OpenAI and DeepSeek profiles. Credentials can be supplied through environment variables or a project-root .env file.

Architecture

The root Composer package is the single Packagist distribution. The packages/ai-integration-tester-cli and packages/ai-integration-tester-multimodal source trees are included in that distribution and loaded through the root Composer autoloader.

The core in src/ provides provider profiles, simulation, scenario registration, tool execution, assertions, and results. Its service-provider interface (SPI) contains the extension contracts:

  • AdachSoft\AiIntegrationTester\Spi\Provider\ProviderProfileInterface, ProviderCapabilityEnum, and CapabilityModelDto describe provider configuration, declared capabilities, and capability-specific models.
  • AdachSoft\AiIntegrationTester\Spi\Scenario\ScenarioInterface and ScenarioPackInterface define scenarios and packs of scenarios.
  • AdachSoft\AiIntegrationTester\Spi\Tool\TestToolInterface defines executable test tools.
  • AdachSoft\AiIntegrationTester\Spi\Assertion\AssertionInterface defines scenario assertions.

The CLI package provides configuration, commands, console I/O, and reports. The multimodal package provides image, file-ID, and audio scenarios. ScenarioPackCollection::getRequiredTools() collects the unique tools required by the registered packs. The CLI registers ToolCallingScenarioPack and MultimodalScenarioPack without a file uploader, so its default listing includes image-message, image-tool, and audio scenarios; the file-ID scenario requires an explicitly configured uploader.

Provider credentials and models

Set the API keys for the providers you use:

OPENAI_API_KEY=your-openai-api-key
DEEPSEEK_API_KEY=your-deepseek-api-key

The built-in profiles use these model defaults and environment overrides:

Provider capabilityEnvironment variableDefault model
OpenAI text and tool callingOPENAI_MODELgpt-4o-mini
DeepSeek text and tool callingDEEPSEEK_MODELdeepseek-flash
OpenAI audio outputOPENAI_AUDIO_MODELgpt-audio-1.5

Override a model by setting its corresponding environment variable. Provider profiles can expose additional models through getCapabilityModel(). Scenarios can request a model for a capability through their session configuration; the scenario catalog lists each requested model capability. If a profile does not provide a model for the requested capability, AgentSimulator uses the profile's default model. Provider model names may be changed or retired; check the provider's current model availability when a request fails.

Writing a provider conformance test

Extend AbstractProviderConformanceTestCase, provide a profile for the provider under test, and return the scenario packs to run. The base test case skips scenarios whose required capabilities are not declared by the profile before sending requests to the provider.

use AdachSoft\AiIntegrationTester\Conformance\AbstractProviderConformanceTestCase;
use AdachSoft\AiIntegrationTester\Scenario\Pack\Collection\ScenarioPackCollection;
use AdachSoft\AiIntegrationTester\Scenario\Pack\ToolCallingScenarioPack;
use AdachSoft\AiIntegrationTester\Spi\Provider\ProviderProfileInterface;
use AdachSoft\AiIntegrationTesterMultimodal\MultimodalScenarioPack;

final class MyProviderConformanceTest extends AbstractProviderConformanceTestCase
{
    protected function getProviderProfile(): ProviderProfileInterface
    {
        return new MyProviderProfile();
    }

    protected static function getScenarioPacks(): ScenarioPackCollection
    {
        return new ScenarioPackCollection([
            new ToolCallingScenarioPack(),
            new MultimodalScenarioPack(),
        ]);
    }
}

A custom profile can extend AbstractProviderProfile and override capability reporting, parameter adaptation, and capability-specific model selection. For example, these overrides advertise the provider's capabilities, force temperature to 1.0, and select an audio model:

use AdachSoft\AiIntegrationTester\Provider\AbstractProviderProfile;
use AdachSoft\AiIntegrationTester\Simulator\Dto\GenerationParametersDto;
use AdachSoft\AiIntegrationTester\Spi\Provider\CapabilityModelDto;
use AdachSoft\AiIntegrationTester\Spi\Provider\Collection\ProviderCapabilityEnumCollection;
use AdachSoft\AiIntegrationTester\Spi\Provider\ProviderCapabilityEnum;
use AdachSoft\AiIntegrationTester\ValueObject\ModelIdVo;

final class MyProviderProfile extends AbstractProviderProfile
{
    public function getCapabilities(): ProviderCapabilityEnumCollection
    {
        return new ProviderCapabilityEnumCollection([
            ProviderCapabilityEnum::Text,
            ProviderCapabilityEnum::ToolCalling,
            ProviderCapabilityEnum::VisionInput,
            ProviderCapabilityEnum::AudioOutput,
        ]);
    }

    public function adaptParameters(?GenerationParametersDto $parameters): ?GenerationParametersDto
    {
        return new GenerationParametersDto(
            temperature: 1.0,
            maxTokens: $parameters?->maxTokens,
            customParameters: $parameters?->customParameters ?? [],
        );
    }

    public function getCapabilityModel(ProviderCapabilityEnum $capability): ?CapabilityModelDto
    {
        if ($capability !== ProviderCapabilityEnum::AudioOutput) {
            return null;
        }

        return new CapabilityModelDto(
            envName: 'OPENAI_AUDIO_MODEL',
            defaultModelId: new ModelIdVo('gpt-audio-1.5'),
        );
    }
}

This excerpt shows capability-related overrides only. A concrete provider profile must also implement its provider identity, API-key environment name, default model, and facade-builder configuration required by ProviderProfileInterface.

To add file-ID image conformance, pass a FileUploaderInterface implementation to MultimodalScenarioPack. The package includes DeepseekFileUploader, which can be created with DeepseekFileUploader::create($deepseekApiKey). Without an uploader, the pack omits file_id_image.

Scenario catalog

The built-in scenarios are grouped into tool_calling (ToolCallingScenarioPack) and multimodal (MultimodalScenarioPack). Capability names below are the ProviderCapabilityEnum values.

ScenarioPackRequired capabilitiesModel capabilityWhat it checks
smoke_chattool_callingtext—The provider returns non-empty assistant text without tools.
tool_calling_conformancetool_callingtext, tool_calling—The provider calls the sum tool and returns the correct answer.
multi_step_agenttool_callingtext, tool_calling—The provider uses a configured tool before answering.
time_and_filetool_callingtext, tool_calling—The provider uses current_time and read_file in a two-step conversation.
multiple_choicestool_callingtext, multiple_choicesmultiple_choicesThe provider returns two non-empty completion choices without tools.
parallel_tool_callstool_callingtext, tool_calling—The provider calls the sum tool in parallel and reports both results.
images_in_user_messagemultimodaltext, vision_inputvision_inputThe provider reads two images included in one user message.
tool_returns_imagemultimodaltext, tool_calling, tool_result_mediatool_result_mediaThe provider reads an image returned by the read_image tool.
audio_round_tripmultimodaltext, audio_outputaudio_outputAn audio response can be referenced by a follow-up request.
file_id_imagemultimodaltext, vision_input, file_idfile_idThe provider reads an image uploaded and referenced by file ID.

A dash in the model-capability column means the scenario uses the profile's default model. file_id_image is added only when a MultimodalScenarioPack receives a file uploader. The default CLI registers the multimodal pack without one.

Custom scenarios, tools, and packs

Add a scenario by implementing ScenarioInterface, group it in a ScenarioPackInterface, and implement TestToolInterface for any tools it needs. Implement AssertionInterface for reusable result checks. Register your pack alongside the built-in packs in a ScenarioPackCollection; its getRequiredTools() method aggregates the unique tools for the configured packs. These extension points let provider projects add coverage without changing the toolkit core.

Usage

Use the CLI to list scenarios or run conformance checks against a configured provider.

CLI

List scenarios without making a provider request, run a selected scenario, validate a provider, or inspect tool contracts:

vendor/bin/ai-test scenario:list
vendor/bin/ai-test scenario:run tool_calling_conformance --provider=openai
vendor/bin/ai-test scenario:run images_in_user_message --provider=openai
vendor/bin/ai-test provider:validate --provider=openai
vendor/bin/ai-test tool:contract

Use --provider=<id> to override the default provider configured in .ai-test.php. The file-ID scenario is not listed by the default CLI configuration because it has no uploader configured.

Tests

The default PHPUnit suite is the unit suite. The separate production suite makes requests to real provider APIs and may incur charges:

vendor/bin/phpunit
vendor/bin/phpunit --testsuite production

Use the production suite only when you intentionally want to run live provider conformance tests and have configured the required API keys.

Upgrade from 0.4

The 0.5 layout moves public extension points and CLI implementation into dedicated namespaces and packages.

0.4 type or layout0.5 migration
ProviderProfileInterfaceAdachSoft\AiIntegrationTester\Spi\Provider\ProviderProfileInterface
ScenarioInterfaceAdachSoft\AiIntegrationTester\Spi\Scenario\ScenarioInterface
TestToolInterfaceAdachSoft\AiIntegrationTester\Spi\Tool\TestToolInterface
AssertionInterfaceAdachSoft\AiIntegrationTester\Spi\Assertion\AssertionInterface
Cli\, Config\, Report\, and Logging\ classesUse the AdachSoft\AiIntegrationTesterCli\ namespace; their source is in packages/ai-integration-tester-cli/.
ConformanceScenarioProviderUse ToolCallingScenarioPack and register scenario packs through ScenarioPackCollection.
Local ObservedToolCallDto, Simulator\Dto\ToolResultDto, and Tool\Dto\ToolDefinitionDtoThe old local DTOs are no longer part of this package. Use the shared tool and content DTOs from adachsoft/ai-conversation-model where applicable.
Support\ DTOsUse the shared content DTOs and collections from adachsoft/ai-conversation-model, such as ContentPartCollection, TextContentPartDto, and MediaContentPartDto.
ProviderFactoryConfigure a provider profile and use the provider registry and FacadeFactory.

The public contracts also changed:

  • TestToolInterface::execute() returns ContentPartCollection.
  • AgentResponseDto::$completion uses AdachSoft\AiIntegration\Contract\ToolCalling\Dto\ChatCompletionDto.
  • AgentSimulatorInterface::sendUserMessage() accepts either a string or ContentPartCollection.
  • GenerationParametersDto::$temperature is nullable; null omits the temperature parameter from the request.
  • Provider profiles declare capabilities with getCapabilities(), select capability-specific models with getCapabilityModel(), and adapt request parameters with adaptParameters().

Update imports, custom tools, and provider profiles to these contracts. The production suite is for real provider APIs; run the unit suite with the default PHPUnit command during migration.