patricksamson / phpunit-flaky-fix
A PHPUnit extension to help reproduce flaky tests.
Requires
- php: ~8.1.0 || ~8.2.0 || ~8.3.0 || ~8.4.0
- phpunit/phpunit: ^10.0.0 || ^11.0.0 || ^12.0.0
Requires (Dev)
- brianium/paratest: ^7.0
- laravel/pint: ^1.24
- pestphp/pest: ^2.0.0 || ^3.0.0
- phpstan/phpstan: ^2.1
- rector/rector: ^2.1
- spatie/ray: ^1.42
This package is auto-updated.
Last update: 2025-08-05 18:16:51 UTC
README
This package provides a PHPUnit extension that helps reproduce flaky tests by seeding the random number generator with a consistent value across test runs. This allows you to reproduce and debug flaky tests more easily.
Compatibility
This package is compatible with the following versions of PHP and PHPUnit:
- PHP :
^8.1 || ^8.2 || ^8.3 || ^8.4
- PHPUnit :
^10.0 || ^11.0 || ^12.0
It has been tested with the following versions of popular tools that wrap PHPUnit:
- Paratest :
^7.0
- Pest :
^2.0 || ^3.0
(this package include a Pest Plugin for improve compatibility)
It is compatible with any tool that uses mt_rand()
for randomness in tests, such as Faker.
Installation
composer require --dev patricksamson/phpunit-flaky-fix
You'll then need to register the extension in your phpunit.xml
file:
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd" bootstrap="vendor/autoload.php" > + <extensions> + <bootstrap class="PatrickSamson\PHPUnitFlakyFix\FlakyFixExtension" /> + </extensions> <testsuites> <testsuite name="unit"> <directory>tests/Unit/</directory> </testsuite> </testsuites> </phpunit>
Usage
When you have bootstrapped the extension, you can run your tests as usual:
vendor/bin/phpunit php artisan test php artisan test --parallel
The generated Flaky Test Seed
will be one of the first lines in the output, and you can use it to reproduce flaky test failures.
vendor/bin/phpunit --colors=always Flaky Test Seed: 2031556362. To reproduce, run `FLAKY_SEED=2031556362 php artisan test --filter ...` PHPUnit 12.3.0 by Sebastian Bergmann and contributors. Runtime: PHP 8.4.10 Configuration: /home/runner/work/phpunit-flaky-fix/phpunit-flaky-fix/phpunit.xml ........... 11 / 11 (100%) Time: 00:00.003, Memory: 14.00 MB OK (11 tests, 35 assertions)
How It Works
The extension works by:
- Generating a new, completely random seed at the start of your test suite execution, and displaying it in the output
- Seeding PHP's random number generator (
mt_rand()
) with this value, beforesetUp()
is called for each test
This ensures that your tests remain deterministic and reproducible, while still allowing for randomness between different test suite runs. To reproduce a flaky test failure, simply re-run that specific test with the provided seed.
In practice, there is some additional complexity to ensure compatibility with parallel test execution and other PHPUnit features.
Limitations
- This extension only affects PHP's
mt_rand()
function and related randomness functions - It has no effect on time-based functions (e.g.,
time()
,date()
, ...), but that can be remediated using something likeCarbon::setTestNow($knownDate)
- It does not control randomness from other sources (e.g., database auto-increment values, UUID generation, external API calls, ...)
- The seed is generated per test suite run, not per individual test
- Some PHPUnit features or third-party tools might interfere with the seed generation process or the output display
License
This project uses the MIT license.
Credits
The testing structure of this package is heavily inspired from ergebnis/phpunit-slow-test-detector
, originally licensed under MIT by Andreas Möller.