innmind / black-box
Test library
Requires
- php: ~7.4|~8.0
- innmind/json: ^1.1
- phpunit/phpunit: ~8.0 || ~9.0
- symfony/var-dumper: ~4.0|~5.0|~6.0
Requires (Dev)
- innmind/coding-standard: ^1.1
- ramsey/uuid: ^4.1
- vimeo/psalm: ~4.1
Suggests
- icomefromthenet/reverse-regex: To be able to use Set\Regex
- dev-develop
- 4.18.1
- 4.18.0
- 4.17.0
- 4.16.0
- 4.15.1
- 4.15.0
- 4.14.0
- 4.13.0
- 4.12.0
- 4.11.0
- 4.10.1
- 4.10.0
- 4.9.0
- 4.8.0
- 4.7.1
- 4.7.0
- 4.6.0
- 4.5.0
- 4.4.1
- 4.4.0
- 4.3.1
- 4.3.0
- 4.2.5
- 4.2.4
- 4.2.3
- 4.2.2
- 4.2.1
- 4.2.0
- 4.1.1
- 4.1.0
- 4.0.0
- 3.0.0
- 2.9.0
- 2.8.1
- 2.8.0
- 2.7.0
- 2.6.0
- 2.5.0
- 2.4.0
- 2.3.0
- 2.2.0
- 2.1.0
- 2.0.0
- 1.0.0
- dev-master
- dev-test-runner
This package is auto-updated.
Last update: 2022-05-04 18:18:52 UTC
README
Contains an ensemble of sets to easily generate data for property based tests.
Philosophy
When I run tests I need some data to assert the validity of my code, the first approach is to hardcode the test data in the test class itself but it lacks enough variety in order to make sure all (or at least enough) cases are covered. In order to generate data we can use a property based testing library such as giorgiosironi/eris
, but the problem is that for each test you need to redeclare the base sets of data you need test against.
The goal of this library is to help build higher order sets to facilitate the understanding of tests.
BlackBox
comes with the Set
s of primitives:
Integers
->int
RealNumbers
->float
Unicode
->string
Strings
->string
UnsafeStrings
->string
(found here)Regex
->string
User defined elements Set
s can be defined with:
Higher order Set
s allows you create structures:
Decorate
-> map a typeA
to a typeB
, ie anint
to an objectAge
Composite
-> map many types to another unique type, iestring $firstname
andstring $lastname
to an objectUser($firstname, $lastname)
Sequence
-> create anarray
containing multiple elements of the same typeEither
-> will generate either a typeA
orB
, ie to create nullableint
s viaEither(Integers::any(), Elements::of(null))
Installation
composer require innmind/black-box
Usage
use Innmind\BlackBox\{ Set, PHPUnit\BlackBox, }; final class Counter { private int $current; public function __construct(int $initial = 0) { $this->current = $initial; } public function up(): void { if ($this->current === 100) { return; } ++$this->current; } public function down(): void { if ($this->current === 0) { return; } --$this->current; } public function current(): int { return $this->current; } } class CounterTest extends \PHPUnit\Framework\TestCase { use BlackBox; public function testCounterValueIsAlwaysHigherAfterCountingUp() { $this ->forAll( Set\Integers::between(0, 100), // counter bounds ) ->then(function(int $initial) { $counter = new Counter($initial); $counter->up(); $this->assertGreaterThan($initial, $counter->value()); }); } }
This really simple example show how the test class is focused on the behaviour and not about the construction of the test data.
By default the library supports the shrinking of data to help you find the smallest possible set of values that makes your test fail. To help you ease the debugging of the code you can use the printer class Innmind\BlackBox\PHPUnit\ResultPrinterV9
that will print the set of generated data that made your test fail.
Stateful testing
When we write tests we tend to focus on evaluating the behaviour when doing one action (like in our counter example above). This technique help us cover most of our code, but when we deal with stateful systems (such as a counter, an entity or a daemon) it becomes harder to make sure all succession of mutations will always result in a coherent new state.
Once again Property Based Testing can help us improve the coverage of behaviours. Instead of describing the initial test to the framework and manually do one action, we describe to the framework all the properties that our system must hold and the framework will try to find a succession of actions that will break our properties.
If we reuse the counter example from above, the property would be written like this:
use Innmind\BlackBox\Property; use PHPUnit\Framework\Assert; final class UpChangeState implements Property { public function name(): string { return 'Counting up always end in a higher count'; } public function applicableTo(object $counter): bool { return $counter->current() < 100; // since upper bound is 100 } public function ensureHeldBy(object $counter): object { $initial = $counter->current(); $counter->up(); Assert::assertGreaterThan($initial, $counter->current()); return $counter; } }
With all the other properties (provided in the fixtures
folder) to test the whole behaviour the test in phpunit would then look like this:
class CounterTest extends \PHPUnit\Framework\TestCase { use BlackBox; public function testProperties() { $this ->forAll( Set\Properties::any( Set\Property::of(DownAndUpIsAnIdentityFunction::class), Set\Property::of(DownChangeState::class), Set\Property::of(RaiseBy::class, Set\Integers::between(1, 99)), Set\Property::of(UpAndDownIsAnIdentityFunction::class), Set\Property::of(UpChangeState::class), Set\Property::of(UpperBoundAtHundred::class), ), Set\Integers::between(0, 100), // counter bounds ) ->then(function($scenario, $initial) { $scenario->ensureHeldBy(new Counter($initial)); }); } }
Note: you should declare the properties as the first set of forAll
to make sure it is shrunk first.
The above example would generate multiple scenarii of counting up and down (it tries to apply up to 100 properties per scenario). In the case it found a failing scenario, it would be displayed as follow in your terminal:
Note: this counter example is used as the test process of this framework, all properties to prove the behaviour of the counter can be found in the fixtures/
folder.
Note 2: this example was taken from an article by Johannes Link on Model-based Testing.
Configuration
Set size
By default it will run 100 iterations of different values to test your properties. You can manually change this value in each test by calling $this->forAll(/** $set */)->take($somethingOtherThan100)
.
Specifying a different value can be repetitive if you want all your tests to run the same number of iterations, with this in mind you can specify an env variable named BLACKBOX_SET_SIZE
in your phpunit.xml.dist
set to the number of iterations you want for all your tests.
Note: of course you can override this value locally in each tests.
Properties informations
By default when a set of properties fail the printer will only display the named of each property to have a more digest information. However in some cases (like paramaterized properties) you may want to have the whole property object displayed to help debug complex scenarii, for such case you can use the environment variable BLACKBOX_DETAILED_PROPERTIES
with a value set to 1
.
Shrinking
By default black box will try to find the minimum test case data that make the test fail to help you better understand the problem at hand. In certain cases, like tests relying on many generators, this shrinking process can take too much time meaning it does more harm than good to the developer experience.
You can disable the shrinking process on a specific test via the method ->disableShrinking()
or for the whole test suite with the environment variable BLACKBOX_DISABLE_SHRINKING=1
.