README

Checks that a PHP application behaves the same after an upgrade: a new PHP version, a new framework version, Rector changes.

It finds every endpoint of the application, generates requests for them, runs the requests on the old and on the new version, and reports what changed: status codes, response bodies, new errors and warnings, sent mail.

Behavior diff  PHP 7.4.33 → PHP 8.1.34

Cases: 69 on 8 endpoints   identical: 61 (88.4%)
Different: critical: 0   high: 6   medium: 0   low: 2

GET /shop/user  2 cases
  GET /shop/user?id=abc  [id=abc, 4c5aef50fcef]
    HIGH Status 200 → 404
    MEDIUM Response body changed
      - Guest
      + Not found

GET /shop/search  3 cases
  GET /shop/search?q%5B0%5D=x  [q=[array], 65055a91dfbe]
    HIGH Different exception: TypeError: strlen(): Argument #1 ($string) must be of type string, array given
      app/Http/Controllers/ShopController.php:39
  GET /shop/search?q=  [q="", 90e71a01c00b]
    LOW New deprecated: strlen(): Passing null to parameter #1 ($string) of type string is deprecated

No production traffic and no manual clicking are needed: everything runs locally or in CI.

This package works with any PHP application where pages are files (/users.php?id=1). For Laravel, install toshkq93/behavior-diff-laravel: it reads the routes, validation rules and model bindings, and runs the requests inside the application with transactions and fakes.

How it works

scan  →  cases.json       every endpoint, every parameter, edge values
run   →  old.jsonl        the old version (e.g. PHP 7.4)
run   →  old-2.jsonl      the old version again: shows what is random
run   →  new.jsonl        the new version (e.g. PHP 8.3)
diff  →  report.html      what the new version does differently

Scan. Every *.php file under the document root is an endpoint. Its parameters come from the code: $_GET['id'], $_POST['email'], $_REQUEST['q'], filter_input(INPUT_GET, 'page'). Casts such as (int) $_GET['id'] and intval(...) give the type; otherwise the name does (id, email, date, price, ...). A file that reads $_POST gets a POST endpoint too.

Cases. For each endpoint: one request with valid values, then one request per parameter variant, the other parameters keeping their valid values. The variants target what changed between PHP versions:

Type	Variants
every parameter	missing, `""`, an array (`id[]=x`)
int	`0`, `-1`, `abc`, `1abc`, `1.5`, `1`, `99999999999999999999`
float	`0`, `abc`, `1,5`, `1e3`, `-0`
string	`0`, `abc`, `1e3`, `" "`, `<b>x</b>`, over the max length
date, email, bool	invalid and edge values

"abc" == 0, strlen([]), "abc" * 1 and null passed to string functions behave differently on PHP 8.

Run. The application is started with PHP's built-in server (php -S). A prepend file records the PHP errors of each request, mail() writes to a file instead of sending, and the random seed is fixed.
Diff. Results are compared case by case. Tokens, session ids, absolute paths, PHP version strings and date-times close to the run time are masked. What differs between the two runs of the old version is ignored.

Installation

composer require --dev toshkq93/behavior-diff
vendor/bin/behavior-diff init

PHP 7.1 or later, no other dependencies: the package runs on the old version of the application too.

Usage

# 1. Find the endpoints (once; commit cases.json if you like)
vendor/bin/behavior-diff scan

# 2. Run with the old PHP, twice
php7.4 vendor/bin/behavior-diff run --out=behavior-diff/old.jsonl
php7.4 vendor/bin/behavior-diff run --out=behavior-diff/old-2.jsonl

# 3. Run with the new PHP (or on the upgraded code)
php8.3 vendor/bin/behavior-diff run --out=behavior-diff/new.jsonl

# 4. Compare
vendor/bin/behavior-diff diff behavior-diff/old.jsonl behavior-diff/new.jsonl \
    --noise=behavior-diff/old-2.jsonl --html=behavior-diff/report.html

With Docker instead of locally installed PHP versions:

docker run --rm -v "$PWD":/app -w /app php:7.4-cli vendor/bin/behavior-diff run --out=behavior-diff/old.jsonl
docker run --rm -v "$PWD":/app -w /app php:8.3-cli vendor/bin/behavior-diff run --out=behavior-diff/new.jsonl

diff exits with 1 when it finds differences of medium severity or higher, so it can gate a CI job. Change the threshold with --fail-on=critical|high|medium|low|never.

Severity

Level	Meaning
critical	The new version crashes or does not respond where the old one worked
high	Different status code, mail, data written; a different exception where the old version also failed
medium	Different response body or headers; new warnings and notices
low	New deprecations

Configuration

behavior-diff.php in the project root:

return [
    'cases' => 'behavior-diff/cases.json',

    'discovery' => [
        'root' => 'public',
        'exclude' => ['vendor', 'node_modules', 'includes/*'],
    ],

    // Known values per parameter name; the first one is used by the base request.
    'values' => [
        'id' => [1, 2],
    ],

    'max_cases_per_endpoint' => 40,

    'server' => [
        'docroot' => 'public',
        'router' => null,          // front controller, if the app has one
        'php' => null,             // PHP binary; defaults to the one running behavior-diff
        'ini' => [],
        'env' => [],
        'prepend' => null,         // your file run before every request (see below)
        'reset' => null,           // shell command that restores the data before the run
        'reset_each' => false,     // restore before every case: slow, but exact
    ],

    // Extra patterns removed before comparing: regex => replacement.
    'masks' => [],
];

Real data

Requests with real ids find real bugs; id=1 on an empty database only finds "not found" pages. Put known values into values, and restore the same data before each run with reset:

'server' => [
    'reset' => 'mysql shop < behavior-diff/db.sql',
],

Cases run in a fixed order. Two runs that behave the same leave the same data behind at every step, so writes do not make the runs drift apart.

Logged-in pages

prepend is your PHP file, included before every request. $behaviorDiffUser holds the case user:

// behavior-diff/login.php
if ($behaviorDiffUser !== null) {
    session_start();
    $_SESSION['user_id'] = (int) $behaviorDiffUser;
}

Add the user to the cases you want to run as that user ("user": 1), or write scenario cases by hand.

Hand-written cases and scenarios

cases.json can be edited. scan regenerates the generated cases and keeps the hand-written ones (those without a label, or with a scenario). Cases with the same scenario share cookies and run in file order:

{"id": "checkout-1", "scenario": "checkout", "method": "POST", "uri": "/login.php", "body": {"email": "a@b.c", "password": "secret"}},
{"id": "checkout-2", "scenario": "checkout", "method": "POST", "uri": "/cart.php", "body": {"product": "7"}},
{"id": "checkout-3", "scenario": "checkout", "method": "POST", "uri": "/checkout.php"}

Commands

Command	Options
`init`	`--force`
`scan`	`--config`, `--out`, `--force` (drop hand-written cases too)
`run --out=<file>`	`--config`, `--cases`, `--only="GET /admin/*"` or a case id, `--label`, `--quiet`
`diff <old> <new>`	`--noise=<file>`, `--html=<file>`, `--cases`, `--fail-on`, `--keep-time`, `--no-colors`

Limitations

Parameters read in included files (includes/common.php reading $_GET['lang']) are not found; add them to the cases by hand.
Front controller apps (index.php?route=...) without a framework adapter have one endpoint; write their cases by hand or use an adapter.
Errors hidden by the application's own error handler are not recorded: the handler replaces the one that records them. The response still shows the effect.
Outgoing HTTP calls (curl, Guzzle) are not intercepted by the native runner. Point them at a stub in the test environment.
The current time cannot be frozen without an extension. Date-times within 15 hours of the run are masked as {now}; --keep-time turns this off.

License

MIT

toshkq93 / behavior-diff

Package info

Security

Statistics