README

A small PHP extension that streams the elements of a huge JSON array out of a file in constant memory. Byte scanning is done by libyajl, a C SAX parser. The extension builds exactly one PHP value per array element and returns them one at a time:

foreach (new YajlStream\ArrayParser('/tmp/catalog.json', '/data/items') as $i => $item) {
    // $i is the index in data.items, $item the element decoded like json_decode($json, true)
}

Why it exists

Marketplace and catalog APIs often return their whole inventory as a single JSON document: {"meta":{...},"data":{"items":[ ...hundreds of thousands of objects... ]}}. A dump like this is hundreds of MB to a few GB, and an importer has to read it every few seconds or minutes.

json_decode needs the whole file as a string and the whole decoded tree in memory at the same time. In the benchmark below that is 2.4x to 25x the file size, depending on the data: a 100 MB catalog dump needs about 800 MB, so a multi-GB dump does not fit a worker.
Pure-PHP streaming parsers keep memory flat but tokenize every byte in PHP. In the benchmark json-machine was 3.5x to 7.5x slower than this extension on structured data (on long strings it was slightly faster, 0.9x). That matters when the import cycle is short.

yajl_stream keeps both properties: memory stays flat, and the per-byte work runs in C. It is not the fastest decoder: simdjson is faster in nearly every scenario of the benchmark, and json_decode is faster on string-heavy documents. Both need the whole document in memory.

Alternatives and why they did not fit

Option	What it does	Why it does not solve the problem
`json_decode` (ext/json)	Decodes a complete string	Whole document plus the whole tree in memory (benchmark: 240 MB to 1.3 GB peak for 50-100 MB files). No streaming API.
simdjson PECL extension (`simdjson_decode`, `simdjson_key_value`)	Very fast DOM decoding of a complete string: `simdjson_key_value` was up to 3.8x faster than this extension in the benchmark	Needs the whole document in memory as a PHP string, and more memory than `json_decode` (benchmark: 440 MB to 1.5 GB process RSS for 50-100 MB files), and its documentation caps documents at 4 GB. `simdjson_key_value` still parses a complete in-memory document. No incremental input. Chunking the file in PHP and feeding simdjson one element at a time is possible, but then element boundaries have to be found by scanning bytes in PHP.
halaxa/json-machine	Pure-PHP pull parser over a stream, JSON Pointer support	Correct and memory-flat, but the tokenizer runs in PHP: 3.5x to 7.5x slower than this extension on structured data in the benchmark (slightly faster on long strings), with OPcache JIT on.
salsify/jsonstreamingparser, cerbero90/lazy-json	Pure-PHP SAX / lazy collections	Same limitation as json-machine: per-byte work in PHP. Not benchmarked here.
Existing PHP bindings of libyajl (rryqszq4/php-yajl, neikos/yajl-php, sfalvo/php-yajl)	`yajl_generate` / whole-document decode	Written for PHP 5 and unmaintained since 2010-2017. They decode whole documents rather than stream array elements. They also register the extension name `yajl`, which is why this one is called `yajl_stream`.
An external process (`jq --stream`, a sidecar in another language)	Streams JSON outside PHP	A process to spawn and supervise, plus a second serialization round-trip back into PHP.

What it solves

Constant memory regardless of document size. In the benchmark, peak PHP memory is 2-4 MB for every document, and process RSS stays at the level of an empty PHP process. Only one read buffer and the elements decoded from it are alive at once.
C-speed tokenizing. On number-heavy and deeply nested documents it is faster than json_decode (1.26x and 1.1x to 1.9x in the benchmark), because it never materializes the full tree. On the other four documents (catalog, text, scalars, wide) json_decode is 1.1x to 1.85x faster, most on long strings and tiny scalars. Part of the reason: libyajl alone validates strings less strictly than json_decode, so the extension checks them a second time (see "Semantics and limits"), which costs 4% to 54% depending on the content.
Selecting a nested array by JSON Pointer. '/items' and '/data/items' work, including ~0/~1 escapes and keys containing U+0000. Only the array at exactly that path matches: an array with the same key elsewhere in the document is ignored.
Early stop. When the target array closes, parsing stops and the rest of the file is not read.
Same values as json_decode($json, true). Objects become associative arrays, integer-like keys become int keys, duplicate keys keep the last value, integers beyond 64 bits become floats, and invalid UTF-8 or an unpaired surrogate escape is an error. Where libyajl alone is laxer than json_decode (it accepts overlong UTF-8, encoded surrogates and lone surrogate escapes), the extension applies ext/json's rules itself.
Clean errors. Malformed JSON, read failures and files that cannot be opened throw YajlStream\ParseException; the extension emits no PHP warnings (a warning that the stream layer raises while opening becomes the exception's getPrevious()). Elements that were fully parsed before the error are still returned first, independent of the buffer size. After that, iteration and every fetch() throw the same error.

API

namespace YajlStream;

/** @not-serializable */
final class ArrayParser implements \IteratorAggregate
{
    /**
     * @param string $filePath   path or stream wrapper URL; must not contain NUL bytes (ValueError)
     * @param string $pointer    '' = the document itself is the array; '/items', '/data/items', ...
     * @param int    $bufferSize bytes read and fed to libyajl per step, 1..MAX_BUFFER_SIZE
     * @throws ParseException when the file cannot be opened
     * @throws \ValueError    when $bufferSize is out of range or $filePath contains a NUL byte
     * @throws \Error         when called twice on the same object
     */
    public function __construct(string $filePath, string $pointer = '', int $bufferSize = 65536) {}

    /** Largest accepted $bufferSize: 64 MiB. */
    public const MAX_BUFFER_SIZE = 67108864;

    /**
     * Single-pass iterator: keys are element indices, values the decoded elements (null included).
     * foreach calls it for you.
     * @throws \Error when the parser was already iterated, or on rewind() after next()
     */
    public function getIterator(): \Iterator {}

    /**
     * Next element, or null when the array (or the file) ends. A JSON null element is also
     * returned as null; use foreach when elements can be null.
     */
    public function fetch(): mixed {}
}

final class ParseException extends \Exception {}

function yajl_version(): string {} // linked libyajl version, e.g. "2.1.0"

Semantics and limits

Pointer. Pointer segments are object keys only. Array indices (/items/0) are not supported and never match. If the path is missing, or the value at the path is not an array, the parser yields nothing, as for an empty array. As in RFC 6901, '' is the document itself and '/' is the key "" (tests/024-pointer-edge-cases.phpt).
Duplicate keys on the pointer path. In {"items": [1], "items": [2]}, '/items' streams [1] and stops, while json_decode keeps the last value, [2]. A streaming parser cannot know that a later duplicate exists without reading the whole file (tests/024-pointer-edge-cases.phpt). Duplicate keys inside the elements behave like json_decode.
Numbers convert like json_decode: an integer that fits 64 bits is an int, a longer one a float, 1e400 is INF, -0 is int(0) and -0.0 is float(-0) (tests/020-numbers-like-json-decode.phpt).
Iteration is single-pass. The file is read as a stream, so a second foreach over the same parser throws Error instead of silently yielding nothing. Iteration and fetch() share one position: after a break, fetch() returns the next element, and a foreach after some fetch() calls continues from there. Keys stay the element indices in both cases (tests/018-foreach.phpt, tests/019-foreach-lifecycle.phpt).
fetch() and null elements. fetch() returns null both for a JSON null element and when the array ends, so a while (($x = $p->fetch()) !== null) loop stops at the first null element (tests/005-types.phpt). foreach has no such ambiguity.
No validation after the target array. Anything after the target array closes is not read, so it is not validated either: {"items":[1]} garbage yields 1 without an error, where json_decode reports a syntax error (tests/022-streams.phpt, tests/023-fuzz-differential.phpt).
Read errors of the stream surface as ParseException("Failed to read from the stream: …"), not as a truncated document (tests/022-streams.phpt).
Objects. Parsers cannot be cloned, serialized, or created without the constructor.

Benchmark

Read the ratios, not the absolute numbers. The run below was made in Docker (OrbStack VM, 8 CPUs, 8 GB) on a MacBook Pro with Apple M1 Pro, with other idle containers on the same VM. Absolute times depend on the machine. The ratios between parsers on the same machine are what carries over.

bench/benchmark.php generates six documents of different character, from a fixed seed (the same PHP version produces the same bytes). Each parser then walks every element of the target array:

Scenario	Content	File	Elements
`catalog`	API catalog dump: objects with 10 mixed fields	100 MB	335,651
`numbers`	telemetry: 16 full-precision floats plus ints per object	100 MB	280,286
`text`	articles: ~4 KB strings with escapes and multi-byte UTF-8	100 MB	24,522
`nested`	7-level trees of small objects and arrays	50 MB	12,568
`scalars`	top-level array of tiny ints, strings, floats, bools, nulls	50 MB	7,861,249
`wide`	huge elements: 2,000-key objects of ~60 KB	100 MB	1,923

Parsers compared:

yajl_stream: foreach over ArrayParser;
json_decode($json, true) on the whole document;
simdjson_decode($json, true) on the whole document;
simdjson_key_value($json, $path, true), which decodes only the target subtree but still needs the whole string;
halaxa/json-machine with ExtJsonDecoder(true).

Versions: PHP 8.5.11 NTS, OPcache JIT tracing, php -n; yajl_stream 0.1.0 on libyajl 2.1.0; simdjson PECL 4.0.0; halaxa/json-machine 1.2.6. Every measurement is a fresh process, and every parser runs 7 times per scenario. All parsers produce the same output: every 100th element is hashed and compared with json_decode.

Time relative to yajl_stream (below 1x is faster). The ranges cover the medians of two full runs, 7 runs each:

Scenario	File	yajl_stream	json_decode	simdjson_decode	simdjson_key_value	json-machine
`catalog`	100 MB	1x (0.63-0.66 s)	0.91-0.93x	0.64-0.71x	0.57-0.68x	5.88-6.47x
`numbers`	100 MB	1x (0.74-0.80 s)	1.26x	0.43-0.52x	0.41-0.47x	3.55-3.68x
`text`	100 MB	1x (0.54-0.57 s)	0.59-0.60x	0.27-0.28x	0.26-0.27x	0.90-0.93x
`nested`	50 MB	1x (0.72-0.85 s)	1.14-1.93x	0.81-1.26x	0.75-1.00x	6.87-7.54x
`scalars`	50 MB	1x (0.63-0.64 s)	0.54-0.55x	0.34-0.36x	0.32-0.33x	4.17-4.58x
`wide`	100 MB	1x (0.61 s)	0.82-0.88x	0.43-0.48x	0.40x	4.17-4.24x

Peak process RSS (an empty PHP process with the same flags is 27.5 MB). RSS is reported because the PHP allocator peak does not see memory that libyajl or simdjson allocate with malloc/new. The PHP peak for yajl_stream was 2-4 MB in every scenario.

Scenario	File	yajl_stream	json_decode	simdjson_decode	simdjson_key_value	json-machine
`catalog`	100 MB	28.8 MB	794 MB	1148 MB	1148 MB	29.5 MB
`numbers`	100 MB	28.5 MB	377 MB	641 MB	641 MB	29.4 MB
`text`	100 MB	28.5 MB	240 MB	440 MB	440 MB	29.4 MB
`nested`	50 MB	30.1 MB	1276 MB	1530 MB	1530 MB	29.8 MB
`scalars`	50 MB	28.6 MB	246 MB	461 MB	461 MB	29.4 MB
`wide`	100 MB	29.1 MB	481 MB	796 MB	796 MB	29.9 MB

What the numbers say:

Memory is where streaming wins. yajl_stream and json-machine stay at the empty-process level for any file size. json_decode needs 2.4x to 25x the file size and simdjson 4.4x to 31x; the deep trees of nested are the worst case.
Speed. simdjson is the fastest: simdjson_key_value by up to 3.8x, and it only ties on nested. yajl_stream beats json_decode on numbers and nested, and json_decode wins on catalog, text, scalars and wide (by 1.1x to 1.85x). json-machine is 3.5x to 7.5x slower, except on long strings (text), where it is slightly faster (0.9x). The whole-document parsers vary a lot between runs on nested (1.3 GB per run), which the ranges show.
When to pick what. If the document fits in memory comfortably, simdjson is faster. If it does not, or several workers run in parallel, yajl_stream gives C speed at constant memory.

Reproduce:

docker build -f bench/Dockerfile -t yajl_stream-bench .
docker run --rm yajl_stream-bench                       # full run, ~3.5 min; prints the markdown report
docker run --rm -e BENCH_SCALE=0.1 -e BENCH_RUNS=1 -e BENCH_ONLY=catalog,text yajl_stream-bench

Without Docker, measure with Xdebug disabled: with Xdebug loaded, an earlier run of json-machine was 6-8x slower.

Install

Requirements: PHP 8.2 or newer (NTS or ZTS), Linux or macOS, libyajl 2.x with its headers, and the usual build tools (phpize, a C compiler, make). Windows is not supported.

Install libyajl first; neither PIE nor ./configure installs it:

apk add yajl-dev                 # Alpine (runtime package: yajl)
apt-get install libyajl-dev      # Debian/Ubuntu (runtime package: libyajl2)
brew install yajl                # macOS

With PIE

PIE builds the extension from source, installs it and enables it:

pie install vlydev/yajl-stream
pie install vlydev/yajl-stream --with-yajl-stream=/opt/yajl   # libyajl in a non-standard prefix

Without --with-yajl-stream, the headers are searched in /usr/local, /usr, /opt/homebrew and /usr/local/opt/yajl. To install from a local checkout: pie repository:add path /path/to/checkout, then pie install 'vlydev/yajl-stream:*@dev' (pie install run inside the checkout ignores configure options).

In a Docker image (the bin image only carries the PIE phar, see the PIE docs):

FROM php:8.5-cli
RUN --mount=type=bind,from=ghcr.io/php/pie:bin,source=/pie,target=/usr/local/bin/pie \
    set -eux; \
    apt-get update; \
    apt-get install -y --no-install-recommends unzip libyajl-dev; \
    pie install --no-cache vlydev/yajl-stream; \
    rm -rf /var/lib/apt/lists/*

For install-php-extensions (mlocati/docker-php-extension-installer), the extension is not supported yet; docs/docker-php-extension-installer.md describes how to add it.

From source

phpize
./configure --with-yajl-stream            # or --with-yajl-stream=/path/to/libyajl/prefix
make
make test                                 # tests/*.phpt
make install
echo "extension=yajl_stream" > "$(php-config --ini-dir)/yajl_stream.ini"   # or add to php.ini

Verify:

php -r 'var_dump(phpversion("yajl_stream"), YajlStream\yajl_version());'

To detect the extension in application code, check class_exists(YajlStream\ArrayParser::class) rather than only extension_loaded().

Testing

Run make test to execute the phpt suite in tests/. The reference for every value is json_decode($json, true):

Values: all JSON types, numbers at the int64 and double limits, floats under a decimal-comma locale, escapes, surrogate pairs, strict UTF-8 and surrogate validation, object keys (duplicate, int-like, empty, U+0000).
Streaming: every buffer size from 1 to 80 bytes; stream wrappers (compress.zlib://, data:, a userspace wrapper that hands out 1-3 bytes per read); early stop without reading the tail; bounded memory on a large array, with fetch() and foreach.
Pointers: nested, escaped, decoys at other paths, the empty key, duplicate keys.
Errors: malformed input, read failures, open failures, sticky errors, elements before an error delivered for any buffer size.
API and lifecycle: foreach and fetch() sharing one position, single pass, rewind, by reference, exceptions from the loop body, double construction, serialization, cloning, argument validation.
Differential fuzzing (tests/023-fuzz-differential.phpt): random documents, pointers and buffer sizes must match json_decode. Byte-mutated documents must give the same outcome for every buffer size, and must be rejected whenever json_decode rejects them, except for garbage after the closed target array. The run is deterministic. For a longer one with another seed:
```
sed -n '/^--FILE--$/,/^--EXPECT/p' tests/023-fuzz-differential.phpt | sed '1d;$d' > /tmp/fuzz.php
YAJL_FUZZ_SEED=7 YAJL_FUZZ_SCALE=50 php -n -d extension=modules/yajl_stream.so /tmp/fuzz.php
```

Every test, the fuzzer included, also runs under valgrind (USE_ZEND_ALLOC=0) and AddressSanitizer in CI; see CLAUDE.md for the exact procedure.

CI and Docker

.github/workflows/ci.yml runs on every push to main and on every pull request:

Job	What it checks
`bench-smoke`	the benchmark on small documents; fails if any parser's output differs from `json_decode`
`checks`	PHP 8.2-8.5 on Debian (x86_64 and arm64), Alpine/musl, ZTS: `-Werror` build (gcc, and clang on Alpine), phpt suite, valgrind (0 errors, 0 leaks), ASAN, and a control run per tool with a planted bug that must be caught
`macos`	Homebrew PHP and libyajl, Apple clang `-Werror` build, phpt suite
`pie`	`pie install` from the checkout, with and without `--with-yajl-stream`, then a smoke test through the enabled ini
`arginfo`	`yajl_stream_arginfo.h` matches what `gen_stub.php` generates from the stub

ASAN runs on PHP 8.5+ only on glibc. Older PHP versions there load extensions with RTLD_DEEPBIND, which the sanitizer runtime refuses (Zend/zend_portability.h). On Alpine it runs on every version. Valgrind covers all versions.

Every job runs ci/run.sh inside the Dockerfile image, and runs the same way locally:

docker build --build-arg PHP_VERSION=8.4 -t yajl_stream:8.4 .           # PHP_VARIANT=cli-alpine, zts-trixie, ...
docker run --rm yajl_stream:8.4 ci/run.sh test valgrind asan           # sources baked into the image
docker run --rm -v "$PWD":/src:ro yajl_stream:8.4 ci/run.sh test       # current checkout, no rebuild

ci/run.sh builds in /tmp and never writes to the source tree. Checks: test, valgrind, asan, valgrind-control, asan-control, arginfo, pie.

License

MIT, see LICENSE. libyajl, which the extension links against, is under the ISC license.

vlydev / yajl-stream

Package info

Security

Statistics