README

JSON Lines (also known as NDJSON) encoding and decoding in PHP 8.2+: one JSON value per line, UTF-8, separated by "\n". An Encoder turns one value into one line, and an incremental Decoder accepts input in chunks of any size. There is no stream, file or generator handling in the package; you read and write with fread() / fwrite() (or any other source and sink) and let the codec do only the format. Requires ext-json.

Its sibling charm/csv has the same method names (encode(), decode(), finish()), without a shared interface.

composer require charm/jsonl

Usage

use Charm\Jsonl\{Decoder, Encoder};

// Decode a stream: read, decode, and call finish() at the end
$decoder = new Decoder();
$h = fopen('events.jsonl', 'rb');
while (!feof($h)) {
    foreach ($decoder->decode(fread($h, 65536)) as $event) {   // ['type' => 'click', 'x' => 10]
        // ...
    }
}
foreach ($decoder->finish() as $event) {                       // the last line, if the file has no final newline
    // ...
}

// Encode to a stream
$encoder = new Encoder();
$out = fopen('out.jsonl', 'wb');
foreach ($events as $event) {
    fwrite($out, $encoder->encode($event));                    // {"type":"click","x":10}\n
}

// Encode to a string, decode from a string
$line = (new Encoder())->encode(['a' => 1]);                   // "{\"a\":1}\n"
$decoder = new Decoder();
$values = [...$decoder->decode($line), ...$decoder->finish()]; // [['a' => 1]]

API

namespace Charm\Jsonl;

final class Encoder
{
    public function __construct(int $flags = JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE, int $depth = 512);
    public function encode(mixed $value): string;      // one line, including "\n"
}

final class Decoder
{
    public function __construct(bool $assoc = true, int $depth = 512, int $flags = 0);
    public function decode(string $chunk): array;      // values of the lines completed by this chunk
    public function finish(): array;                   // end of input: the last line; resets the decoder
}

final class ParseException extends \RuntimeException {}   // public readonly int $lineNumber; the JsonException is the previous exception

Encoder

JSON_THROW_ON_ERROR is always added, so a value json_encode() cannot represent throws a \JsonException.
JSON_PRETTY_PRINT is rejected with a ValueError, because it would put one value on several lines. A depth below 1 is a ValueError too.
The default flags write slashes and non-ASCII characters as they are. Passing $flags replaces the defaults, it does not add to them.
Raw newlines never occur in the output: json_encode() escapes them inside strings, and U+2028/U+2029 stay on one line either way.

Decoder

Each line is decoded on its own with json_decode($line, $assoc, $depth, $flags | JSON_THROW_ON_ERROR). So big integers, depth limits, JSON_BIGINT_AS_STRING and the invalid-UTF-8 flags behave exactly as they do for json_decode() on that line alone: a line is one JSON value, and the depth limit applies per line.
Lines end with "\n"; a "\r" directly before it is stripped (a "\r" anywhere else is JSON whitespace, a lone "\r" is not a line break).
A UTF-8 BOM at the very start of the input is stripped (once).
Empty lines are skipped. A line with only whitespace is not empty: it is invalid JSON.
A last line without "\n" is returned by finish(), which also resets the decoder so it can be used for another input.
Objects decode as associative arrays by default; assoc: false gives stdClass.
decode() returns the values of the lines completed by the chunk, as soon as the "\n" has arrived. The first three bytes of the input are held back until they are known not to be a BOM. Where you cut the input into chunks never changes the result, including cuts inside a multibyte character or between "\r" and "\n".
A line that is not valid JSON throws ParseException with its 1-based lineNumber (counted over the whole input, across chunks, empty lines included) and the \JsonException as previous exception. After a ParseException discard the decoder.
A line is held in memory until its "\n" arrives; chunks without a newline are appended, not re-scanned, so a very long line in small chunks stays linear.

Performance

Measured on PHP 8.5.12, Linux 6.8, Intel Xeon E5-2697 v3 @ 2.6 GHz, 200 000 lines, best of 9 runs after 2 warm-up runs (medians are within a few percent of the best). The data is generated by bench/bench.php (git-ignored, in bench/data/): "records" are nested objects of about 180 bytes with strings, unicode, floats, booleans, null and arrays (35 MB); "tiny" lines are {"i":1,"ok":true,"n":5} (6 MB). The Decoder is fed through fread(…, 65536). "collect" builds the complete result array (for every method in the table), "stream" counts the values and discards them. "explode() loop" reads the whole file into memory first, the others do not.

Command (starts the benchmark with the tracing JIT, then without it, and checks that the JIT compiled something):

composer install
php bench/run.php                      # --lines=50000 --runs=3 for a quick run

which runs php -d opcache.enable_cli=1 -d opcache.file_update_protection=0 -d opcache.jit_buffer_size=128M -d opcache.jit=1255 bench/bench.php and the same with -d opcache.jit_buffer_size=0 (opcache on, JIT off).

With the tracing JIT (opcache.jit=1255)

Decoding	records ms (MB/s)	vs explode loop	tiny ms (MB/s)	vs explode loop
`explode("\n")` + `json_decode()`, collect	464 (76)	1.00x	120 (51)	1.00x
`fgets()` + `json_decode()`, collect	526 (67)	0.88x	140 (44)	0.86x
`SplFileObject` + `json_decode()`, collect	558 (63)	0.83x	152 (41)	0.79x
`SplFileObject` + `json_decode()`, stream	453 (78)	1.02x	132 (47)	0.91x
`Decoder`, collect	508 (70)	0.91x	115 (53)	1.04x
`Decoder`, stream	478 (74)	0.97x	112 (55)	1.07x

Encoding to `php://memory`	records ms	tiny ms
`json_encode() . "\n"` + `fwrite()` loop	453	72
`Encoder` + `fwrite()`	455	73

Without the JIT (opcache on, opcache.jit_buffer_size=0)

Decoding	records ms (MB/s)	vs explode loop	tiny ms (MB/s)	vs explode loop
`explode("\n")` + `json_decode()`, collect	469 (76)	1.00x	132 (47)	1.00x
`fgets()` + `json_decode()`, collect	537 (66)	0.87x	156 (40)	0.84x
`SplFileObject` + `json_decode()`, collect	547 (65)	0.86x	162 (38)	0.81x
`SplFileObject` + `json_decode()`, stream	456 (78)	1.03x	140 (44)	0.94x
`Decoder`, collect	507 (70)	0.93x	133 (47)	0.99x
`Decoder`, stream	485 (73)	0.97x	127 (49)	1.04x

Encoding to `php://memory`	records ms	tiny ms
`json_encode() . "\n"` + `fwrite()` loop	461	75
`Encoder` + `fwrite()`	469	86

What this shows: json_decode() is almost all of the time, so the incremental Decoder runs at about the speed of the naive loops. When collecting, it is faster than fgets() and SplFileObject (1-9%); when discarding, SplFileObject is ahead on large lines (453 vs 478 ms) and behind on small ones. It is up to 9% behind an explode() over a whole in-memory file on larger lines, and it needs no more memory than one chunk plus one line. The Encoder is json_encode() plus validation of the flags; it is not faster. Without the JIT the numbers differ slightly (here by 0-16%, mostly on the small lines, where PHP-level work per line matters more); json_decode() and json_encode() are native code and do not change. The machine this was measured on was not idle, so differences of a few percent are noise.

Design note: decoding all complete lines of a chunk with one json_decode('[' . implode(',', $lines) . ']') was measured and rejected. It was 5% faster on the records and about 10-15% faster on tiny lines (best of 7, JIT and no JIT), but it is not equivalent: a line such as 1,2, or two lines [1 and 2], form valid JSON in the batch while each line alone is invalid, so errors would go undetected (or need a per-line re-validation that removes the gain).

Tests

composer install
vendor/bin/phpunit

Differential tests compare the decoder with a plain reference (explode("\n"), strip "\r", json_decode() per line) on a set of documents (big integers, depth limits, unicode, invalid UTF-8, \r\n, BOM, empty lines, invalid lines) with the input cut at every byte offset, at every chunk size from 1 to 9, at every pair of offsets in a small document, and on 400 random documents with random chunkings and options. The encoder is checked against json_encode() and by random round trips through the decoder. CI runs PHP 8.2-8.5 with deprecations and notices as failures.

License

MIT

charm / jsonl

Package info

Security

Statistics