README

Installation

Install via Composer (pure PHP; always works without a native extension):

composer require eloquage/chunk

Optional native acceleration

The v1 release is pure PHP. packages/chunk/native/ stays empty and no TypePHP Composer dependency is required. The public contract is complete without an extension.

The future feasibility/build path is Docker-only and uses extension mode:

docker pull ghcr.io/eloquage/typephp-builder:latest
docker/typephp/build-package.sh chunk

Generated build/*.so files are optional maintainer artifacts, not a release gate. See TYPEPHP.md for the extension contract.

Usage

use Eloquage\Chunk\Chunk;

$chunk = new Chunk();

echo $chunk->name(); // chunk

$records = $chunk->split('A short source document.', [
    'size' => 10,
    'overlap' => 2,
]);

// Each record has: text, index, start, end.

Every record has the exact source text, a zero-based index, and inclusive start / exclusive end offsets measured in UTF-8 characters. Character windows advance by size - overlap, so ordinary windows have the requested overlap. overlap is a bounded target for packed Markdown structure: complete paragraphs or other structural pieces are repeated only when they fit the next budget.

For natural boundaries, use pack. Plain packing prefers paragraphs and then lines; Markdown packing also keeps ATX headings with their first body paragraph and fenced code blocks intact:

$records = $chunk->split("# Guide\n\nKeep this section together.\n\n```php\nreturn true;\n```", [
    'strategy' => 'pack',
    'format' => 'markdown',
    'size' => 80,
]);

Select unit => 'tokens' only when the constructor receives the counter your application already uses. The callable accepts a string and returns a non-negative integer; the package does not create or require a tokenizer:

$chunk = new Chunk(static fn (string $value): int => count(preg_split('/\s+/', trim($value), -1, PREG_SPLIT_NO_EMPTY)));
$records = $chunk->split($text, ['unit' => 'tokens', 'size' => 128]);

strategy is window or pack; format is plain or markdown; and unit is characters or tokens. size must be positive, overlap must be non-negative and smaller than size, and unknown options are rejected. Empty input returns an empty list. An oversized fenced block is returned intact as the one documented budget exception, including an unclosed fence through the end of the input.

Testing

composer test
vendor/bin/pest --coverage --min=90

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Pull requests and issues are welcome on GitHub.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

License

The MIT License (MIT). Please see License File for more information.

Development

See AGENTS.md for agent context, tests, and TypePHP Docker builds.

eloquage / chunk

Package info

Security

Fund package maintenance!

Statistics