cerbero / lazy-json
Framework-agnostic package to load JSONs of any dimension and from any source into Laravel lazy collections.
Installs: 281 526
Dependents: 1
Suggesters: 0
Security: 0
Stars: 248
Watchers: 3
Forks: 4
Open Issues: 0
Requires
- php: ^8.1
- cerbero/json-parser: ^1.1
- illuminate/collections: >=8.12
Requires (Dev)
- pestphp/pest: ^2.16
- phpstan/phpstan: ^1.9
- scrutinizer/ocular: ^1.8
- squizlabs/php_codesniffer: ^3.0
- tightenco/duster: ^2.0
This package is auto-updated.
Last update: 2024-11-30 02:18:53 UTC
README
LazyCollection::fromJson($source, 'data.*.users.*') ->map($this->mapToUser(...)) ->filter($this->filterUser(...)) ->values() ->chunk(1_000) ->each($this->storeUsersChunk(...));
Framework-agnostic package to load JSON of any size and from any source into Laravel lazy collections.
Lazy JSON recursively turns any JSON array or object into a lazy collection, consuming only a few KB of memory while parsing JSON of any dimension.
It optionally allows to extract only some sub-trees, instead of the whole JSON, with an easy dot-notation syntax.
Under the hood, ๐งฉ JSON Parser is used to parse JSONs and extract sub-trees.
Need to lazy load items from paginated JSON APIs? Consider using ๐ผ Lazy JSON Pages instead.
๐ฆ Install
Via Composer:
composer require cerbero/lazy-json
๐ฎ Usage
๐ฃ Basics
Depending on our coding style, we can call Lazy JSON in 3 different ways:
use Cerbero\LazyJson\LazyJson; use Illuminate\Support\LazyCollection; use function Cerbero\LazyJson\lazyJson; // auto-registered lazy collection macro $lazyCollection = LazyCollection::fromJson($source); // static method $lazyCollection = LazyJson::from($source); // namespaced helper $lazyCollection = lazyJson($source);
The variable $source
in our examples represents any JSON source. Once we define the source, we can chain any method of the Laravel lazy collection to process the JSON in a memory-efficient way:
LazyCollection::fromJson($source) ->values() ->map(/* ... */) ->where(/* ... */) ->each(/* ... */);
๐ง Sources
A JSON source is any data point that provides a JSON. A wide range of sources are supported by default:
- strings, e.g.
{"foo":"bar"}
- iterables, i.e. arrays or instances of
Traversable
- file paths, e.g.
/path/to/large.json
- resources, e.g. streams
- API endpoint URLs, e.g.
https://endpoint.json
or any instance ofPsr\Http\Message\UriInterface
- PSR-7 requests, i.e. any instance of
Psr\Http\Message\RequestInterface
- PSR-7 messages, i.e. any instance of
Psr\Http\Message\MessageInterface
- PSR-7 streams, i.e. any instance of
Psr\Http\Message\StreamInterface
- Laravel HTTP client requests, i.e. any instance of
Illuminate\Http\Client\Request
- Laravel HTTP client responses, i.e. any instance of
Illuminate\Http\Client\Response
- user-defined sources, i.e. any instance of
Cerbero\JsonParser\Sources\Source
For more information about JSON sources, please consult the ๐งฉ JSON Parser documentation.
๐ฏ Dots
If we only need a sub-tree of a large JSON, we can use a simple dot-notation syntax to extract the desired path (or dot).
Consider this JSON for example. To extract only the cities and avoid parsing the rest of the JSON, we can set the results.*.location.city
dot:
$source = 'https://randomuser.me/api/1.4?seed=json-parser&results=5'; $dot = 'results.*.location.city'; LazyCollection::fromJson($source, $dot)->each(function (string $value, string $key) { // 1st iteration: $key === 'city', $value === 'Sontra' // 2nd iteration: $key === 'city', $value === 'San Rafael Tlanalapan' // 3rd iteration: $key === 'city', $value === 'ฺฏุฑฺฏุงู' // ... });
The dot-notation syntax is very simple and it can include any of the following 4 elements:
- a key of a JSON array, e.g.
0
- a key of a JSON object, e.g.
results
- a dot to indicate the nesting level within a JSON, e.g.
results.0
- an asterisk to indicate all items within an array, e.g.
results.*
If we need to extract several sub-trees, Lazy JSON supports multiple dots:
$dots = ['results.*.gender', 'results.*.email']; LazyCollection::fromJson($source, $dots)->each(function (string $value, string $key) { // 1st iteration: $key === 'gender', $value === 'female' // 2nd iteration: $key === 'email', $value === 'sara.meder@example.com' // 3rd iteration: $key === 'gender', $value === 'female' // 4th iteration: $key === 'email', $value === 'andrea.roque@example.com' // ... });
๐ Change log
Please see CHANGELOG for more information on what has changed recently.
๐งช Testing
composer test
๐ Contributing
Please see CONTRIBUTING and CODE_OF_CONDUCT for details.
๐งฏ Security
If you discover any security related issues, please email andrea.marco.sartori@gmail.com instead of using the issue tracker.
๐ Credits
โ๏ธ License
The MIT License (MIT). Please see License File for more information.