survos / folio-bundle

2.30.10 2026-09-17 13:31 UTC

Requires

• php: ^8.5
• ext-pdo_sqlite: *
• ext-sqlite3: *
• doctrine/dbal: ^4.4
• doctrine/doctrine-bundle: ^2.14|^3.0
• doctrine/orm: ^3.7
• league/commonmark: ^2.4
• survos/babel-bundle: ^2.10
• survos/data-contracts: ^2.5
• survos/dataset-bundle: ^2.5
• survos/field-bundle: ^2.5
• survos/jsonl-bundle: ^2.5
• survos/search-bundle: ^2.5
• symfony/ai-agent: ^0.13
• symfony/config: ^8.1
• symfony/console: ^8.1
• symfony/dependency-injection: ^8.1
• symfony/framework-bundle: ^8.1
• symfony/http-client: ^8.1
• symfony/http-kernel: ^8.1
• symfony/intl: ^8.1
• symfony/routing: ^8.1
• tacman/gcf-php: ^1.0
• twig/twig: ^3.24|^4.0
• voku/stop-words: ^2.0
• zenstruck/bytes: ^1.1

Requires (Dev)

• api-platform/core: ^4.3
• phpstan/phpstan: ^2.1
• survos/fetch-bundle: ^2.5
• survos/iiif-bundle: ^2.5
• survos/imgproxy-bundle: ^2.5
• survos/tabler-bundle: ^2.5
• symfony/ux-chartjs: ^3.0

Suggests

• api-platform/core: Required for the PhotoGrid Twig component's data source (Row's #[ApiResource]).
• presta/sitemap-bundle: For folio:sitemap — static per-folio XML sitemaps of item pages.
• survos/fetch-bundle: Provides ChunkDownloader for HTTP folio:pull downloads.
• survos/import-bundle: Produces normalized JSONL files before ingesting them into folios.
• survos/schema-org-bundle: Publishes JSON-LD on the row detail page (RowSchemaOrgBuilder), reading the #[SchemaOrg] attributes survos/data-contracts already puts on the item DTOs. Without it the row page has no structured data.
• survos/settings-bundle: Lets RowMenu gate OCR/Handwriting behind a per-user beta_features setting instead of only the dev environment.
• symfony/ux-twig-component: Required for the PhotoGrid Twig component.

MIT

• ux
• symfony-ux

Published on Packagist.org on 2026-09-17 13:33 UTC

README

Folio stores normalized/enriched dataset JSONL as portable SQLite archive files. It is the database, archive, and browsing layer for data that has already been normalized by dataset/import tooling.

harvest and md produce normalized JSONL. folio:ingest turns that JSONL into a standalone folio SQLite file. Consumers such as zm can use the Symfony/DataContracts stack when present, while Python/R/SQLite users can query the archive directly.

Required: survos/field-bundle, survos/data-contracts. Suggested for ingest/write workflows: survos/jsonl-bundle, survos/import-bundle.

See docs/configuration.md for the required multi-connection Doctrine setup. See docs/archive-metadata.md for the standalone archive metadata contract. See docs/presentation-layer.md for the proposal to use folios as narrative institutional presentation packages.

Archive Contract

A folio file stores canonical rows in item and self-describing metadata alongside them:

• schema_table and schema_property describe observed DTO types and fields in this archive.
• schema_property.stats stores field profile output from survos/jsonl-bundle's profiler.
• docs stores generated JSON/Markdown documentation for humans, report writers, and AI agents.
• generated dto_* SQLite views project JSON fields into query-friendly columns.
• term_set and term store standalone controlled vocabularies and facets.

The metadata snapshot describes actual observed data, not the entire DTO contract universe. DTO classes from survos/data-contracts annotate observed fields with labels/descriptions when available, but consumers do not need PHP code to understand an archived folio.

Search and Publication Notes

• folio:ingest loads rows, snapshots observed schema/docs/views, and rebuilds the SQLite FTS5 table item_fts.
• Existing folios can rebuild search with bin/console folio:fts:rebuild <provider/dataset> --query="search terms".
• folio:archive refreshes archive metadata before packaging.
• FTS tables are derived data. Published archive files may drop item_fts, VACUUM, compress, ship, then rebuild FTS on the consuming side.
• SQLite views and docs are also derived from persisted metadata, but they are intentionally lightweight and useful for standalone consumers.
• Vector search is intentionally deferred. When added, start with a hybrid SQLite design: FTS5/BM25 for exact keyword strength, sqlite-vec for semantic retrieval, and Reciprocal Rank Fusion to merge ranks without normalizing incompatible score scales. Reference: https://ceaksan.com/en/hybrid-search-fts5-vector-rrf

Direct SQLite Examples

select * from schema_table where kind = 'dto';
select * from schema_property where table_id = ? order by position;
select local_id, label, dto_type, dto_data, extras from item limit 20;
select * from dto_document limit 20;
select id, type, audience, body from docs order by position;

TODO

• Add fieldSet support to the api-grid spreadsheet view to avoid displaying every DTO field at once.
• Rebuild views/docs on restore, not only FTS, if the archive was packaged without them.