README

Folio stores normalized/enriched dataset JSONL as portable SQLite archive files. It is the database, archive, and browsing layer for data that has already been normalized by dataset/import tooling.

harvest and md produce normalized JSONL. folio:ingest turns that JSONL into a standalone folio SQLite file. Consumers such as zm can use the Symfony/DataContracts stack when present, while Python/R/SQLite users can query the archive directly.

Required: survos/field-bundle, survos/data-contracts. Suggested for ingest/write workflows: survos/jsonl-bundle, survos/import-bundle.

See docs/configuration.md for the required multi-connection Doctrine setup. See docs/archive-metadata.md for the standalone archive metadata contract. See docs/presentation-layer.md for the proposal to use folios as narrative institutional presentation packages.

Archive Contract

A folio file stores canonical rows in item and self-describing metadata alongside them:

schema_table and schema_property describe observed DTO types and fields in this archive.
schema_property.stats stores field profile output from survos/jsonl-bundle's profiler.
docs stores generated JSON/Markdown documentation for humans, report writers, and AI agents.
generated dto_* SQLite views project JSON fields into query-friendly columns.
term_set and term store standalone controlled vocabularies and facets.

The metadata snapshot describes actual observed data, not the entire DTO contract universe. DTO classes from survos/data-contracts annotate observed fields with labels/descriptions when available, but consumers do not need PHP code to understand an archived folio.

Search and Publication Notes

folio:ingest loads rows, snapshots observed schema/docs/views, and rebuilds the SQLite FTS5 table item_fts.
Existing folios can rebuild search with bin/console folio:fts:rebuild <provider/dataset> --query="search terms".
folio:archive refreshes archive metadata before packaging.
FTS tables are derived data. Published archive files may drop item_fts, VACUUM, compress, ship, then rebuild FTS on the consuming side.
SQLite views and docs are also derived from persisted metadata, but they are intentionally lightweight and useful for standalone consumers.
Vector search is intentionally deferred. When added, start with a hybrid SQLite design: FTS5/BM25 for exact keyword strength, sqlite-vec for semantic retrieval, and Reciprocal Rank Fusion to merge ranks without normalizing incompatible score scales. Reference: https://ceaksan.com/en/hybrid-search-fts5-vector-rrf

Direct SQLite Examples

select * from schema_table where kind = 'dto';
select * from schema_property where table_id = ? order by position;
select local_id, label, dto_type, dto_data, extras from item limit 20;
select * from dto_document limit 20;
select id, type, audience, body from docs order by position;

TODO

Add fieldSet support to the api-grid spreadsheet view to avoid displaying every DTO field at once.
Rebuild views/docs on restore, not only FTS, if the archive was packaged without them.

Catalog consumers and published shared folios

survos_folio.read_only: true opens FolioService contexts using SQLite mode=ro, disables schema auto-migration/folio-row creation and avoids WAL mode changes. DatasetBundle's SQLite middleware must also include read-only parameter support. Publish a compatible, completed, checkpointed folio before sharing; the reader will not upgrade it. Explicit archive restore and inflate commands still write their selected local targets.

archive_api_prefix defaults to /folio and controls folio:pull listing/fallback download URLs independently of this app's route_prefix or a remote /f browse prefix. Download URLs returned by the catalog remain authoritative. local_passthrough: true skips already-present shared folios even with --force.

PeriodicalCoverageService::compute() preserves periodical block/article/ad distinctions and returns counts plus per-issue details. It scans folio rows: run it offline when recording catalog summaries, never as a directory-page cache-miss fallback. zm's publisher command persists the compact versioned summary in its registry; Ink consumes that recorded API data.

Optional collection table

The collection catalogue uses survos/grid-bundle for its server-rendered cells. It is suggested, not required: install and enable it in applications that use survos_folio_collection. Without it, the page displays an installation warning. Folio does not require Simple DataTables or pentiminax/ux-datatables.

Search and pagination remain server-side (50 folio artifacts per page); the grid's client-side paging, search and ordering are disabled so they cannot misleadingly operate on only the current page. Row and photo browsing can use the existing optional API Platform integration independently of the catalogue grid.

survos / folio-bundle

Package info

Security

Fund package maintenance!

Statistics