survos / alto-raw

2.43.11 2026-10-20 15:23 UTC

Requires

• php: ^8.5
• ext-dom: *
• ext-hash: *
• ext-libxml: *
• ext-phar: *
• ext-zlib: *
• survos/bagit: ^2.43.9
• survos/data-contracts: ^2.0

Requires (Dev)

• phpunit/phpunit: ^13.0

MIT

Published on Packagist.org on 2026-10-20 15:23 UTC

README

survos/alto-raw converts cached Penn newspaper sources to PHP-generated v2 issue packs and Harvest raw JSONL, preserving original METS and ALTO in a verified BagIt. Requires PHP 8.5, DOM, Phar, zlib, survos/bagit and survos/data-contracts. No Symfony, Harvest, Python or HTTP runtime is required.

Convert and preserve

use Survos\AltoRaw\PennConverter;
use Survos\AltoRaw\BagArchive;

$converter = new PennConverter();
$receipt = $converter->convert(
    $capture,
    'batch_pst_aura_ver01',
    'sn83032227',
    $newOutputDirectory,
    // Omit the selection for the whole title:
    ['sn83032227-1857-06-24-ed-1'],
);
$converter->createBag($newOutputDirectory, 'batch_pst_aura_ver01', $newBagDirectory);
(new BagArchive())->pack($newBagDirectory, $newTarFile);

The capture layout matches the Penn capture produced by Harvest and Depot: title.json, <issue>/issue.json, and batches/<batch>/{bagit.txt,bag-info.txt,manifest-md5.txt,tagmanifest-md5.txt,data/...}. Each issue's payload directory includes METS, ALTO and seq-N.json canvases. Optional <issue>/rights.json or the previous complete.json carries recorded rights into the raw document; the converter does not invent a rights statement. JP2s are referenced but need not be present for offline conversion.

The converter checks the tagmanifest chain, selects issues from manifest JP2 identities, verifies METS/ALTO MD5s, matches each canvas to one unique image, and writes one gzip member per issue. It retains one issue in memory. A locked, temporary directory is verified before atomic publication. A repeat invocation verifies the existing output and requires the same manifest revision and selection. Use a new destination when changing inputs or scope.

Output includes:

• originals/<Penn payload path>: exact source XML bytes, including comments, whitespace, tails and namespace prefixes.
• source/: source manifests, title, issue, canvas and recorded rights evidence.
• packs/, catalog/, batches/: Python-compatible decoded v2 records, ranged gzip member offsets/checksums and an explicit completion scope.
• raw/doc.jsonl.gz, meta/dataset.json: ONI raw import documents and metadata.
• checksums.json: SHA-256 inventory of all generated payloads.

The BagIt wraps these under data/ with SHA-256 manifests and the dataset external identifier. BagArchive produces an uncompressed deterministic tar; destinations must not exist. This is a preservation bag for the selected XML and derived raw, not a claim that every upstream JP2, TIFF and PDF was acquired. Depot verifies the JP2s separately before the application workflow advances.

XmlTree deliberately follows the lossy Python representation: it omits comments, processing instructions, tail text and formatting whitespace. DTD inputs are rejected and external entity expansion is disabled. Never reconstruct an archival original from this tree. SourceXml and OriginalXmlSource are also available for callers that only need verified exact-byte XML preservation.

Lebanon parity tests

Fixtures use The Lebanon advertiser, sn83032227, June 24, 1857, edition 1, batch_pst_aura_ver01. The complete four-page issue, both METS files and original raw row are checked against the existing Python output. Golden expectations were extracted from the previous capture, not generated by PHP. Source provenance is recorded in tests/Fixtures/lebanon/provenance.json; gzip fixture decompression reproduces original XML bytes. Tests need no network or images.

composer install
composer test
# From mono with BagIt's test dependencies installed:
lib/bagit/vendor/bin/phpunit -c lib/alto-raw/phpunit.xml.dist

Tests cover decoded pack/raw parity, original bytes, deterministic output, BagIt/tar verification, retries, corruption and duplicate canvas identities. The Python implementation under survos_alto_raw/ remains an unchanged reference. Its archive module is also the frozen LOC implementation. This PHP implementation does not change or restart the LOC fleet.

Depot supplies the application command penn:run: inventory, local reuse/download, convert, bag, immutable S3 upload, authenticated Harvest workflow request, fresh folio retrieval and local rend. Its separate penn_pipeline queue and run.json checkpoints make application retries independent of conversion library internals.