Federal-grade digital preservation, proven at the scale of a historical newspaper preservation program.

Under Aretum (Prime), StandardData rebuilt the OCR and metadata pipeline for a historical newspaper preservation program — the engine behind the program's public archive. Roughly 300,000 historic newspaper pages have been reprocessed at full federal preservation standards, at about $50 per ~5,000-issue batch, 100× cheaper than commercial OCR and 99% faster than the legacy on-prem path.

That work is the basis for the program's public announcement that the pipeline will be open-sourced in 2026 for community use and development. Under Aretum (Prime), our codebase has been prepared to federal OCIO open-source compliance standards — third-party notices, license headers, dependency audits, CIO-level review — the posture any federal cultural-heritage institution would expect of vendor work touching irreplaceable collections.

Same page, reprocessed: clean sentences, recovered named entities (Shipka Pass, Suleiman, Constantinople, Gabrova) — fully searchable for researchers.

Source: Team Aretum, Historic Newspaper OCR pipeline — British Library presentation (10 Sep 2025), slides 13–14. PDF ↗

What we built

Distributed serverless cloud compute. The pipeline runs on AWS Batch and processes newspaper pages in parallel — workloads that used to take weeks on a single on-prem server complete in hours in the cloud. The same code path also runs locally on a workstation, on an isolated network, or on any organization's own distributed compute, with no rewrite required. Sovereignty is a deployment choice, not a code rewrite.

Underneath: Tesseract built from source on AMD64, the AmericanStories segmentation model in ONNX for column and line detection, and standards-aligned outputs — ALTO XML word and line bounds, METS structural metadata, Dublin Core descriptive metadata, NISO image headers. Every page emits a searchable PDF plus the metadata payload the public archive's full-text search expects.

• Operator-friendly delivery: program staff submit jobs from a simple Python CLI shipped to PyPI; the pipeline scales up on demand and back to zero when idle.
• Auditable production: every change traceable; production container carries zero critical CVEs; Terraform-managed infrastructure with multi-stage GitLab CI/CD.
• fsspec-generalized I/O: the same pipeline reads from and writes to S3 or local disk with no code change — the foundation for an air-gap deployment when needed.

Why it matters — four reasons this approach is different

Inside the pipeline — from pixels to structured, searchable text

The segmentation step is where most legacy OCR fails. Newspapers are dense, multi-column, and frequently mixed with photographs, ads, and maps. Bad zoning means text from one column bleeds into the next; words appear in the wrong reading order; mixed-content pages are abandoned entirely. We retrained the segmentation model on real historic newspaper pages and fixed the ALTO ordering algorithm so reading order survives multi-column layouts. The result is dense, accurate full-text coverage that downstream search and discovery can actually use.

Source: Team Aretum, Historic Newspaper OCR pipeline — British Library presentation (10 Sep 2025). PDF ↗

How we know it works

ALTO output carries per-word confidence. On real pages from the program's public archive, that means recovered proper nouns, intact sentences, and searchable corpora where the 2005-era OCR delivered garbled tokens. A handful of word-level confidences from a single re-OCR'd page: Monterey 0.96, Highland 0.96, County 0.96, December 0.95, Doctrine 0.95, Forcible 0.96, Increase 0.94, Foreign 0.91. That is the difference between an archive scholars can search and one they cannot.

The defence of the Shipka Pass. The blockade maintained by Suleiman's forces. Both sides waiting for more troops, Constantinople, Thursday, Aug. 30, 1877.Recovered text from a reprocessed NY Tribune (1877) page. Source: Team Aretum, Historic Newspaper OCR pipeline — British Library presentation (10 Sep 2025).

What the program got

• Searchable, standards-compliant outputs — every page emits ALTO XML plus a searchable PDF, conformant with a historical newspaper preservation program's technical guidelines for preservation and ingest into the program's public archive.
• Operator-friendly, secure, auditable — program staff submit jobs from a simple command; the pipeline scales up on demand and back to zero when idle; every change traceable; production container carries zero critical CVEs.
• Cost profile makes full-archive reprocessing viable — the kind of project that used to be a multi-year planning exercise is now economically routine.

Why this matters beyond newspapers

The architecture and the standards posture (ALTO, METS, Dublin Core, NISO) extend to any cultural-heritage corpus that needs structured, searchable preservation. The segmentation and OCR stack tuned for English newspapers carries directly to multi-language and multi-script materials, and to source media beyond microfilm.

• Multi-language & multi-script: the OCR stack extends to German (Fraktur and Antiqua), Yiddish (Hebrew script), Polish, Russian, Hungarian, Czech, and Ladino — the core language span of most American cultural-heritage holdings.
• Source-media adaptability: battle-tested on microfilm-scanned newspapers, architected for adaptation to bound volumes, typescripts, photographs with embedded text, and handwritten survivor testimonies.
• Sovereignty by default: on-prem and air-gap-capable deployment for sensitive, donor-restricted, or pre-clearance material; cloud burst when full-archive scale is needed.
• Federal compliance posture: delivered to federal OCIO open-source compliance standards — the posture any federal cultural-heritage institution would expect of vendor work touching irreplaceable collections.
• Standards-aligned output: ALTO, METS, Dublin Core, NISO — directly ingestible by modern discovery and preservation systems.

Open-source in 2026

The program has publicly announced the pipeline will be open-sourced in 2026 for community use and development. The codebase has been prepared to federal OCIO open-source compliance standards — third-party notices, license headers, dependency audits, CIO-level review — the same discipline any federal cultural-heritage institution would expect of vendor work touching irreplaceable collections.

If you have a federal preservation problem, or a commercial one with similar contours — millions of source documents, strict standards, irreplaceable originals — we are happy to walk through how this work would map to your environment.