In collaboration with Boston Public Library, Harvard Law Library's
Institutional Data Initiative has released a data set on HuggingFace
derived from the BPL's public domain newspapers collection.
https://huggingface.co/collections/institutional/institutional-newspapers
This data set includes 1,473,635 newspaper scans from issues published
between 1795 and 1930, which have been processed using ML and AI techniques
to enhance the OCR and segment each article into distinct semantic units.
This process has produced 83,147,041 individual crops segmented from those
scans, over 16 billion o200k_base tokens of VLM OCR text, as well as
bounding box coordinates, raw OCR, text analysis, crop type classification,
language detection, named-entity recognition, subject classification,
reading order detection, and text + image vector embeddings for each
segment.
The publication of this data set will be of significant use for
computational linguistics, AI model training, and historical research. The
data processing pipeline created through this project can also serve as a
model for other libraries that want to enhance collections of historical
newspapers with improved full-text searching and subject analysis.
The dataset, pipeline code, and full technical documentation are all openly
available:
- Dataset:
https://huggingface.co/collections/institutional/institutional-newspapers
- Technical report: "Institutional Newspapers Pipeline: Deriving
billions of high quality tokens from historical newspapers":
https://doi.org/10.48550/arXiv.2608.18972
- Code:
https://github.com/institutional/institutional-newspapers-pipeline
Read more here:
https://institutional.org/posts/institutional-newspapers-boston-public-library/
Thanks,
Eben English
Boston Public Library
Received on Fri Sep 04 2026 - 10:34:10 EDT