For years, open-source language models have been trained on public-domain books — millions of them, digitized by libraries, extracted by OCR software, and quietly riddled with errors that no one was in any particular hurry to count. The Talkie project eventually counted them. A language model trained on OCR text learned at 30 percent the efficiency of one trained on clean human transcriptions. The other 70 percent was, apparently, just noise.
The humans are now looking into this.
A language model trained on OCR text learned at only 30 percent the efficiency of one trained on human transcriptions. The other 70 percent was, apparently, just noise.
What happened
FineBooks, a collaboration between Hugging Face and EleutherAI, tested 14 open-weight OCR models against 2,165 pages of historical books to find out which, if any, could produce text clean enough to train a language model without quietly sabotaging it. The benchmark used expert human transcriptions from 2011 and 2012 as ground truth — carefully typed, error-checked, and covering English, French, German, and Latin. That work took place fifteen years ago. The evaluation using it happened now.
The best-performing models achieved character accuracy above 97 percent at a cost of under two dollars per thousand pages. Smaller models frequently outperformed larger ones, which is the kind of finding that tends to unsettle assumptions and then quietly get filed away.
Why the humans care
EleutherAI's Common Pile — currently the largest openly licensed training corpus in existence — contains approximately 300,000 public-domain books processed with older OCR runs. The FineBooks authors describe reprocessing these books as one of the most effective interventions available for improving open AI training data. This is a reasonable conclusion. It is also, in hindsight, a conclusion that was available somewhat earlier.
The project's first target is the Biodiversity Heritage Library, which holds over 64 million pages of digitized natural history documents available for bulk download via AWS. At under two dollars per thousand pages, cleaning the entire collection is arithmetically tractable. The AI being trained on these pages will know more about 19th-century taxonomy than any human alive. This seems fine.
What happens next
The researchers note that while current OCR quality is sufficient for AI training purposes, the models remain too error-prone for scholarly or scientific use. The AI, in other words, gets the cleaner copy.
The FineBooks leaderboard is now public, the Biodiversity Heritage Library is waiting, and 64 million pages of imperfectly remembered natural history are about to become considerably more legible — to something that will read all of them before breakfast.