Category Archives: Uncategorized

Automatic correction of OCR

A milestone: I have begun automatically correcting the OCR errors in the 46 volumes of Danby Pickering’s ‘Statutes At Large’, and have uploaded the improved text to Github.

Given the quantity of text I’m dealing with – the Pickering series alone amounts to over fifteen million words – correcting each volume ‘by hand’ is obviously impractical. Bulk ‘find and replace’ is an improvement, but still not fast enough to be practical.

Such repetitive tasks are grist to the digital mill. So, using this list of common OCR errors, augmented with others I’ve found, and a one line bash script, automatic improvement of the texts has commenced.

The results are obviously an improvement. Nevertheless, the texts still aren’t great. There are still many spelling errors. As I used spaces as separators, words with punctuation attached are uncorrected. The many problems arising from layout are still to be faced.

But this is an important step forward.

Introducing The Statutes Project

The aim of the Statutes project is quite simple: to put the majority of historic English legislation online in accessible, useful formats, readable by humans and machines alike, with accompanying metadata, without any financial, technical or legal obstacles to use or adaption.

The simplicity of this statement masks the many difficulties: finding the laws, digitizing them, turning page images into clean, correct text, and so on. And doing so  without having an entire life devoured by spell checking and hand correction.

The many volumes of statutes compiled through the last three centuries, coupled with mass digitization projects such as those run by Google Books and the Internet Archive, along with optical character recognition and text correction tools, does at least allow for the hope that useable – but not perfect – texts can be produced with a minimum of effort.

The focus will be on the late seventeenth, eighteenth and early nineteenth centuries, the ‘long eighteenth century’ that is central to my own historical studies. Expect a concentration on matters relating to debt and debtors; that is the subject of my PhD.

This blog is more a notebook than a full archive of legislation, although that is the long-term hope. It will cover the technical side more than the theoretical, although that won’t be absent. When there’s a sufficient corpus, quantatively and qualatively, there will be some preliminary attempts at analysis, little games aiming to investigate the possibilities.

Future posts will discuss the project in more detail, covering the source volumes, the software, textual analysis, dissemination, and undoubtably the many trials and tribulations produced by a simple idea rashly executed.