Automatic correction of OCR

A milestone: I have begun automatically correcting the OCR errors in the 46 volumes of Danby Pickering’s ‘Statutes At Large’, and have uploaded the improved text to Github.

Given the quantity of text I’m dealing with – the Pickering series alone amounts to over fifteen million words – correcting each volume ‘by hand’ is obviously impractical. Bulk ‘find and replace’ is an improvement, but still not fast enough to be practical.

Such repetitive tasks are grist to the digital mill. So, using this list of common OCR errors, augmented with others I’ve found, and a one line bash script, automatic improvement of the texts has commenced.

The results are obviously an improvement. Nevertheless, the texts still aren’t great. There are still many spelling errors. As I used spaces as separators, words with punctuation attached are uncorrected. The many problems arising from layout are still to be faced.

But this is an important step forward.

March and April 2017 Updates

Work on the Statutes Project in March and April 2017:

0: Numerous corrections to Pickering’s series of Statutes at Large. Latest versions to be found, as ever, on Github.

1: More tables of statutes uploaded to Github. Currently, there are tables for public acts 1716 to 1736, with just 1721 missing. This I’ll upload shortly.

2: More legislation collected, to the point that the menus are getting unweildly and I’ll have to do some reorganizing. Acts added include:
The Murder Act of 1751, giving the corpses of the hanged to the surgeons (and occasioning many a riot).
The Regency Act 1729, allowing the Queen to govern whilst George the Second went off to Hanover.
The Septennial Act, extending the life of a parliament to seven years. A quite undemocratic act, had there been any meaningful suffrage

On the to do list for May 2017: due to the demands of my PhD, I’ll be working on the insolvent debtor relief acts from 1649 to 1813 over the next month; consequently, those texts will be corrected and added.

February 2017 Updates

Work on the Statutes Project in February 2017:

0: Numerous corrections to the OCR of the Pickering and Ruffhead editions of the Statutes At Large, uploaded to Github. Still a long way from readable, but getting there.

1: A new series OCR’d, or at least half a series. The Statutes of the Realm was the most academic, comprehensive and careful collection of acts, the text generally taken from the statute rolls themselves. Consequently, it is a typographical nightmare, and the OCR is  worse than for the – admittedly less reliable – series of the Statutes At Large. I have put on Github the text for two volumes (numbers 3 and 5) found on Google, and, thanks to the University of Southampton waving their No Derivatives license, the text for volumes 6 to 11 from the British Parliamentary Publications set, digitized by Soton, on

2: I have also started extracting the tables of acts from the OCR’d volumes, and uploading them to Github. The idea is to create a reliable list of legislation enacted, with the long title of each act. Given the length of titles, this will constitute a corpus of sufficient size for text mining and distance reading (I hope). It also constitutes the first step in creating metadata for this project.

3: Laws collected from around the web:

1536 27 Henry 8 c.19: An act limiting an order for Sanctuaries and Sanctuary persons.

The 1918 Parliament (Qualification of Women) Act: Allowing women to sit in Parliament, and the shortest statute at a mere 27 words, when preamble and short title clause are put aside.

4: And also: a short post on James I’s laws on sanctuary, over at my  Alsatia blog.

Planned for March: More acts collected from round the web, and more tables of statutes.

January Updates

Work on the Statutes project for January 2017:

0: OCRed two volumes of ‘The Statutes Of The Realm’, as digitized by Google. This is an important collection of legislation from Magna Charta to 1714, derived from close study of the original manuscripts, and contains laws not found in other collections. Raw OCR can be found on Github, and be warned, it’s very raw, as this series goes to great lengths to transcribe the original texts, with all their irregularities and without softening them for the modern eye. I hope to have more news concerning other volumes soon.

1: Tidying up of, and various corrections made to – by hand and by find-and-replace – the OCR of the Pickering and Ruffhead editions of The Statutes At Large. I needed to locate every use of the term ‘sanctuary’, so waded through the alphanumeric soup. Investigations into automatic correction are ongoing.

2: Added a bibliography for the digitized volumes of Ruffhead’s series of ‘The Statutes At Large.’ This gives the particular volumes I have OCRed, each with their own idiosyncracies and missing pages. Also added a bibliography for historic French statutes up to the revolution of 1789, though I have no plans to do anything with these right now.

3: Added to the statutes collected from round the web:
The Quartering Act 1774
1849: 12 & 13 Victoria c.92: Cruelty to Animals Act
The Debtors Act 1869

December Updates

Work on the statutes project in December 2016:

Not much accomplished this month, given the seasonal festivities. However:

0: On a whim I have OCR’d the 12 volumes collecting the statutes of Ireland, 1310 – 1800. The raw and very messy OCR can be found on Github. As with many of these retrospective collections, they are far from complete, neglecting statutes expired or repealed. I have not decided what to do with them, given that the English / British statutes take priority, and are daunting enough a task alone, but I will probably be producing corrected versions of legislation dealing with debt, as and when I need them. After the Union of 1800, Irish legislation is to be found in the main body of United Kingdom law.

1: Added to the general collection of statutes gathered from around the internet: the infamous ‘Cat and Mouse’ act of 1913, used against imprisoned Suffragette hunger-strikers.

On the to do list for January: thinking about automatic correction of OCR’d text, using titles of statutes as metadata, and hopefully some long overdue blog posts.

November Updates

Work on the Statutes Project in November 2016:

0: The big news is that the OCRing of the digitized volumes of statutes is now complete. That’s a total of 137 separate volumes. Quite how many words that is I haven’t checked yet, but the Danby Pickering series alone contains around 13 million words. There should be a more or less complete set of public acts from 1761 to 1875, 115 years worth of legislation in the volumes published contemporaneously. Before 1761, the statutes are incomplete as many acts that had either been repealed or had just expired were not included in the collections. The numbers missing are yet to be ascertained.

The raw OCR is available via github:

This stage complete, I now need to consider how best to correct the OCR and organize the texts. News on this next month.

1: Added the Riot Act of 1714 and the Cruelty to Animals Act 1876 to the collection of miscellaneous statutes.

October Updates

Work on the statutes project in October 2016:

0: Further OCRing of volumes of statutes. Current status is that the complete set of Danby Pickering’s Statutes At Large, from Magna Carta to 1806 have been put through the machine, as has what I’m calling the Butterworths series, spanning 1807 to 1839. Investigation of the avaiable digitized volumes of statutes suggest that there is a continuous series up until 1875, whereuon coverage gets very patchy. So for all intents and purpose, 1875 is the cut-off point for this project. Just another 35 years worth of statutes left to OCR!

1: Added two laws collected from around the internet:  The Witchcraft Act, 1735 and The Poor Law, 1601.

2: Added some volumes to the list of Scottish statutory resources, and started a page for Acts etc. for Burma / Myanmaar.

3: A visit to the V & A to see a perpetual motion machine that was not in motion. The promised post on the statute organizing its auction will be coming this month. Promise.

4:Discussion of getting the statutes on to Wikidata, with Andrew Gray.

5: What I’ve been reading (about law). Two short papers recommended by Law & History:

On the agenda for next month: the end, I hope, for the moment of the OCRing of the statutes, then some organizing and error correction of the resultant texts.

September Updates

A regular report of updates to the Statutes Project.

0: Launched this site.

1: Links to digitized volumes of the Laws of Grenada, 1763 to 1875, added.

2: Volumes 31 and 32 of Danby Pickering’s edition of the Statutes uploaded to the Github repository.

3: All 6 volumes of Pickering’s edition so far OCR’d run through Ted Underwood’s OCR Normalizer. The text is still poor, but nevertheless considerably improved over what came straight out of Abbyy Finereader. Again, all to be found on Github.

4: Began posting the specific volumes I am using as sources for British legislation. Note that the particular scans I am using as sources are important, due to individual blemishes and stamps on the original volume, and technical distortions – not to mention stray fingers – of the digitization used.

Planned for October: more OCRing, more normalizing, some thinking about the titles of the statutes, and a very odd lottery.

Introducing The Statutes Project

The aim of the Statutes project is quite simple: to put the majority of historic English legislation online in accessible, useful formats, readable by humans and machines alike, with accompanying metadata, without any financial, technical or legal obstacles to use or adaption.

The simplicity of this statement masks the many difficulties: finding the laws, digitizing them, turning page images into clean, correct text, and so on. And doing so  without having an entire life devoured by spell checking and hand correction.

The many volumes of statutes compiled through the last three centuries, coupled with mass digitization projects such as those run by Google Books and the Internet Archive, along with optical character recognition and text correction tools, does at least allow for the hope that useable – but not perfect – texts can be produced with a minimum of effort.

The focus will be on the late seventeenth, eighteenth and early nineteenth centuries, the ‘long eighteenth century’ that is central to my own historical studies. Expect a concentration on matters relating to debt and debtors; that is the subject of my PhD.

This blog is more a notebook than a full archive of legislation, although that is the long-term hope. It will cover the technical side more than the theoretical, although that won’t be absent. When there’s a sufficient corpus, quantatively and qualatively, there will be some preliminary attempts at analysis, little games aiming to investigate the possibilities.

Future posts will discuss the project in more detail, covering the source volumes, the software, textual analysis, dissemination, and undoubtably the many trials and tribulations produced by a simple idea rashly executed.

Launch Date, Redux

The Statutes Project launches properly on September 1st. Honest guv, not ‘aving you on or anything. Got lots of lovely OCRed statutes, just need a bit of tender care and they’ll be as right as rain.