AI is changing the way we do research

One the more interesting real stories in The Innocents Abroad was Mark Twain’s interaction with an American colony in Jaffa. Led by a charasmatic preacher named George Washington Joshua Adams (born George Jones Adams), over 120 followers from his church in Maine moved to Jaffa with a shipload of building materials, and built wooden homes in Jaffa. By the time Twain met them, roughly a year after their arrival, they were destitute and dying, with no way to return home.

The ship Twain was traveling on, the Quaker City, took over 30 colony members from Jaffa to Alexandria, Egypt where it was easier to find passage back to the United States. In our edition of the book, the story takes up only two pages, and includes four added footnotes. The footnotes give added background to the story (explaining that 17 colonists had already died, and 54 had already returned to the US). Twain doesn’t list the names of those they helped in the book, but he did list them in one of his letters to the New York Tribune. Those names are also reproduced in a footnote in our book.

The book has only two pages related to the Adams Colony, with four added footnotes. The letter Twain wrote was significantly longer than what he shared in the book. The story of the colony was covered in American newspapers at the time, but it’s not a particularly well-known story today. An article was written about it in 1964 (Prophet in Zion: The Saga of George J. Adams in The New England Quarterly). A single book about the colony was published my Dr. Reed M. Holmes, who moved to Israel and helped save one of the remaining houses from the Adams Colony (The ForeRunners on Amazon). Other than those, there’s not a lot out there.

Which brings us to the topic of this article, which is actually about Artificial Intelligence and its place in the world of research. When researching the book, I had access to scans of bound dispatches from the US Consul in Jerusalem from 1856-1906. Those dispatch books were photographed and put on microfilm by the US National Archives in 1969. I ordered scans over a decade ago for a completely different project that I never got to complete. When working on the book, I knew it was likely that the Adams Colony was mentioned in the dispatches, but they were hard to read (hand-written over 150 years ago, microfilmed over 50 years ago, and scanned over a decade ago with not great levels of contrast). I skimmed the documents, but for a section that only took up two pages in the book, it wasn’t worth the effort it would have taken to find relevant documents. So the scans continued to sit on my computer. That is, they sat there until I was trying to clear unused files from my computer to make more room. The folder holding the US Consul scans was almost 8GB. As I was trying to figure out where to move that directory, it occured to me that maybe I could use AI to make these previously hard-to-access documents more accessible.

Within a week, I had over 1500 documents extracted from the microfilm scans, all fully transcribed. For those documents not in English, they were translated. The process was not easy, and the transcriptions are bound to have mistakes, but there’s no question that these previously difficult-to-access records are now very accessible. Currently 1520 documents have been transcribed, each set up with its own page on a web site. From those documents, over 1900 people have been identified. There are many lists of people in the documents, including the list of 150+ followers of George J. Adams that arrived with him on the Nelly Chapin ship from Maine. The documents are fully searchable, and each document page allows users to comment on the document, including suggesting corrections to the transcriptions if they see a problem.

One fun way to see what topics are discussed, and which people are mentioned, is to look at the Word Clouds for the site. One thing you might notice if you look at the word clouds, is that the largest phrase in the Keywords cloud is Adams Colony. Clicking on the key phrase brings up a list of all the documents connected to the Adams colony, and there are 96 of them. This opens up an amazing resource for those researching the region, and certainly for anyone researching the Adams colony.

Take a look yourself at consul.quintbooks.com to see the full archive and browse the documents.