I would like to search all of the ICZN opinions in https://www.biodiversitylibrary.org/bibliography/51603 for the term ‘genotype’. However, I can’t do that automatically as far as I know without going through each volume and searching through the OCR’ed text. I can download all OCR for all pages in each volume, but that’s computationally a bit intensive, as each request is a separate API call and I don’t want to flood the system.
Is there some other way to search all volumes in the same publication for a term?
1 Like
@RichardLitt You can do it manually 
using “Search inside” on each volume.
If you don’t want to do that manually (and who would?), there is the API, see docs https://www.biodiversitylibrary.org/docs/api3/PageSearch.html
The URl below will search for “holotype” in all pages in item 333567 once you add your own API key.
https://www.biodiversitylibrary.org/api3?op=PageSearch&itemid=333567&text=holotype&format=json&apikey=
1 Like
I know of no way to search all volumes at once**, but Rod pointed to searching one volume. Searching 70+ volumes would be time consuming.
To search all, you need to get creative.
The GetTitleMetadata API call with items=t will return get the list of items. Using the ItemID from those results, you can either
- Use the
GetItemMetadata API call with ocr=t
- Craft a URL such as
/itemtext/43924
Either get you the text for the entire volume. It’s not broken out by page, but it’ll get you close.
Just for fun, I tried to do this on the Linux command line using the latter method:
wget -O - 'https://www.biodiversitylibrary.org/api3?op=GetTitleMetadata&id=51603&items=t&format=json&apikey=YOURAPIKEY' | \
jq '.Result[0].Items[].ItemID' | \
sed 's|^|https://biodiversitylibrary.org/itemtext/|' | \
xargs -L 1 wget -nv
This will create files named 43924, 43930 and so on. Those are the same ItemIDs used on the URL: https://www.biodiversitylibrary.org/item/43924
That was fun. Enjoy!
** – I stand corrected. See Colleen’s response below.
2 Likes
Depending on what you’re ultimately trying to do with the output of this search, you can also use Advanced Search to search across all volumes of the publication (Title: Bulletin of zoological nomenclature [use exact phrase]) and for the word genotype in the OCR (Text: genotype). It brings back all or nearly all the volumes (71). The search results include Details where you can quickly scan the placement of the word genotype within the full text, though it won’t link you to each page. If that helps you narrow down the volumes you want search in, you can then Search Inside specific volumes to get the page number for each instance of “genotype”. https://www.biodiversitylibrary.org/search?SearchTerm=Bulletin+of+zoological+nomenclature&tinc=P&lname=&yr=&subj=&lang=&col=&nt=&ntinc=A&txt=genotype&txinc=A&SearchCat=T&stype=C&return=ADV#/titles
5 Likes
This thread inspired me to throw together a simple web app for this specific use case: https://bhl-title-search.dev4pgh.com/
You can find a BHL publication, select it, and then search across all of its volumes/items, with links back to the matching pages in BHL. Since the question brought up the concern of flooding the system I tried to be careful about the frequency of API calls: the app searches items one at a time, adds a short pause between BHL requests, and obeys `Retry-After` if it receives a rate-limit response.
The code is here: GitHub - dev4pgh/bhl-title-search · GitHub
A few caveats: this was built quickly, with some AI help, so please treat it as an experiment rather than polished software. It may be slow for publications with lots of volumes, and the UI/code could definitely be improved. But it seemed like an interesting thing to try based on the question and replies here.
For context, I have been working on small tools that provide new ways to access existing data, especially around natural history/sciences, and BHL is the most excellent source for these sorts of things. This one is more of a test than anything, so I cannot promise any particular future development, but I’m happy to hear if people find it useful or have suggestions. I’ll keep it on my server for now in case anyone does find it useful.
2 Likes
Thank you all for all of these responses! This is really useful, and what I was looking for (not including your manual suggestion, @rdmpage D:).
As a side question - how big is the BHL archive, currently? Would it be possible to search the text of all publications using the API?
Well, in principle yes (that’s what the search does), but full text search is going to be offline for a bit while the servers move (see BHL server move update: biggest milestone underway ). I gather that @cajunjoel thinks things should be back to normal by July 1.
Note that full text search in BHL has an interesting wrinkle. https://www.biodiversitylibrary.org/docs/api3/PublicationSearch.html treats the OCR text from an entire item as the unit of search, which means that it is possible that the best match to your search may not actually be the top hit in BHL. This counter intuitive result can happen if you are searching for a specific string of text, such as a phrase (or, say a specimen citation or the title of a paper) and the words in that phrase occur multiple times through one item, but only occur together on one or a couple of pages in another item.
When search is back online try searching for Observations of fossil crabs from Tertiary deposits in Sind and Kutch and you’ll see what I mean. The item that has that actual string is the fourth hit in the search results (!)
This is a function of the way the Elasticsearch index has been configured in BHL. IMHO it’s not ideal.
