search/retrieve via url (sru)

I created a Search/Retrieve via URL (SRU) interface that I'd like to share.

As you may or may not know, SRU is a Web-based protocol designed to query bibliographic indexes, like library catalogs. [1] Over the past few years I have cached about 1.4 million open access articles and etexts. I used MARC to describe the cached items, and then I indexed and provided access to them via Koha. My Koha interface is simplistic. [2] I then turned on Koha's out-of-the-box SRU interface, and I had the interface's output enhanced with facets. The root of the interface is here: [3]

I then spent time writing various SRU clients. One was a "thin" client, which eliminated my need to know SRU (or more specifically Context Query Language). It worked pretty well, but since search results included bunches o' links, and since Internet spiders found it, the spiders fell into infinite loops. I took down the thin client until further notice.

Somewhere along the line I found a command-line client I wrote, which works pretty well too. [4] Given a query and an output directory, the client searches the catalog and downloads the full text of found items. Example usage includes:

  $ ./reader-sru-retrieve.py love ./love
  $ ./reader-sru-retrieve.py dc.author=plato ./plato
  $ ./reader-sru-retrieve.py '(dc.title=truth and dc.title=beauty)' ./truth-and-beauty
  $ ./reader-sru-retrieve.py '(dc.subject=covid and koha.ccode=cord)' ./covid

Making it easy to download found items is very important to me. I think we -- us librarians -- ought to make it trivial to download the content from our collections. Click, save. Click, save. Click, save...  Such is not easy nor scalable.

I am currently working on an SRU via Model Context Protocol (MCP) interface, and it works even better, but the distribution s&cks. [5] More specifically, the MCP interface takes natural language commands as input, uses a large-language model (LLM) to transform the command into CQL, submits the query, gets back the results, the LLM interprets the results, and human language is returned to me. This eliminates my need to know what fields the SRU interface supports as well as eliminates my need to formulate complex Boolean queries. More importantly -- very importantly -- the LLM analyzes the facets as well as the bibliographic results to create meaningful interpretations. Truly scalable, useful, and insightful. See, for example: [6]

Fun with SRU.


Links

[1] SRU - https://www.loc.gov/standards/sru/

[2] Koha interface -   http://catalog.distantreader.org

[3] SRU interface - http://catalog.distantreader.org:2100/biblios

[4] SRU client - https://distantreader.org/tmp/reader-sru-retrieve.py

[5] SRU via MCP - https://github.com/ericleasemorgan/reader-sru

[6] MCP output - https://distantreader.org/tmp/sru-results.htm

--
Eric Morgan <eric_morgan_at_infomotions.com>