On Jul 1, 2026, at 9:37 AM, Amy Schuler <000000088c12581f-dmarc-request_at_LISTS.CLIR.ORG> wrote:
> Thanks to Eric and Jason for the excellent advice on these two APIs. The C4L list members never disappoint!
>
> --
> Amy C. Schuler (she/her)
> Director, Information Services & Library
> Cary Institute of Ecosystem Studies | 2801 Sharon Turnpike | Millbrook, NY
I have packaged up my Web of Science tool. [1] From the README file:
TL;DNR ./bin/search2reader.sh big-science 'AB="big science"'
Introduction
------------
Given a valid Web Of Science query and an ouput directory,
the files in this directory will:
1. Search WOS and cache the result as a collection of JSON files
2. Extract DOIs from the JSON files and cache the result
3. Loop through the DOIs to cache OpenAlex Works files
4. Read the Works files to create a set of abstracts and
an associated metadata file
This is quick and easy way to search and download hundreds
(if not thousands) of citations from Web of Science. Given
enough abstracts, the student, researcher, or scholar can do
a significant amount of reading.
Actually, the files in this directory do additional things,
but they surround the use of a tool of my own design -- the
Distant Reader. For example, there is fledgling work to
transform the search results into a Zotero bibliography. But
what people really care about, at least for right now, is
the searching of Web of Science and enhancing the search
results.
Doing the Work
--------------
1. You will need to know your Web of Science Starter key,
and you will have to save it to your environment with the
variable named WOSSTARTERKEY. Similarly you will need an
OpenAlex API key and save it to your environment with the
variable named OPENALEXKEY. For example:
export WOSSTARTERKEY=foobarbazfoobarbazfoobarbazfoobarbaz
export OPENALEXKEY=xyzzyxyzzyxyzzyxyzzyxyzzyxyzzyxyzzyxyzzy
2. Use the Web-based interface to Web of Science to
articulate a query. This is difficult, and in the end your
query might look like as follows, which searches abstracts
for the phrase "big science":
AB="big science"
3. Use the following command to search Web of Science and
cache the results to a set of JSON files. You will need to
install the Web of Science Python module
(clarivate.wos_starter):
./bin/search-wos.sh ./corpora/big-science/batches 'AB="big science"'
4. Next, extract all of the DOIs from the cached JSON, like this:
./bin/json2dois.py ./corpora/big-science/batches > ./corpora/big-science/dois.csv
5. Loop through the DOIs to get OpenAlex Works files, and
the results are more JSON files containing all sorts of
bibliographic goodness:
./bin/dois2works.py ./corpora/big-science/dois.csv ./corpora/big-science/works
7. Finally, loop through the Works files to create a set
of abstracts and an associated metdata file
(metadata.csv):
./bin/works2reader.py ./corpora/big-science/works/ ./corpora/big-science/reader/
You can run all of these commands in one go, like this:
./bin/search2reader.sh big-science 'AB="big science"'
Next Steps
----------
If all goes well, you ought to have accumulated all sort of
bibliographic goodies. Authors. Titles. Dates. Citations.
DOIs. Controlled vocabuarly terms. Abstracts. And citation
measures, I believe.
Realistically, scientists do not read the actual journal
articles but only the abstracts. That probably being the
case the content of the finally created directory (reader)
may be very useful.
On the other hand, if you want the actual articles, then you
can feed the URLs to the PDF files (as found in the
metadata.csv file) to you favorite harvesting program, but
alas, so many of the URLs are now behind HTTP challenges.
Sigh.
Lastly, I have used this suite to search Web of Science for
just about anything, and I download hundreds (if not
thousands) of citations. Most of the DOIs can be found on
OpenAlex, and thus I can get the abstracts. Given the
authors, titles, dates, and abstracts, I can use my Distant
Reader Toolbox to create a data set from the whole. I can
then use the Toolbox's API to analyze the data set to
address all sorts of interesting questions. Moreover, I can
vectorize the sentences of each abstract and apply
retreival-augmented generation (RAG) to it. I can also put
the data set behind an MCP server and ask questions of the
data set such as: What is "big science"?, What are the names
of the people who played a significant role in defining "big
science"?, or How has the definition of "big science"
changed over time?
I think this is really a whole lot of fun, but I must admit,
your milage will vary.
Fun in Library Land!
[1] tool - https://github.com/ericleasemorgan/wos-helper
--
Eric Lease Morgan, Librarian Emeritus
University of Notre Dame
Received on Wed Jul 08 2026 - 10:20:02 EDT