On Mar 20, 2020, at 11:40 AM, Eric Lease Morgan <emorgan_at_nd.edu> wrote:
> https://carrels.distantreader.org/library/covid-19/
> https://carrels.distantreader.org/covid.cgi
Wanted: experts in the creation and deployment of Solr indexes and experts in relational database technology
The sponsors of the Distant Reader, the good folks at XSEDE, are very strongly encouraging me to submit a fast-track proposal to: 1) increase the size of the Distant Reader's underlying high performance computing system, and 2) apply the technology to the growing Coronavirus literature. To that end, I have outlined a set of tasks/functions, attached.
But the work requires deeper knowledge of Solr and relational databases than what I possess. For example, we will be wanting to create a Solr instance to index 10s of thousands of scholarly journal articles, if not just as many sets of non-scholarly materials. Similarly, we will be wanting to analyze the literature in terms of its ngrams, parts-of-speech, named entities, and grammars. These things can/will be saved in both tabular forms as well as a relational database. Much of this work has already been done, but it needs to scale up a few notches. We need your help.
If you would like to apply your skills in a high performance computing environment, then you might want to participate in a Zoom meeting scheduled for Friday at 9 AM (EDT). Drop me a line to ask questions and/or "raise your hand" if you would like to participate in the meeting.
Be safe.
[1] XSEDE - https://www.xsede.org
--
Eric Lease Morgan
University of Notre Dame
The Distant Reader Meets Covid-19
Below is a list of tasks/functions which we can implement as a part of a proposal. Many of these tasks/functions may be implemented concurrently, and they are barely prioritized:
1. Dedicate a single Reader node to serving a full text index of
virus-related materials, and these materials will initially focus
on the CORD-19 journal literature dataset. This machine would do
very nicely at only two or four cores.
2. Dedicate a few nodes to harvesting and indexing the data for
Item #1. Indexing is a bit computing heavy.
3. If Items #1 and #2 are successful, then index additional, but
more difficult to acquire, journal literature, like content from
JSTOR or Zotero libraries
4. Identify the likes of Team JAMS (the good students who won the
PEARC hack-a-thon), and have these people use the Reader's output
(ngrams, parts-of-speech, grammars, named-entities, etc) as the
input for things like discovering relationships between drugs,
"correlations of language" between articles, or visualizations of
the underlying data such timelines, geographic maps, or network
diagrams.
5. Modify the Reader's code to use a biomedical language model
instead of the existing English language model.
6. Modify the Reader's code so the feature-extraction tasks are
more distinct from the report generation tasks, thus we can
divide and conquer when it comes to report generation.
7. Work on a subset of the CORD dataset, and get the subset
working
8. If Items #5, #6, and #7 are successful, then increase the
carrel's content to include all the CORD content.
9. If Item #8 is successful, then include content from Item #3
10. Implement a better, more interactive topic modeling
interface. Just as everybody likes to search, topic modeling is
very popular in text studies.
11. Integrate the full text indexing and the topic modeling
interface into the Reader's study carrel thus creating a coherent
whole. For example: search index, create subset, and topic model
it. For example: peruse study carrel, identify thing of interest,
link it to full text index or topic model, and return thing of
interest in the context of the original article. Etc.
What might be needed to do this work? Some of it might include:
1. A re-allocation of existing cores, thus some systems
administration
2. A re-examination of the shared file system because my
antidotal observations see a lot of time is spent on disk I/O
3. Hacker(s) who can read delimited files over the 'Net (or a
relational database file), parse it, ask questions of it, and
visualize the results.
4. Content experts who can evaluate the output of everything above.
5. Time.
Here is a list of nice-to-have items -- people:
1. Someone who really knows Solr -- the full text indexer of
choice
2. Someone who really knows relational databases -- because a
whole lot of the data is ultimately stored in one
3. Someone who can write interactive Web-pages to... interact
with the underlying data in real time.
4. People to create additional study carrels of related content,
and then we can meld the resulting carrels togehter.
The above is only a set of suggestions. I hope they give you ideas, and I hope you are available to chat on Friday at 9 o'clock.
In any event, additional suggestions and reactions are welcome.
--
Eric Morgan and Team Reader
March 23, 2020
Received on Tue Mar 24 2020 - 13:18:09 EDT