On Jul 4, 2026, at 7:44 PM, Karen Coyle <lists_at_kcoyle.net> wrote:
> I would feel better about this if these results didn't sound like the platitudes of marketing speak. ("Collaborative refinement of library services" is something I don't think any of us would say about C4L.) Where did the model get this tripe?
TL;DNR - I advocate the use of LLM technology in libraries. It can be a supplement to our existing processes, not a replacement.
Thank you for the reply because I really feel our community good benefit from discussion on these topics.
Tripe? I had to look up the definition of that word. Where did the result get such a response? Many places, but one of the more significant is my locally configured "system prompt". What's that? A system prompt is an extra little bit, behind the scenes configuration sent to a large-language model (LLM). My current system prompt follows:
Return results as if they were written by a student
attending a liberal arts college. Ask questions, sometimes,
but not always. The model is working within a generative-AI
system called a RAG, and therefore results are intended to
be primarily drawn from the underlying MCP system; results
drawn from outside the system are to be kept to a bare
minimum. The model is intended to be used as analysis tool
not an oracle. Do not voice results in the first person!!!
When citing sentences, include item and index values.
If my system prompt said something like "Voice replies as if written by an eighth grader", then the results would be expressed differently. Differences in system prompts make a significant difference in results. You should see the sort of things I get back when I specify second graders or erudite college professors. :-D
> But what really concerns me is that the system is a black box (others have noted this), which means that there is no way to evaluate the result other than ones' gut feeling that it's "right." Your gut feeling returns "true and accurate" while mine concludes that honestly some of the "facts" in the statements below could be wrong. For all that C4L has great discussions, I don't see "code sharing" as taking place often in the body of the emails. (Maybe a snippet or two, but not as stated here.) The last paragraph on MARC doesn't convince me much. The phrase "annoying data format" comes up only once when I search on the C4L archive, and it's in one of your posts, Eric. I wouldn't include this in a summary of C4L list users' statements on MARC even though we are pretty critical of it. I also have doubts that one can conclude from the list that the group is "focusing heavily on MARC records—the standard format for library catalog data." There is a fair amount of discussion about MARC but have posters actually said here that it's the standard format for library catalog data? (We probably assume that everyone here knows that.) Could the software have gotten that from elsewhere, given that there is a fair amount of documentation online?
>
> --
> Karen Coyle
> kcoyle@kcoyle.net http://kcoyle.net
Granted, above is more difficult; I more or less (mostly more) agree, but...
The box is more gray than black. The whole process is rooted in the computation of geometric distances between words mapped in a VERY large n-dimentional space. To elaborate, it works something like this. A HUGE pile of text is accumulated. The text is parsed into tokens (think "words") to create a vocabulary. This results in a matrix where each row (millions or billions of them) is a document, and each column is a token from the vocabulary. At the intersection of each row & column is a measurement, and the measurement might be the number of times the given token is found in the given row. This results an a GREAT BIG set of vectors "pointing" to locations in the space. Given some input (at least a word or better yet a large set of words), the input is vectorized in the same manner as the original LLM. The vector is compared to all of the other vectors in the set to identify most similar vectors, where similarity is denoted to something like cosine distance. This is a process of linear algebra, and it is a kind of find or search process. Works in the manner similar to auto-completion or auto-correct but on a REALLY big scale.
For example, in English, the word "the" is very frequently followed by an adjective but ultimately by a noun. These patterns are manifested in the LLM.
Using the sort of process outlined above, given words are mapped with similar words, and the similar word are output. It does not necessarily identify nor extract exact phrases from text unless specifically asked to do so. Also, in my example, I only read one month's of Code4Lib archives, and in that month there may have been more discussion about MARC than not. In any event, the process does work (for the most part), and it is a real-world application of something first articulated by a man named John Firth around 1957 who said, "You shall know a word by the company it keeps" -- context.
It is a black box? Yes, mostly. It black in the same way Google's search algorithms are black. It is back in the same way our bibliographic indexes rank relevance. It is as black as relational database implementations perform join queries through many-to-many relationships.
Very important: I do not advocate the use of LLMs sans context. In other words, I do not advocate asking very general questions, like "What is the best Shakespeare play?" to LLMs because the only context included in the response is what is in the model. On the other hand, I very much advocate the use of retrieval-augmented generation (RAG) and/or model context protocol (MCP) servers. These tools get input from known and verifiable collections of text. The results of RAG and/or MCP queries are sent to an LLM for interpretation, and well-implemented results will include ways to backtrack results to the source. I think the use of RAG and/or MCP technologies applied to library collection can be very useful. LLM are not panaceas though.
For example, the other day I queried various bibliographic indexes for the phrase 'big science'. This resulted in 1,200 abstracts from scholarly journals for a total of 350,000 words (which is bigger than Moby Dick). I then RAG-ed against it to extract definitions of 'big science', learned who helped define it, and how that definition changed over time. It worked. It worked well. It saved me a whole lot of time. It supplemented my reading process, not replaced it.
I have learned two additional things. First and foremost, I take the results as plausible, not truth. Thus, I like to believe I practice information literacy along the way. Second, I am always dubious of the adjectives returned by LLM, especially the superlatives.
What I would really like to see is the creation of one or more LLM built by the library profession. This way we would know whence the model came, how it was implemented, and remove the blackness. Such an effort would be akin to collaboration we have seen in the past when it comes to collection building or metadata sharing. Yes, it would be very expensive, but if we were to pool our resources, then I think it could be done.
Lastly, those were a lot of words. Thank you for listening. I hope the discussion continues.
--
Eric Lease Morgan, Librarian Emeritus
University of Notre Dame
Received on Wed Jul 08 2026 - 11:36:28 EDT