So, I'm very interested in this topic and think BibTip is an
interesting test. That said let me disagree with most of your email.
> In fact, your "ant navigation" analogy is a faulty one in this case. BibTip works astoundingly well, and it is not because it simply follows "where users go". Instead, BibTip uses "Repeat Buying Theory" as a framework to statistically analyze user search behavior. Repeat Buying Theory is a highly successful and well-tested statistical framework to describe the regularity of repeat-buying behavior of consumers within a distinct period of time.
So, I don't want to get snippy, but I am pretty familiar with the
statistical problems of this approach and of others, and have done
extensive work—and with appropriately large datasets—on LibraryThing.
Pardon the contradiction, but nothing about my description of the "ant
tracking" problem was faulty, so let me explain it again.
> The developers of BibTip at Karlsruhe University very skillfully adapted this theory to the session-based search behavior of library OPAC users. They key is that BibTip only records the inspection of the full details of an individual bib record selected from a larger list of search results. It does not "follow" the user.
I understand it doesn't "follow the user" on two legs, but it records
what books discrete users visit and then makes statistical inferences
from it. This amounts to a picture of where are and where they go,
albeit without the order-of-events data which, actually, would improve
it.
> In this framework, clicking-on and reading the full details of a given record is an economic choice. The choice of one record over all of the others in a given list is viewed as an economic choice, very similar to individual's choice to purchase one thing over another during a given trip to the store. There is a real cost in time (e.g. an economic cost) for the user each time he/she selects and views a record. It can be assumed that the "search cost" to a user is high enough that he/she is willing only to view the details of a record which is truly of interest. Users, in effect, are self-selecting. That is, users with common interests will select the same documents, and, since recommendations are only provided to users from the full details view, we can surmise that recommendations are only offered to interested users.
All this is obvious. But systems built on statistics have flaws. One
of them is reinforcement—the ant problem. The more you recommend
something the more people will follow your recommendations and the
more co-occurrences there will be. Success breeds success. The ant
trail, once started, has a tendency to get stronger. This is true in
any recommendation system, unless you adjust for it explicitly—which
has statistical problems too.
At base, the quality of the recommendations from a screen-watching
model are related to three factors: (1) how good is the searching? and
(2) how costly is failure?
Take Amazon purchases. Finding a book on Amazon is easy, and the cost
is high. So there are few mistakes—people tend to buy the books they
want, so the signal is strong. The main problems relate to agency,
people buying books for other people. That's hard to correct for, so
you just hope that you can discern signal from noise when the quantity
of data is great.
Unfortunately, the OPAC is not Amazon. Leaving aside scale—WorldCat
receives 0.7% of the hits Amazon does!—there's the issue of search
quality. Bad OPAC search means that people spend a lot of time on
detail pages they weren't aiming for. Search for "Da Vinci Code" at
SPL, for example,and you get a page full of results without an actual
paper copy of the Da Vinci Code. Results matter. People hate reading
results, they hate revising searches and they hate looking at
subsequent search pages. Many would rather dive into a record quickly
and back out. They're rather dive in and see if they can leverage
partial success. Personally, I've learned to click on some version I
don't want—the Spanish or the eBook—knowing there's an author link
there I can leverage to get to the paper version I really want. I
suspect I am not alone. And each time I do it, I create noisy data for
a recommendation system.
> In order to build relationships among given documents, BibTip analyzes record pairs.
That's the beginning of a good system. Among various improvements,
Amazon keeps track of order, because order matters. If 50 people who
look at the Spanish-language Harry Potter also look at the English,
that's interesting. That 49 of 50 went from the Spanish to the
English, that's more interesting. It suggests the Spanish should
recommend the English more highly than the reverse.
> For each record X that has been viewed in the full details view of the OPAC, a "purchase history" is built. This is simply a list of all of the sessions in which record X has been viewed. Record X is then compared with all other records (Y) which have been viewed in the same session as X. For each pair of records (X,Y) that have been viewed in the same session, a second purchase history is built. The number of users who have viewed record X and another record Y in the same session is statistically analyzed and the probability of a "co-inspection" of records X and Y in a given session is calculated. A recommendation for record X (That is, users who liked X also liked.) is created when record Y has been viewed more often in the same session that can be expected from random selections.
I do hope they have a threshold—that a single incidence of
co-occurrence will never trigger a suggestion. Otherwise it's a
privacy problem waiting to happen. John Blyberg discussed this problem
when SOPAC was released.
> This "repeat buying theory" is remarkably good at automatically determining relevant recommendations for a given item. It takes some time for enough data to be collected so that good recommendations are available for a substantial part of a collection, but what is the hurry?
> Of course, the longer you have the algorithm running, the better your recommendations become. The more users you have, the better your recommendations become. But, time is on our side in this case ;-)
Well, except that you also need to expire data, or weight it less over
time. That, ten years ago people were examining the Bible and the
Bible Code is not an accurate predictor of what Bible readers want
today.
> BibTip is a signal example of harnessing collective intelligence to serve the needs of the library.
I agree very much.
> Frustratingly, for all the talk here and elsewhere of the features of next generation catalogs, I rarely find anything that convinces me that librarians understand that collecting/harvesting and re-using user (and usage) data is the key to most (if not all) of the services we want these new catalogs to provide. Without seriously thinking about the implications of harnessing collective intelligence - and taking steps *now* to build systems that do - we are not going to get very far. BibTip as a service is a big step in the right direction.
Absolutely. I agree. I agree completely. I am a fan of the idea.
Nobody should take my words as a wet blanket on the fires of
experimentation. But moving past my desire to promote interesting
things and into analysis and experience with the topic, I am skeptical
on two fronts:
1. There are real privacy implications to collecting user data. I
think they can be solved, but they cannot be dismissed. And solving
them hurts your data quality/quantity.
2. I am skeptical that libraries can accumulate enough high-quality
data to compete against other systems.
For curiosity's sake, I am tempted to try the idea, leveraging the
LibraryThing for Libraries traffic. But, for the reasons above and
having experimented a lot around with recommendation data from
different sources and of different qualities, I'm very skeptical that
OPAC-based path-watching will ever be a significant source of
recommendations. But you may label me an interested party—we sell
recommendation data to libraries.
Tim
Received on Fri May 16 2008 - 18:44:46 EDT