Am 28.03.2011 23:07, schrieb Jonathan Rochkind:
> Instead, relevance ranking, in actual testing (in general, not sure
> about library domain specifically), does _very well_ at putting the
> documents most users most of the time will find most valuable first.
Definitely. And that's nothing new. The discussion about "best match"
search engines (with relevance ranking) vs. "exact match" (Boolean
retrieval, no ranking possible, because an item either matches or not,
1/0) is dating back to the 1970s. And some conclusions from that made it
even into libraryland as early as the 1980s (s. for example writings by
Charles Hildreth, one article from 1987 even being titled "Beyond
Boolean: ...").
And today, for example Google is making big money by delivering relevant
items (not only by ranking search results, but also by providing
"relevant" adverts), so there seems to be some (even financial) value in
that... Same applies to Amazon: They make a living from selling stuff
that people find(!) relevant enough to buy... That can't be all
crap(-shoot) they do, would be a risky business model...?!?
> It just doesn't do very well at providing an objective _measure_ of
> relevance, that can be compared accross searches or corpuses.
No, of course not. Solr (just like other text search engines) basically
uses a common information retrieval model, that is widely used in search
engines: the vector space model (which implements a "best match"
paradigm). The "relevance value" there is a measure for the similarity
of a query and an index document, calculated on term statistics (with
TF*IDF weighting, and, if you want, additional weights applied). The
values calculated always depend on term statistics in the query, the
result set and the index. So they can't be compared across searches or
beyond that.
That such a value based on term statistics in queries and index
documents doesn't magically meet users' information needs should be
obvious. That's just a stupid machine calculating values based on an
often very sparse user query (that doesn't necessarily express an
information need). But web savvy users nowadays are used to relevance
ranked search results, so they seem quite good at dealing with such
lists and finding out pretty quickly where the pertinent stuff ends (and
either find something useful or enhance the query).
More input from users beyond the entered query could be used to improve
the relevance calculation, to better meet users' needs. That can be
something simple as weighting results in certain languages higher based
on the language preference settings in the browser. Or taking the
browsing history, location and other personal and contextual data into
account (as web search engines do). In the end that gives personally
ranked search results. If we agree that information needs (and their
solution) is personal and based in a context, then personalized ranking
seems an apparent way to go (and then it's obvious that an "general
relevance value" for an item doesn't make sense at all)... Though, there
are valid remarks on that, eg, by Roy Tennant:
http://blog.libraryjournal.com/tennantdigitallibraries/2010/03/26/when-the-answer-you-get-is-not-the-answer-you-need/
Can we, instead if discussing the usefulness of relevance ranking over
and over again (for, it seems, at least about 25 years, I think, all has
been said), perhaps just start doing and improving it? I mean, we do in
some way, driven by products from vendors, but there are few places in
libraryland where real work on that is done. The conclusion from seeing
"strangely" sorted search results is not "relevance ranking doesn't
work", but "So, what can we do to improve it in our (special?) domain?"...
Just my 0.02 €,
Till
--
Till Kinstler
Verbundzentrale des Gemeinsamen Bibliotheksverbundes (VZG)
Platz der Göttinger Sieben 1, D 37073 Göttingen
kinstler@gbv.de, +49 (0) 551 39-13431, http://www.gbv.de
Received on Wed Mar 30 2011 - 11:35:16 EDT