Andrew,
Thank you for this response. In my opinion one of the best things that Vufind is doing for the library community is in presenting an open source option where libraries have a fairly low barrier to experimentation and testing on a real "next generation opac." The ability to experiment with Vufind, and to really engage in how relavance ranking, search algorithms and facets work, makes the theortical discussions about OPACs much more meaningful and helpful to librarians. I should point out that some of the points you have raised are directly related to some of decisions made at Yale in our VuFind implementation. This is especially true for spell checking. You are also correct that we were not using the most current relevancy ranking available for VuFind. I am eager to test again with the more refined algorithm, and I expect that this should improve results.
In this testing we intentionally did not lead participants to functionality such as Browse and Advanced search but we let them decide how to search. We could look at tests designed to measure that functionality. While most users seem to gravitate to the basic search box, we have found that Advanced Searches comprise about 12% of all Yale catalog searches, and so it would be appropriate to include a question on Advanced Search.
When you say "I have been thinking about changing the facet results to only include the results from the AND'd part of the query to increase the accuracy." I too had thought in looking at results that this would perhaps be a useful way to explore creating better topic facets. Part of the problem with topic facets was that results were so broad that the faceted headings just didn't seem to make much sense. I would encourage that type of experimentation. Participants seemed perplexed by topic facets, but they recognized them as potentially useful. Another problem that I see with topic facets is that patrons are now very accustomed to using the catalog as a known item locator, and not a means of discovery. Exploring through topics is not second nature to most patrons now.
Katie
-----Original Message-----
From: Next generation catalogs for libraries [mailto:NGC4LIB_at_LISTSERV.ND.EDU] On Behalf Of Andrew Nagy
Sent: Wednesday, May 28, 2008 4:19 PM
To: NGC4LIB_at_LISTSERV.ND.EDU
Subject: Re: [NGC4LIB] usability tests on Vufind
Katie - This is quite impressive and I appreciate you sharing the results of your study with this community. As the Lead Developer of VuFind I wanted to share some of my thoughts.
Let me start by saying that our goals of VuFind are to help solve the problems that you state in your document - library patrons, and even librarians, have become so comfortable with websites such as Amazon, Google Scholar, etc and the rich features that they offer that they have begun to use that as their discovery tool. The OPAC has become a tool to determine if the library has the item they are looking for and whether or not it is available - not a search & discovery platform. We are trying to bring the user habits back to the library web presence to better steer our users in the right direction with VuFind.
I think your study has been conducted well and so I just wanted to let you as well as the community at large know where we are heading with VuFind.
Your study was conducted with version 0.8 it appears by your document. Since 0.8 we have heavily modified the query parser as well as the relevancy system and are still working on it today. We hope that with version 0.8.3 we have a strong relevancy system and with the upcoming 1.0 release we will have an even stronger relevancy system.
We hope to integrate federated searching this summer into VuFind to allow article searching within the same interface. We are leaning heavily towards the Pazpar2 engine from indexdata. This engine has proven to be fast and highly compatible and even works with Web Feat translators through the OpenTranslators project provided by CARE Affiliates that will even more highly increase target availability.
Our *current* search model is very similar to other common search engines. VuFind weighs AND'd search results higher than an OR'd search - however it does combine both into 1 query which can augment the quality of the facets since it includes the results from the OR's part of the query. I have been thinking about changing the facet results to only include the results from the AND'd part of the query to increase the accuracy. The problem this presents is an mismatch between the facets and the search results. However, the average user will probably never notice the difference. With version 0.8 - it only used OR'd searches and never took into consideration AND'd searches resulting in very broad results.
It also appears that your search test did not include the use of the advanced search page - I think this would be interesting to see how your results changed with the use of this tool.
It would also be nice to see a study on the use of the Browse tool that allows a patron to search the catalog without ever typing in a search query.
VuFind has been compatible with Zotero for a while now and has had the code for EndNote exporting. The EndNote export feature was missing from the User Interface - but will be present in the upcoming 1.0 release.
You also mention a bit about the spelling correction. This has been a thorn in the side for VuFind since it uses a traditional dictionary to do term comparison and has not been very accurate. We will be scraping this model and replacing it with an index based spelling correction algorithm which will be much more accurate. It will also help with spelling of terms that are not in a traditional dictionary since it will be based off of the terms in the index within VuFind. I think this will work much better. This is slated to be included in the upcoming 1.0 release.
I think your points about the use of facets and especially around the topic facet would be an interesting conversation if you would like to discuss your thoughts about this more in depth on this listserv. Facets have always been a big "buzzword" but no one has really discussed their usefulness. We could also discuss feature enhancements with the VuFind community on the vufind-general mailing list. Feel free to join and discuss at vufind-general_at_lists.sourceforge.net
VuFind is an open-source project and can only get better through community involvement - I challenge all who are interested to think about ways to enhance VuFind so we can all benefit from the great ideas.
Andrew Nagy
> -----Original Message-----
> From: Next generation catalogs for libraries
> [mailto:NGC4LIB_at_LISTSERV.ND.EDU] On Behalf Of Bauer, Kathleen
> Sent: Wednesday, May 28, 2008 10:53 AM
> To: NGC4LIB_at_LISTSERV.ND.EDU
> Subject: [NGC4LIB] usability tests on Vufind
>
> Hello,
>
> Of possible interest to those who may be contemplating doing usability
> testing on their OPAC, Yale recently conducted two tests on pilot
> VuFind installations at Yale. One study looked at a subject based
> presentation of ebooks for the Cushing/Whitney Medical Library
> http://www.library.yale.edu/libepub/usability/studies/summary_medical.d
> oc
>
> and the other looked at a pilot test of Vufind with a sample of 400,000
> records drawn from the Library's Voyager system
> http://www.library.yale.edu/libepub/usability/studies/summary_undergrad
> uate.doc
>
> Test questions were drawn from user search logs in the current library
> system, and some were designed to test for those problems that the logs
> have demonstrated exist for patrons, such as incorrect spellings, and
> incomplete title information. In reading the reports please be aware
> that some of the problems uncovered may have had a lot to do with
> peculiarities of the Yale implementation, such as the sample of records
> imported into VuFind for this test, and less to do with VuFind itself.
>
> In general participants were intrigued by the possiblities offered by
> facets, although the topic facets in particular did not always seem to
> function as they expected or desired. The most desired feature
> participants wanted to see developed was an easy direct export from the
> catalog to a bibliographic citation management tool such as RefWorks
> and Endnote (while other catalogs may have this feature already, the
> current Voyager system at Yale does not a direct export feature.)
>
> Katie Bauer
>
>
> Katheen Bauer
> Director of Usability and Assessment
> Yale University Library
> 130 Wall Street
> P.O. Box 208240
> New Haven, CT 06520-8240
>
> 203-248-3359
> kathleen.bauer_at_yale.edu
Received on Wed May 28 2008 - 15:28:28 EDT