> It doesn't let us evaluate potential changes until we
> deploy them over a large community in production
> and measure how it works.
Obviously, we need methodologies, like usability testing, for figuring out what is wrong with our current systems and how we might go about improving them.
But in usability testing, you're usually measuring how successful participants are in completing specific tasks. Can the user get to actual books and articles on a given topic, say. You're not simply asking people how they "feel" about the search results.
So whether you're evaluating an existing system or conducting a usability study on a new prototype, the _metric_ is still the same, even if the scales are different. At the end of the day, you're measuring how well each system gets users to content.
> 'More' is a relative word. More than... what?
More than with a previous system. Or maybe more than other libraries of a similar size and user population.
--Dave
==================
David Walker
Library Web Services Manager
California State University
http://xerxes.calstate.edu
________________________________________
From: Next generation catalogs for libraries [NGC4LIB_at_LISTSERV.ND.EDU] On Behalf Of Jonathan Rochkind [rochkind_at_JHU.EDU]
Sent: Thursday, October 29, 2009 9:39 AM
To: NGC4LIB_at_LISTSERV.ND.EDU
Subject: Re: [NGC4LIB] New Laws
Walker, David wrote:
>
> I think you measure it by the number of check-outs and downloads.
>
> Searches, in and of themselves, only tell you that people are "shopping" (to borrow a metaphor from e-commerce). Check-outs and downloads tell you the number of times people are "buying." If our systems are giving our users relevant results, then that _should_ lead to more check outs and downloads.
>
'More' is a relative word. More than... what?
I agree this is a useful general overview metric. I guess it allows you
to compare one version of your software in your patron community to
another version of your software in your same user community... unless
something else changed in the meantime to change the net number of
checkouts/downloads, which you should be cautious of.
But it doesn't let us measure it until it's deployed. It doesn't let us
figure out _what_ needs fixing, what can be improved in a given system,
or how. (Doesn't let us know if users do okay with a known title search,
but miserable with a subject/topic search). It doesn't let us evaluate
potential changes until we deploy them over a large community in
production and measure how it works. Etc. I doubt this metric is
sufficient.
Jonathan
Received on Thu Oct 29 2009 - 13:43:32 EDT