I was a Google Search Quality Rater for some time. There aren't any job openings for search quality raters at this time, but there are for ad quality raters, and the job posting for the two are very similar, so you can see what they look for (http://www.google.com/intl/en/jobs/uslocations/multiple/custsupport/ads-quality-rater-temporary-english/).
Training is done in a custom system using already-rated search results; if you get X number correct, you can move on in the process. All ratings have several sets of eyes on them, and even more if the ratings differ (say between a 3 and a 5 on a 5 point scale). There is room (and a requirement) that you argue for your rating in that situation. In my experience, fellow raters were educated, tech-savvy individuals with the ability to make logical arguments; they look for people with broad knowledge since you have to be able to rate results for Lady Gaga, tsunamis, cricket results, and space exploration equally well (as an example).
Yes, the algorithm is always being tuned and to some extent personalized; for many popular subjects (people, current events) it's unlikely that two people will get the same results even an hour apart , especially if they're in different geographic locations.
There are hundreds of quality raters (or there were a few years ago) at any given time.
- Julie
Julie Meloni
Lead Technologist/Architect, Online Library Environment
University of Virginia Library
jcmeloni_at_virginia.edu // 434-243-1974
-----Original Message-----
From: Next generation catalogs for libraries [mailto:NGC4LIB_at_LISTSERV.ND.EDU] On Behalf Of James Weinheimer
Sent: Monday, August 29, 2011 8:05 AM
To: NGC4LIB_at_LISTSERV.ND.EDU
Subject: [NGC4LIB] How Google makes improvements to its search algorithm
http://www.youtube.com/watch?v=J5RZOU6vK4Q
This is essentially an advertisement for Google Search, but still shows
a few rather peculiar mindsets, at least from the librarian point of
view. The first thing I noticed is where one of their specialists said
that they are interested in getting the best results *for each user*,
which actually reveal quite a bit I think.
Someone notices that a search "is not performing as well as they would
like" and it goes to "ranking engineers" who work with it. Then it goes
to "raters", i.e. external people who have been trained to judge whether
one ranking is more relevant or higher quality than another. Then it all
goes to a "search analyst" and a related committee, where the ultimate
goal is "to provide an informed, data-driven decision and to present an
unbiased view." Then comes a real example. Also, they claim they make
over 500 updates a year to the search results.
So, from this short film there is an overall impression of search
results that are changing constantly, i.e. updates of almost 2 a day;
plus a great deal of subjectivity, i.e. the best results *for each
user*, mixed with supposed objectivity (reliance on "experts" and
data-driven analysis to get an unbiased view) that, when you consider
it, doesn't really explain anything at all. For instance, who are these
"raters" and who trains them and how? They seem to be the focal point.
I just discovered this (via a Google search!): Google General Guidelines
for Remote Quality Raters (2007)
http://www.seobook.com/full-text-googles-general-guidelines-remote-quality-raters-april-2007
It was taken down along with some videos at Google's request! (That is
revealing!) But I found a summary of it:
http://www.seochat.com/c/a/Google-Optimization-Help/Googles-Quality-Rater-Guidelines-Leaked/.
In spite of all this, it seems that Google remains completely a black
box that takes in information and spits it out and no one, at least no
one outside of the company, really knows why it ranks sites the way it
does. What does it mean when they say a search "does not perform as well
as they would like"? Still, reading those guidelines for raters makes me
wonder if this is going to be what the Subject Heading Manuals will
become someday.
I hope not!
--
James Weinheimer weinheimer.jim.l_at_gmail.com
First Thus: http://catalogingmatters.blogspot.com/
Cooperative Cataloging Rules: http://sites.google.com/site/opencatalogingrules/
Received on Mon Aug 29 2011 - 08:33:13 EDT