Wow, my post about how Google makes algorithm changes sure got a LOT of attention. While I happened to think the post itself was pretty darn informative (if I can be so humble…lol), it turns out that the majority of folks visiting just wanted a copy of the 2011 Google Quality Raters Handbook. Makes sense, but as most know by now, I was contacted by Google and had to stop sharing and linking to that document. So, let’s move on and talk about these Google Quality Raters. Who are they? What do they do?
Google Quality Raters are out there rating not only organic search results, but also Google ads (AdWords) and Videos, and probably more things but those are the three types of raters I am sure of.
There is a good forum out there that is all “Quality Raters” info and discussion. I poured through a couple hundred pages of posts but you are welcome to read over at that forum here.
The Raters I will be talking about today are the ones that rate the organic results – called Search Quality Raters. In a nutshell, Google outsources this job to outside companies and those outside companies hire independent contractors to do the work.
What Is a Google Search Quality Rater? How Do They Get Hired?
Here is a quote from one of the forum posts that sums up the position of a Google Search Quality Rater quite well:
“There are a few names for this position. The companies hiring for this are Lionbridge, Leapforce and Butler Hill. I am not sure if Workforce Logic still hires for this or not. Lionbridge titles this position as an internet assessor and Leapforce titles it as Search Engine Evaluator. You do not apply with Google. These companies contract with Google and hire independent contractors.
I work for Lionbridge. I enjoy the pay, but the task availability is not always steady and the work can get boring sometimes. Basically what it entails is assessing the utility of search engine results. In order to get started, you must apply and if they have openings in your area and you qualify, they will invite you to take a test.
The test is long and you must study carefully.
The pay is great(I cannot disclose due to confidentiality agreement), but it is hourly.
It is a great way to earn a part-time income.”
So, to sum up – these people are “stay at home” type folks…moms, between jobs, students, etc.
They can apply for a Quality Rater position through one of these outside companies.
If accepted, they then take a 2-part exam to qualify their ability.
“The exam consists of two sections: Section 1 has 24 theory questions and Section 2 consists of 270 practical exams. “
If they pass the exams, they are then hired to work as an independent contractor (ie, work from home, no taxes taken out of pay – no employee benefits). They tend to work 10-20 hours per week and are paid by the hour. From what I can gather, they make something like $12 – $15 per hour.
These QRs (Quality Raters) are only allowed to work for one year. Then, they must wait 3 months before they can apply for that position to work again.
Now, these folks, for the most part, are not internet marketers or especially experienced with Google or organic rankings from the perspective we are. They are “normal” users of Google. They hire these raters from all over the world and those speaking all types of languages.
These quality raters are not new for Google. One Quality Rater in that forum said this:
“Just to reiterate from earlier in the thread, I was in the original group of raters back when the project started in late 2003/early 2004….”
So Google has been doing this for about 8 years.
Got all that? Good – let’s move on.
What Do Google Search Quality Raters Do?
These Quality Raters for organic results in Google are given 2 types of rating assignments. One type is when they are given one keyword and one url and are instructed to rate the relevance (ie, “utility”) of the url to the intent of the keyword. “Intent” according to the quality raters handbook is summed up as a keyword being a “do”, “know”, or “go” type keyword.
The rater decides if the keyword is relevant to something that searcher would want to do (ie, buy something, watch something, etc), something they want to know (ie, info on a topic, reviews, etc), of somewhere they want to go (ie, go to YouTube.com when they search for ‘youtube’).
Then they look at the url and decide if that url is relevant to that specific search query. They are also given an opportunity to mark that url as spam and to give notes about that url. Important to mention: A Rater is allowed to rate a url as BOTH spam AND as being relevant. An example of this, to me, would be a keyword like ‘buy digital camera’ and the url DOES offer a way to buy a digital camera but the site is chock full of banner ads of other “spam-like” stuff.
Important Take-Away: Since these raters are typically “normal” users of Google, first impressions of your url count BIG TIME. I think we all agree that when we each come to a web page, we cast judgement within 3 seconds of landing on that page, don’t we? Quality Raters are totally the same.
The other type of assignment these raters get is when they get 2 sets of search results (ie, a first page result for a keyword search). One result page is the “before” page and the other is the “After” (if you don’t know how this works, learn how Google makes algo changes on this post here at PotPieGirl.com. The video is short and very helpful – and yes, there is an info-graphic picture, too).
With this type of assignment the raters pretty much rate which set of results is “better” in their opinion.
What Are The Performance Requirements for a Google Quality Rater?
Since these quality raters are paid by the hour, there is a certain level of performance they are required to maintain. They receive progress reports on their performance that appears to be a star-rating type system.
Their performance is judged by how many urls they do per hour.
For the single url/keyword assignments, it appears they are required to do 30 per hour.
From that forum:
“Hi, I would like to ask you how many URLs per hour do you usually do? More than 60 or less than 60? I have no idea how many is the average because no one ever told me how many should I do. Thanks “
One other rater posted this as a response:
“I believe that 30 is the minimum expected for rating, within a few weeks after you’re hired”
Another posted this:
“I believe that 30 is the minimum expected for rating, within a few weeks after you’re hired (not including the test week). You’ll get an email if you fall below standards. They stress quality over quantity, however, so that overall, a slower rater who rates really well may be preferable to a fast one who misses the mark more often. “May” is the operative word, we don’t know what the criteria are for judging our work.”
It really appears as none of them are actually SURE about what level they are expected to perform at. Interesting.
Another Quality Rater posted this:
“Experienced QRs, on average, how many U*Ls do you personally complete in one hour? I just got my 2nd progress report and productivity is only at 3 stars when I do about 45 to 50 per hour.”
Only a 3-star rating when he/she does 45 for 50 an hour?!?! Yikes!
With these “before and after” type assignments (called “side by side tasks” – SxS) it appears they are required to do 20 per hour.
What REALLY concerns me about this is one main thing – how in the world can you really rate a web page in 2-3 minutes? So, as I mentioned above – First Impressions REALLY matter.
One quality rater posted this in that above mentioned forum (emphasis mine):
“It’s not possible to do 20 SxS tasks an hour if you click through to each result. It’s pretty easy on the occasions when the sides are nearly identical and you just have to decide whether you’d rather have A or B somewhere. When I was doing 20 I did a lot based just on snippets… I had been doing 2-3 an hour without complaint from them, mind you. And then I figured that if they wanted me to go faster maybe they didn’t want me to be QUITE so careful about judging every single result. Now I am not sure what they care about! Maybe it helps them to have a lot of different perspectives and styles of rating. Maybe no one is checking up on our work at all! “
Anyone else thinking, “Oh boy…that’s not good news”?
Your url can be judged simply by your snippet in the search results! It’s possible no one actually LOOKS at your url at ALL!
Important Take-Away – Be SURE your snippet (ie, url meta description) is as relevant to your target query as possible. Now, Google can, and does, auto-generate these on their own MANY times, but try and control what you can.
Another quality rater posted this….which really bothers me:
“I would guess that for most people, the biggest challenge is remembering *precisely* what it is that you’re rating. “
Which backs up my impression that many aren’t exactly sure about what is expected of them. Awesome.
Can ONE Quality Rater Change the Ranking of a Url?
In this interesting 2009 interview with Google’s Engineering Director, Scott Huffman. John Paczowski asked Mr Huffman this (emphasis mine):
JP: So you’re describing a process in which these evaluators are going to specific Web pages and rating them according to a specific criteria. Do these data have any effect on those sites’ page ranks or pay-per-click and Ad Word bids?
To which Mr Huffman from Google replied (again, emphasis mine):
SH: We don’t use any of the data we gather in that way. I mean, it is conceivable you could. But the evaluation site ratings that we gather never directly affect the search results that we return. We never go back and say, “Oh, we learned from a rater that this result isn’t as good as that one, so let’s put them in a different order.” Doing something like that would skew the whole evaluation by-and-large. So we never touch it.
Now this makes sense to me – ONE rater can not cause a rankings change. However, I do believe that if a certain percentage of raters mark one url as spam or non-relevant, that it does throw up some type of flag in the system that can cause something to happen to that url. Now I naturally do not KNOW this, but I get that sneaky feeling.
Do Quality Raters Rate EVERY Query Space?
It is impossible to have a human rater out there rating every single query space. Heck, 15% or more of searches each and every month are NEW phrases that have never been searched for before. Yes, brand new combinations of words that Google has never had before!
However, especially with the side-by-side rater assignments, Google is testing potential algo changes. The way the sample queries are rated can cause the algo change to roll out…which can affect a MUCH larger set of query spaces without a human ever looking at YOUR url.
In that situation, there is bound to be many “false positives” and there is not a whole lot you can do about it other than wrack your brain and try and figure out what the algo change was targeting (and good luck with that!)
How To Survive Google Raters
It’s tough to answer how to survive a Google Rater visit or a subsequent algo change due to Quality Raters at play. Here are 4 tips to help survive a manual review:
1. Accept what you cannot change – There is nothing we can do about Human Raters judging our urls or the things that happen to the algo due to OTHER urls being rated. Therefor, I think the most important thing we can do is – don’t stress over it. Manual reviews have been going on for years and I don’t see them going away any time soon. Just roll with it the best you can.
2. Be proactive – make sure you site/url gives a great FIRST impression – and don’t look at it as a marketer…look at it as a general CONSUMER. Would YOU keep reading on your site? Would YOU buy something from your site? Be a “normal person” and judge your own urls just as we naturally judge any OTHER url we visit.
3. Check Your Snippets – Keep an eye on how your snippet reads in a Google search result for your target keyword(s). Does it tell a potential visitor that your page IS what they are looking for? Does your snippet match what a visitor will actually FIND on your page?
4. Evaluate Intent and Be Relevant – Lastly, really think about your target keyword(s)….if YOU typed that phrase into Google, what would YOU expect/want to find? Is your url and content truly relevant to the intent of the keyword used to find your web page?
All in all, remember that these raters are people simply trying to earn some money from home. For the most part, they really don’t care what happens to a web page, they just want to do the job that is expected of them. Many aren’t exactly sure what IS expected of them, either. I would also imagine that many aren’t sure, don’t know, or simply don’t care how their actions fit into the BIG picture either.
Google was not created for webmasters – it was created for SEARCHERS. These human raters are the people they (Google) are catering to – not us marketers, ok?
Want a “professional” human review of your web page(s)? Here’s an idea….
The holidays are coming. Many of us will have a house full of people or be IN a house full or people. I’m willing to bet that YOUR house full of people is like mine – “normal people” that use Google to find stuff. Ask THEM what they think about your web page. Heck, show them 2 or 3 for your search query (one being yours) and ask THEM which page they like the best. Also, don’t forget to ask them WHY they choose one web page over another.
Your family can start a new holiday tradition – The Google Game!
Just an idea 😉
Thoughts? Questions? Ask away in the comments.
Edited 12/6/2011 To Add: Just to let y’all now, there is a follow-up to this post now published here at PotPieGirl.com. The follow up has clarification from Matt Cutts.
You can read the follow up post here: Matt Cutts Debunked Me