Google Raters – Who Are They?

Wow, my post about how Google makes algorithm changes sure got a LOT of attention. While I happened to think the post itself was pretty darn informative (if I can be so humble…lol), it turns out that the majority of folks visiting just wanted a copy of the 2011 Google Quality Raters Handbook. Makes sense, but as most know by now, I was contacted by Google and had to stop sharing and linking to that document. So, let’s move on and talk about these Google Quality Raters. Who are they? What do they do?

Google Quality Raters are out there rating not only organic search results, but also Google ads (AdWords) and Videos, and probably more things but those are the three types of raters I am sure of.

There is a good forum out there that is all “Quality Raters” info and discussion. I poured through a couple hundred pages of posts but you are welcome to read over at that forum here.

The Raters I will be talking about today are the ones that rate the organic results – called Search Quality Raters. In a nutshell, Google outsources this job to outside companies and those outside companies hire independent contractors to do the work.

What Is a Google Search Quality Rater? How Do They Get Hired?

Here is a quote from one of the forum posts that sums up the position of a Google Search Quality Rater quite well:

“There are a few names for this position. The companies hiring for this are Lionbridge, Leapforce and Butler Hill. I am not sure if Workforce Logic still hires for this or not. Lionbridge titles this position as an internet assessor and Leapforce titles it as Search Engine Evaluator. You do not apply with Google. These companies contract with Google and hire independent contractors.
I work for Lionbridge. I enjoy the pay, but the task availability is not always steady and the work can get boring sometimes. Basically what it entails is assessing the utility of search engine results. In order to get started, you must apply and if they have openings in your area and you qualify, they will invite you to take a test.
The test is long and you must study carefully.
The pay is great(I cannot disclose due to confidentiality agreement), but it is hourly.
It is a great way to earn a part-time income.”

So, to sum up – these people are “stay at home” type folks…moms, between jobs, students, etc.

They can apply for a Quality Rater position through one of these outside companies.

If accepted, they then take a 2-part exam to qualify their ability.

“The exam consists of two sections: Section 1 has 24 theory questions and Section 2 consists of 270 practical exams. “

If they pass the exams, they are then hired to work as an independent contractor (ie, work from home, no taxes taken out of pay – no employee benefits). They tend to work 10-20 hours per week and are paid by the hour. From what I can gather, they make something like $12 – $15 per hour.

These QRs (Quality Raters) are only allowed to work for one year. Then, they must wait 3 months before they can apply for that position to work again.

Now, these folks, for the most part, are not internet marketers or especially experienced with Google or organic rankings from the perspective we are. They are “normal” users of Google. They hire these raters from all over the world and those speaking all types of languages.

These quality raters are not new for Google. One Quality Rater in that forum said this:

“Just to reiterate from earlier in the thread, I was in the original group of raters back when the project started in late 2003/early 2004….”

So Google has been doing this for about 8 years.

Got all that? Good – let’s move on.

What Do Google Search Quality Raters Do?

These Quality Raters for organic results in Google are given 2 types of rating assignments. One type is when they are given one keyword and one url and are instructed to rate the relevance (ie, “utility”) of the url to the intent of the keyword. “Intent” according to the quality raters handbook is summed up as a keyword being a “do”, “know”, or “go” type keyword.
The rater decides if the keyword is relevant to something that searcher would want to do (ie, buy something, watch something, etc), something they want to know (ie, info on a topic, reviews, etc), of somewhere they want to go (ie, go to YouTube.com when they search for ‘youtube’).

Then they look at the url and decide if that url is relevant to that specific search query. They are also given an opportunity to mark that url as spam and to give notes about that url. Important to mention: A Rater is allowed to rate a url as BOTH spam AND as being relevant. An example of this, to me, would be a keyword like ‘buy digital camera’ and the url DOES offer a way to buy a digital camera but the site is chock full of banner ads of other “spam-like” stuff.

Important Take-Away: Since these raters are typically “normal” users of Google, first impressions of your url count BIG TIME. I think we all agree that when we each come to a web page, we cast judgement within 3 seconds of landing on that page, don’t we? Quality Raters are totally the same.

The other type of assignment these raters get is when they get 2 sets of search results (ie, a first page result for a keyword search). One result page is the “before” page and the other is the “After” (if you don’t know how this works, learn how Google makes algo changes on this post here at PotPieGirl.com. The video is short and very helpful – and yes, there is an info-graphic picture, too).

With this type of assignment the raters pretty much rate which set of results is “better” in their opinion.

What Are The Performance Requirements for a Google Quality Rater?

Since these quality raters are paid by the hour, there is a certain level of performance they are required to maintain. They receive progress reports on their performance that appears to be a star-rating type system.

Their performance is judged by how many urls they do per hour.

For the single url/keyword assignments, it appears they are required to do 30 per hour.

From that forum:

“Hi, I would like to ask you how many URLs per hour do you usually do? More than 60 or less than 60? I have no idea how many is the average because no one ever told me how many should I do. Thanks “

One other rater posted this as a response:

“I believe that 30 is the minimum expected for rating, within a few weeks after you’re hired”

Another posted this:

“I believe that 30 is the minimum expected for rating, within a few weeks after you’re hired (not including the test week). You’ll get an email if you fall below standards. They stress quality over quantity, however, so that overall, a slower rater who rates really well may be preferable to a fast one who misses the mark more often. “May” is the operative word, we don’t know what the criteria are for judging our work.”

It really appears as none of them are actually SURE about what level they are expected to perform at. Interesting.

Another Quality Rater posted this:



“Experienced QRs, on average, how many U*Ls do you personally complete in one hour? I just got my 2nd progress report and productivity is only at 3 stars when I do about 45 to 50 per hour.”

Only a 3-star rating when he/she does 45 for 50 an hour?!?! Yikes!

With these “before and after” type assignments (called “side by side tasks” – SxS) it appears they are required to do 20 per hour.

What REALLY concerns me about this is one main thing – how in the world can you really rate a web page in 2-3 minutes? So, as I mentioned above – First Impressions REALLY matter.

One quality rater posted this in that above mentioned forum (emphasis mine):

“It’s not possible to do 20 SxS tasks an hour if you click through to each result. It’s pretty easy on the occasions when the sides are nearly identical and you just have to decide whether you’d rather have A or B somewhere. When I was doing 20 I did a lot based just on snippets… I had been doing 2-3 an hour without complaint from them, mind you. And then I figured that if they wanted me to go faster maybe they didn’t want me to be QUITE so careful about judging every single result. Now I am not sure what they care about! Maybe it helps them to have a lot of different perspectives and styles of rating. Maybe no one is checking up on our work at all!

Anyone else thinking, “Oh boy…that’s not good news”?

Your url can be judged simply by your snippet in the search results! It’s possible no one actually LOOKS at your url at ALL!

Important Take-Away – Be SURE your snippet (ie, url meta description) is as relevant to your target query as possible. Now, Google can, and does, auto-generate these on their own MANY times, but try and control what you can.

Another quality rater posted this….which really bothers me:

“I would guess that for most people, the biggest challenge is remembering *precisely* what it is that you’re rating. “

Which backs up my impression that many aren’t exactly sure about what is expected of them. Awesome.

Can ONE Quality Rater Change the Ranking of a Url?

In this interesting 2009 interview with Google’s Engineering Director, Scott Huffman. John Paczowski asked Mr Huffman this (emphasis mine):

JP: So you’re describing a process in which these evaluators are going to specific Web pages and rating them according to a specific criteria. Do these data have any effect on those sites’ page ranks or pay-per-click and Ad Word bids?

To which Mr Huffman from Google replied (again, emphasis mine):

SH: We don’t use any of the data we gather in that way. I mean, it is conceivable you could. But the evaluation site ratings that we gather never directly affect the search results that we return. We never go back and say, “Oh, we learned from a rater that this result isn’t as good as that one, so let’s put them in a different order.” Doing something like that would skew the whole evaluation by-and-large. So we never touch it.

Now this makes sense to me – ONE rater can not cause a rankings change. However, I do believe that if a certain percentage of raters mark one url as spam or non-relevant, that it does throw up some type of flag in the system that can cause something to happen to that url. Now I naturally do not KNOW this, but I get that sneaky feeling.

Do Quality Raters Rate EVERY Query Space?

It is impossible to have a human rater out there rating every single query space. Heck, 15% or more of searches each and every month are NEW phrases that have never been searched for before. Yes, brand new combinations of words that Google has never had before!

However, especially with the side-by-side rater assignments, Google is testing potential algo changes. The way the sample queries are rated can cause the algo change to roll out…which can affect a MUCH larger set of query spaces without a human ever looking at YOUR url.

In that situation, there is bound to be many “false positives” and there is not a whole lot you can do about it other than wrack your brain and try and figure out what the algo change was targeting (and good luck with that!)

How To Survive Google Raters

It’s tough to answer how to survive a Google Rater visit or a subsequent algo change due to Quality Raters at play. Here are 4 tips to help survive a manual review:

1. Accept what you cannot change – There is nothing we can do about Human Raters judging our urls or the things that happen to the algo due to OTHER urls being rated. Therefor, I think the most important thing we can do is – don’t stress over it. Manual reviews have been going on for years and I don’t see them going away any time soon. Just roll with it the best you can.

2. Be proactive – make sure you site/url gives a great FIRST impression – and don’t look at it as a marketer…look at it as a general CONSUMER. Would YOU keep reading on your site? Would YOU buy something from your site? Be a “normal person” and judge your own urls just as we naturally judge any OTHER url we visit.

3. Check Your Snippets – Keep an eye on how your snippet reads in a Google search result for your target keyword(s). Does it tell a potential visitor that your page IS what they are looking for? Does your snippet match what a visitor will actually FIND on your page?

4. Evaluate Intent and Be Relevant – Lastly, really think about your target keyword(s)….if YOU typed that phrase into Google, what would YOU expect/want to find? Is your url and content truly relevant to the intent of the keyword used to find your web page?

All in all, remember that these raters are people simply trying to earn some money from home. For the most part, they really don’t care what happens to a web page, they just want to do the job that is expected of them. Many aren’t exactly sure what IS expected of them, either. I would also imagine that many aren’t sure, don’t know, or simply don’t care how their actions fit into the BIG picture either.

Google was not created for webmasters – it was created for SEARCHERS. These human raters are the people they (Google) are catering to – not us marketers, ok?

Want a “professional” human review of your web page(s)? Here’s an idea….

The holidays are coming. Many of us will have a house full of people or be IN a house full or people. I’m willing to bet that YOUR house full of people is like mine – “normal people” that use Google to find stuff. Ask THEM what they think about your web page. Heck, show them 2 or 3 for your search query (one being yours) and ask THEM which page they like the best. Also, don’t forget to ask them WHY they choose one web page over another.

Your family can start a new holiday tradition – The Google Game!

Just an idea 😉

Thoughts? Questions? Ask away in the comments.


Edited 12/6/2011 To Add:
Just to let y’all now, there is a follow-up to this post now published here at PotPieGirl.com. The follow up has clarification from Matt Cutts.

You can read the follow up post here: Matt Cutts Debunked Me

Would You Like a

FREE Copy Of My

Blog Post Mechanic Guide?

($19.99 value)

 

If you’ve ever had a blog post that seemed to be doing well and moving up the ranks in Google only to see if stall out and STOP improving, you will LOVE this guide.

This guide contains 5 unique and simple steps to improve a stalled, or totally broken down, blog post.

 

Also very helpful for posts that never ranked or made sales at all.

 

Learning to be a Blog Post Mechanic will be one of the most important ranking/blogging skills you will learn.

Just enter your email below and I will send you this guide for free.

 

 

blog-post-mechanic-ppg-cover

Success! Now head on over to your inbox to click to confirm your email so I can send your free Blog Post Mechanic Bonus Guide to your right now =)

125 Comments

  1. 3bagsfull November 17, 2011
    • Juicing With Rika Susan December 3, 2011
      • Keith Brown December 12, 2011
  2. Kevin Taylor November 17, 2011
    • PotPieGirl November 17, 2011
  3. Paul November 17, 2011
  4. P miller November 17, 2011
    • PotPieGirl November 17, 2011
  5. steve November 17, 2011
  6. Derek Blandford November 17, 2011
    • Alex December 12, 2011
  7. Jonathon November 17, 2011
  8. Mark P November 17, 2011
  9. Linda H November 17, 2011
    • Jonathon November 17, 2011
  10. Wendy Owen November 17, 2011
    • Ewan December 5, 2011
  11. jess weagle November 17, 2011
  12. Candice November 17, 2011
  13. Juan Duron November 17, 2011
  14. Cindy November 17, 2011
  15. Gerald Potratz November 17, 2011
  16. Suzanne Prochaska November 18, 2011
    • PotPieGirl November 18, 2011
  17. Dale Reardon November 18, 2011
  18. Duy Nguyen November 18, 2011
  19. Cuisinart coffee grinder November 18, 2011
    • Edgar November 18, 2011
      • cybergrl November 21, 2011
  20. JP Adams November 18, 2011
    • Jonah Stein November 18, 2011
  21. Mindy November 18, 2011
  22. Longboards November 18, 2011
  23. hcg ultra November 18, 2011
  24. Alex November 18, 2011
    • Denis November 18, 2011
  25. Kushal November 18, 2011
  26. Ninja Web Services November 18, 2011
    • Joe Youngblood November 18, 2011
  27. Ninja Web Services November 18, 2011
    • PotPieGirl November 18, 2011
  28. Netleg November 18, 2011
  29. Rose November 18, 2011
  30. simo November 18, 2011
  31. Steve@Affiliate Marketing Tips November 18, 2011
  32. Matt_Cutts November 18, 2011
    • PotPieGirl November 18, 2011
    • Joe Youngblood November 18, 2011
    • Matt McGee November 18, 2011
      • Rodrigo November 19, 2011
      • Joe Youngblood November 19, 2011
    • Matt Cutts November 19, 2011
  33. Joe Youngblood November 18, 2011
    • Joe Youngblood November 18, 2011
  34. Angeline Quinto November 19, 2011
  35. David November 19, 2011
  36. Eric Demmers November 19, 2011
  37. Matt Nixon November 19, 2011
  38. Elizabeth M November 20, 2011
  39. Lauryn Doll November 20, 2011
  40. Elizabeth M November 21, 2011
    • PotPieGirl November 21, 2011
  41. cybergrl November 21, 2011
    • PotPieGirl November 21, 2011
      • cybergrl November 22, 2011
  42. Pierre November 21, 2011
    • AffraidOFGoogleRaters December 13, 2011
  43. Eugene November 21, 2011
  44. aura November 21, 2011
    • Erin November 24, 2011
  45. Seo Vietnam November 26, 2011
  46. Rajesh Magar November 28, 2011
  47. ankey December 1, 2011
  48. Amy December 2, 2011
  49. Heather December 2, 2011
  50. Heather December 2, 2011
  51. Grant in Nashville December 2, 2011
  52. Oli December 3, 2011
  53. John December 3, 2011
  54. Manish December 3, 2011
  55. Pascal - "German SEO" December 3, 2011
  56. Julian - SEO Specialist December 3, 2011
  57. John Charlton December 3, 2011
    • Beth Parker December 3, 2011
      • John Charlton December 3, 2011
  58. John Charlton December 3, 2011
  59. Financial Samurai December 3, 2011
  60. Ryan December 3, 2011
  61. SeoStrong December 3, 2011
  62. Cody Baird December 3, 2011
  63. Andre Morris December 4, 2011
  64. Lachezar Yordanov December 4, 2011
  65. Krisztian December 4, 2011
  66. Transcription Service December 5, 2011
  67. steve December 5, 2011
  68. Gowtham v December 5, 2011
  69. Upton MA Real Estate December 5, 2011
  70. Matt Cutts December 7, 2011
    • Jon Cooper December 7, 2011
      • Tim Burns December 7, 2011
      • Matt Cutts December 7, 2011
        • Tim Burns December 7, 2011
          • AffraidOFGoogleRaters December 13, 2011
    • Jonathon December 13, 2011
  71. Diego Velasquez December 7, 2011
  72. Thomas Rosenstand December 7, 2011
    • Alex December 12, 2011
  73. goseethem December 7, 2011
    • Tim Burns December 7, 2011
  74. Janice December 7, 2011
    • Janice December 7, 2011
      • Jon December 7, 2011
        • Dr Janice Duffy December 8, 2011
        • Dr Janice Duffy December 8, 2011
          • PotPieGirl December 8, 2011
            • Dr Janice Duffy December 8, 2011
  75. Search Ramble December 7, 2011
  76. Dr Janice Duffy December 9, 2011
    • AffraidOFGoogleRaters December 13, 2011
  77. Alex December 12, 2011
    • joe December 13, 2011
  78. starcook December 13, 2011
    • Nicholas Massey December 16, 2011
      • AffraidOFGoogleRaters December 16, 2011
        • Nicholas Massey December 17, 2011
blog-post-mechanic-ppg-cover

Can I Send You a FREE PotPieGirl Bonus Guide?

If you’ve ever had a blog post that seemed to be doing well and moving up the ranks in Google only to see if stall out and STOP improving, you will LOVE this guide.

This guide contains 5 unique and simple steps to improve a stalled, or totally broken down, blog post.

Also very helpful for posts that never ranked or made sales at all.

Learning to be a Blog Post Mechanic will be one of the most important ranking/blogging skills you will learn.

Just enter your email below and I will send you this guide for free (a $19.99 value)

 

 

Success! Now head on over to your inbox to click to confirm your email so I can send your free Blog Post Mechanic Bonus Guide to your right now =)