Wow, my post about how Google makes algorithm changes sure got a LOT of attention. While I happened to think the post itself was pretty darn informative (if I can be so humble…lol), it turns out that the majority of folks visiting just wanted a copy of the 2011 Google Quality Raters Handbook. Makes sense, but as most know by now, I was contacted by Google and had to stop sharing and linking to that document. So, let’s move on and talk about these Google Quality Raters. Who are they? What do they do?
Google Quality Raters are out there rating not only organic search results, but also Google ads (AdWords) and Videos, and probably more things but those are the three types of raters I am sure of.
There is a good forum out there that is all “Quality Raters” info and discussion. I poured through a couple hundred pages of posts but you are welcome to read over at that forum here.
The Raters I will be talking about today are the ones that rate the organic results – called Search Quality Raters. In a nutshell, Google outsources this job to outside companies and those outside companies hire independent contractors to do the work.
What Is a Google Search Quality Rater? How Do They Get Hired?
Here is a quote from one of the forum posts that sums up the position of a Google Search Quality Rater quite well:
“There are a few names for this position. The companies hiring for this are Lionbridge, Leapforce and Butler Hill. I am not sure if Workforce Logic still hires for this or not. Lionbridge titles this position as an internet assessor and Leapforce titles it as Search Engine Evaluator. You do not apply with Google. These companies contract with Google and hire independent contractors.
I work for Lionbridge. I enjoy the pay, but the task availability is not always steady and the work can get boring sometimes. Basically what it entails is assessing the utility of search engine results. In order to get started, you must apply and if they have openings in your area and you qualify, they will invite you to take a test.
The test is long and you must study carefully.
The pay is great(I cannot disclose due to confidentiality agreement), but it is hourly.
It is a great way to earn a part-time income.”
So, to sum up – these people are “stay at home” type folks…moms, between jobs, students, etc.
They can apply for a Quality Rater position through one of these outside companies.
If accepted, they then take a 2-part exam to qualify their ability.
“The exam consists of two sections: Section 1 has 24 theory questions and Section 2 consists of 270 practical exams. “
If they pass the exams, they are then hired to work as an independent contractor (ie, work from home, no taxes taken out of pay – no employee benefits). They tend to work 10-20 hours per week and are paid by the hour. From what I can gather, they make something like $12 – $15 per hour.
These QRs (Quality Raters) are only allowed to work for one year. Then, they must wait 3 months before they can apply for that position to work again.
Now, these folks, for the most part, are not internet marketers or especially experienced with Google or organic rankings from the perspective we are. They are “normal” users of Google. They hire these raters from all over the world and those speaking all types of languages.
These quality raters are not new for Google. One Quality Rater in that forum said this:
“Just to reiterate from earlier in the thread, I was in the original group of raters back when the project started in late 2003/early 2004….”
So Google has been doing this for about 8 years.
Got all that? Good – let’s move on.
What Do Google Search Quality Raters Do?
These Quality Raters for organic results in Google are given 2 types of rating assignments. One type is when they are given one keyword and one url and are instructed to rate the relevance (ie, “utility”) of the url to the intent of the keyword. “Intent” according to the quality raters handbook is summed up as a keyword being a “do”, “know”, or “go” type keyword.
The rater decides if the keyword is relevant to something that searcher would want to do (ie, buy something, watch something, etc), something they want to know (ie, info on a topic, reviews, etc), of somewhere they want to go (ie, go to YouTube.com when they search for ‘youtube’).
Then they look at the url and decide if that url is relevant to that specific search query. They are also given an opportunity to mark that url as spam and to give notes about that url. Important to mention: A Rater is allowed to rate a url as BOTH spam AND as being relevant. An example of this, to me, would be a keyword like ‘buy digital camera’ and the url DOES offer a way to buy a digital camera but the site is chock full of banner ads of other “spam-like” stuff.
Important Take-Away: Since these raters are typically “normal” users of Google, first impressions of your url count BIG TIME. I think we all agree that when we each come to a web page, we cast judgement within 3 seconds of landing on that page, don’t we? Quality Raters are totally the same.
The other type of assignment these raters get is when they get 2 sets of search results (ie, a first page result for a keyword search). One result page is the “before” page and the other is the “After” (if you don’t know how this works, learn how Google makes algo changes on this post here at PotPieGirl.com. The video is short and very helpful – and yes, there is an info-graphic picture, too).
With this type of assignment the raters pretty much rate which set of results is “better” in their opinion.
What Are The Performance Requirements for a Google Quality Rater?
Since these quality raters are paid by the hour, there is a certain level of performance they are required to maintain. They receive progress reports on their performance that appears to be a star-rating type system.
Their performance is judged by how many urls they do per hour.
For the single url/keyword assignments, it appears they are required to do 30 per hour.
From that forum:
“Hi, I would like to ask you how many URLs per hour do you usually do? More than 60 or less than 60? I have no idea how many is the average because no one ever told me how many should I do. Thanks “
One other rater posted this as a response:
“I believe that 30 is the minimum expected for rating, within a few weeks after you’re hired”
Another posted this:
“I believe that 30 is the minimum expected for rating, within a few weeks after you’re hired (not including the test week). You’ll get an email if you fall below standards. They stress quality over quantity, however, so that overall, a slower rater who rates really well may be preferable to a fast one who misses the mark more often. “May” is the operative word, we don’t know what the criteria are for judging our work.”
It really appears as none of them are actually SURE about what level they are expected to perform at. Interesting.
Another Quality Rater posted this:
“Experienced QRs, on average, how many U*Ls do you personally complete in one hour? I just got my 2nd progress report and productivity is only at 3 stars when I do about 45 to 50 per hour.”
Only a 3-star rating when he/she does 45 for 50 an hour?!?! Yikes!
With these “before and after” type assignments (called “side by side tasks” – SxS) it appears they are required to do 20 per hour.
What REALLY concerns me about this is one main thing – how in the world can you really rate a web page in 2-3 minutes? So, as I mentioned above – First Impressions REALLY matter.
One quality rater posted this in that above mentioned forum (emphasis mine):
“It’s not possible to do 20 SxS tasks an hour if you click through to each result. It’s pretty easy on the occasions when the sides are nearly identical and you just have to decide whether you’d rather have A or B somewhere. When I was doing 20 I did a lot based just on snippets… I had been doing 2-3 an hour without complaint from them, mind you. And then I figured that if they wanted me to go faster maybe they didn’t want me to be QUITE so careful about judging every single result. Now I am not sure what they care about! Maybe it helps them to have a lot of different perspectives and styles of rating. Maybe no one is checking up on our work at all! “
Anyone else thinking, “Oh boy…that’s not good news”?
Your url can be judged simply by your snippet in the search results! It’s possible no one actually LOOKS at your url at ALL!
Important Take-Away – Be SURE your snippet (ie, url meta description) is as relevant to your target query as possible. Now, Google can, and does, auto-generate these on their own MANY times, but try and control what you can.
Another quality rater posted this….which really bothers me:
“I would guess that for most people, the biggest challenge is remembering *precisely* what it is that you’re rating. “
Which backs up my impression that many aren’t exactly sure about what is expected of them. Awesome.
Can ONE Quality Rater Change the Ranking of a Url?
In this interesting 2009 interview with Google’s Engineering Director, Scott Huffman. John Paczowski asked Mr Huffman this (emphasis mine):
JP: So you’re describing a process in which these evaluators are going to specific Web pages and rating them according to a specific criteria. Do these data have any effect on those sites’ page ranks or pay-per-click and Ad Word bids?
To which Mr Huffman from Google replied (again, emphasis mine):
SH: We don’t use any of the data we gather in that way. I mean, it is conceivable you could. But the evaluation site ratings that we gather never directly affect the search results that we return. We never go back and say, “Oh, we learned from a rater that this result isn’t as good as that one, so let’s put them in a different order.” Doing something like that would skew the whole evaluation by-and-large. So we never touch it.
Now this makes sense to me – ONE rater can not cause a rankings change. However, I do believe that if a certain percentage of raters mark one url as spam or non-relevant, that it does throw up some type of flag in the system that can cause something to happen to that url. Now I naturally do not KNOW this, but I get that sneaky feeling.
Do Quality Raters Rate EVERY Query Space?
It is impossible to have a human rater out there rating every single query space. Heck, 15% or more of searches each and every month are NEW phrases that have never been searched for before. Yes, brand new combinations of words that Google has never had before!
However, especially with the side-by-side rater assignments, Google is testing potential algo changes. The way the sample queries are rated can cause the algo change to roll out…which can affect a MUCH larger set of query spaces without a human ever looking at YOUR url.
In that situation, there is bound to be many “false positives” and there is not a whole lot you can do about it other than wrack your brain and try and figure out what the algo change was targeting (and good luck with that!)
How To Survive Google Raters
It’s tough to answer how to survive a Google Rater visit or a subsequent algo change due to Quality Raters at play. Here are 4 tips to help survive a manual review:
1. Accept what you cannot change – There is nothing we can do about Human Raters judging our urls or the things that happen to the algo due to OTHER urls being rated. Therefor, I think the most important thing we can do is – don’t stress over it. Manual reviews have been going on for years and I don’t see them going away any time soon. Just roll with it the best you can.
2. Be proactive – make sure you site/url gives a great FIRST impression – and don’t look at it as a marketer…look at it as a general CONSUMER. Would YOU keep reading on your site? Would YOU buy something from your site? Be a “normal person” and judge your own urls just as we naturally judge any OTHER url we visit.
3. Check Your Snippets – Keep an eye on how your snippet reads in a Google search result for your target keyword(s). Does it tell a potential visitor that your page IS what they are looking for? Does your snippet match what a visitor will actually FIND on your page?
4. Evaluate Intent and Be Relevant – Lastly, really think about your target keyword(s)….if YOU typed that phrase into Google, what would YOU expect/want to find? Is your url and content truly relevant to the intent of the keyword used to find your web page?
All in all, remember that these raters are people simply trying to earn some money from home. For the most part, they really don’t care what happens to a web page, they just want to do the job that is expected of them. Many aren’t exactly sure what IS expected of them, either. I would also imagine that many aren’t sure, don’t know, or simply don’t care how their actions fit into the BIG picture either.
Google was not created for webmasters – it was created for SEARCHERS. These human raters are the people they (Google) are catering to – not us marketers, ok?
Want a “professional” human review of your web page(s)? Here’s an idea….
The holidays are coming. Many of us will have a house full of people or be IN a house full or people. I’m willing to bet that YOUR house full of people is like mine – “normal people” that use Google to find stuff. Ask THEM what they think about your web page. Heck, show them 2 or 3 for your search query (one being yours) and ask THEM which page they like the best. Also, don’t forget to ask them WHY they choose one web page over another.
Your family can start a new holiday tradition – The Google Game!
Just an idea 😉
Thoughts? Questions? Ask away in the comments.
Edited 12/6/2011 To Add: Just to let y’all now, there is a follow-up to this post now published here at PotPieGirl.com. The follow up has clarification from Matt Cutts.
You can read the follow up post here: Matt Cutts Debunked Me
Thanks for doing all this research – It is truly eyeopening. Definitely gives me something to ponder.
I agree, Jennifer. A ton of research went into this post. Thanks. Everyone else seems to be writing about the document itself. First time I see something comprehensive and useful about the Google raters.
Google’s manual process isn’t usually talked about in depth, so it’s nice to see the explanation…
As always a super great post! You must have to do a ton of research for these info posts you do. Keep em coming, I love to read them!
Thanks, Kevin! Yes, I put in a lot of time before I ever get to writing a post.
Y’all are worth it 😉
Jennifer
I initially downloaded the guide and also applied for a job as a Google Rater just to see if it could help me out in my business. Wouldn’t matter if the job paid or was a an intern. Its always good to have inside information and the latest and greatest.. just don’t trade stock with it or you could end up in the slammer.
interestingly enough I applied to leapforce s few months back, just to get some extra money and see what it’s all about. In the application it asked for online work, experience etc….. I put Internet marketing,Seo etc….. And have not received a response.
Do you think that these companies are against hiring those in IM,Seo etc??
I would imagine that they wouldn’t really want someone with that type of experience, but who knows?
Thanks for sharing!
Jennifer
This is something I have been watching closely, especially since the last post. It kind of scares me when you realize these folks have no idea how much work goes into building a good website and that they don’t have a clear idea of how to evaluate one.
This really hits home because this week I lost my YouTube account without really knowing why. All I have on it was product reviews. I am not sure if it was somehow offensive to rater or a member of their community although I don’t see how a bike or game review could be. But one thing is for sure Big Brother is watching you.
Hey Jennifer,
These last few posts have been great. Honestly, I didn’t even know about Google Raters until you started writing about them. Interesting to know they’ve been around for 8 years. (apparently 8 yrs of my life passed by without me paying attention to anything *lol*)
It’s also interesting that none of them really know exactly what their job is…or at least what is expected of them in terms of performance.
I thought the most important part of what you wrote was when you said, “Google was not created for webmasters – it was created for SEARCHERS. These human raters are the people they (Google) are catering to – not us marketers, ok?”
I think many marketers tend to forget that. Whether you or I make any money isn’t Google’s concern. What they care about is making sure people find what they are looking for.
Interesting point, I think Google do not really want feedback from experienced online users but more from the average users in order to create and average response from the algorithm. I would think they use the profile of the raters (mainly location) in order to modify what the search engine returns as useful results.
Another excellent Google evaluation. I think the key point to remember is that EVERY visitor to your site(s) is effectively a rater and their actions during the visit are an evaluation. IMHO what Google says and what Google does are usually two different things. Whilst they suggest that “quality content” is a key ranking factor most of us know from our own searches that there are plenty of “poor quality” pages ranking on page #1 for many search queries. Just try “product + review” and check the page #1 results – especially for IM products – and you’ll find ranked pages that offer little or no content relevant to the search. Whilst SEO is important much of it is still a guessing game.
You’re a ‘giver’ Jennifer! Thanks for sharing what’s obviously been a ton of research. I for one sincerely appreciate your work. Thanks.
I am just starting to try some of this, or rather reading about it, and I have purchased your One Week Marketing. I have to say, I am inspired by someone from Winder, Ga doing all you do. My husband is from Monroe, although we live in Carrollton now. So, it’s good to know that real people succeed at this. I am retired, and just looking for a little extra income. I could never do what you do, but congratulations on your success! Do you think there is hope for someone who can’t do this full time, to make just a part-time income?
@Linda. Age is no barrier online. I’m 62 and only started 6 months ago. Wish I’d started 6 years ago but hindsight is always 20/20. Just find a niche you are passionate about and you’ll find this is a great retirement hobby with cash benefits. Keep following Jennifer’s posts and you’ll get regular inspiration.
I have been through a human rating for one of my sites. It has been on top of Google for a few years (for some keywords) and has thankfully survived. I have mostly originally content on the main pages, but have article pages with syndicated content – the site was created in 2004 and we did those sorts of things back then!
I have 2 adsense blocks on each page, one above the fold. There are a few afilliate products being sold on the site.
Not quite sure what this all adds up to, but I think it depends on the human reviewer relating to your content?
Thanks Jennifer
Wendy
Hi Wendy,
My daft question for the day. How did you become aware that one of your sites had been through a rating? Do they inform you in an official capacity or did you discover it some other way?
I think this is a good reminder that when we build a website we need to think about the people who read them and not just SEO
Thanks Jennifer, I didn’t know anything about all this. I really appreciate you sharing this with us!
I applied for a Google Rater job through Lionbridge but did not get accepted. At the time I had no idea what a Google Rater was, in fact, I thought it was a scam at first because I didn’t imagine Google would sub-contract work like this. The process was long and I didn’t even get to the exam portion. Not much was described about the day to day tasks involved but pay rate was about what you noted.
Thanks for your insight Jennifer.
Hey Jennifer,
You are the only marketer that I know of that will spend days or even weeks of your time researching something (but it seems your favorite topic is Google LOL), and then share your research with your followers! I’m sooo happy that I DO follow you because I would never have the time or patience, for that matter, to do all of that research!
Thanks Jennifer!
Cindy
These recent posts are eye opening and definite “value add” from my viewpoint. Give us great insight into considerations that should be addressed as we design our websites. Thanks for the great work !
This is so interesting, but depressing at the same time! I’m a fellow Georgia girl making a living online. This year has been so crazy – I keep thinking about the book “who moved the cheese”. Thanks for all your research on this, I learned a lot.
I love that book!
Jennifer
Hi,
Thanks so much for a great summary of the process and how it all works.
Dale.
Hi Jennifer,
It’s been a long time since your last post. But every time you come back, you come back with epic stuff lol! I agree that we should see our sites from consumers perspective not from marketer’s. As “normal” people might have different things to look at when they visit a webpage.
And again, your survival guide is really helpful. I think the best way to deal with it is to try providing more useful information and make a good first impression.
Duy.
I try not to let these quality raters bother me. The reason is because Google won’t target the query spaces I work in. Imagine google targeting a keyword that has only 500 exact searches and asking the quality raters to rate the urls. It’s almost impossible. I only care about algorithm updates like caffeine and panda. I know you put a lot of effort into creating this post and we, the readers, appreciate it.
Sorry for the keyword in the name field.
Charley
Just curious, Charley, but what is the point of your “cuisinart coffee grinder” website?
Edgar I would say that Charley is trying to see if he can rank for that keyword or it is just a dummie site he uses when making comments on peoples blogs.
Wouldn’t it also be a good thing to get these raters to remove a lot of the old, irrelevant sites that are clogging up the internet while they are working through their urls? There are some blogs and sites that have not been updated for several years yet they seem to take up valuable space within the top search numbers.
Big thumbs up to that comment. The #2 result for “digital signature” is a post on how to add a public key to your email on a site that hasn’t been updated since 2004. On the homepage (youdzone.com), he proudly proclaims
Thanks for doing this! I have never seen such a comprehensive post on who the raters are and what they do. In fact, I really didn’t even know they existed until you brought them to my attention. It seems that if we work to make sure our site is a quality one, we stand a good chance of surviving them. 🙂
I believe this is a good reminder that after all of us build a website we have to think about the individuals who study them and not simply SEO
There has been a lot of people talking about these human reviews and I thought that they had just started, so thanks for letting us know that its been going on for years, so we dont have to panic quite as much. It may be however that Google is going to concentrate on this element much more over the coming months so this is a timely reminder to have another look at our websites through the eyes of the consumer.
Thanks for a very interesting post, Jennifer.
I imagine that a rater’s opinion would not often directly affect a website’s ranking. There are just too many websites out there for a manual process like that to be effective.
I expect that Google are mainly using their raters’ opinions so they can improve their algorthms, and also to confirm that their algorithm changes ARE actually improving the user experience.
Really appreciate the work you are doing to help us better understand how Google works. It must be quite a delicate mission, given that Google don’t seem to want us to know how they work!
Thanks for sharing.
+1
It looks like many people this of them as of “website raters” while they are only “result raters”.
They give signals about how good/bad Google’s algorithms do, not about what site should go up or down.
If many of them mark some result as spam (result, not a site), it won’t be a signal to Google to remove/penalize the site. It will be a signal to improve the algorithm so that only relevant results make it into top 10.
Hi, Jennifer i don’t think i have come across any information on the internet so far about google human reviewers. Its just obscure and guru’s just do not speak much about that and i strongly advice them to have a look at your blog post for learning more about the google working.
All these so called seo masters keep stressing on page and off page seo but they do not speak about the all important things like quality raters.
Also downloaded that “google quality raters pdf” from your previous post and it was very useful for me to understand about the quality and what google wants from a website.
Again i don’t forget to thank you very much for a great post. 🙂
After reading your article, I’m thinking that reviewers do NOT affect the search engine results. What they do affect, in the aggregate, is the search engine algorithm.
And personally, based on what I’m seeing in Google these days, it’s not working very well. Which doesn’t surprise me, based on the time these reviewers are spending on each site.
that would be a logical use. google doesnt like doing things manually, they like algos. so you can imagine that if they tweak a signal and aggregate ratings go down they see that as a bad tweak.
as a person with a research heavy background i would just hope Google’s field service firms are adept at weeding out research “professionals”. From the sounds of it the exams likely do just that.
Oh, and if you could send over a copy of that pdf . . . 🙂
LOL! If I had a dime for every time someone has asked me that……
Jennifer
“After reading your article, I’m thinking that reviewers do NOT affect the search engine results. What they do affect, in the aggregate, is the search engine algorithm.”
Things to consider:
1. Google lies. All the time. To all people and governments.
2. This is expensive.
3. This is (supposedly) thorough
4. Google never wastes data
5. Ocam’s razor.
If the raters are trying to mimic searchers, it occurred to me that they are probably spending too much time on each url. I believe that people make up your mind about your site within 15 seconds (wasn’t there a book called “Blink” by Malcolm Gladwell that said we assess everyone and everything that fast?). And I think G is aware of the speed at which people make up their mind (hence the “block this site” if you backspace within about 15 seconds.
The thing about snippets also made me smile – because that’s how I decide what to click on when I’m searching! I never just click on the first result. I scan for the snippet that seems to be to be the most useful to me. Don’t know why I do this, or even when I started to do this – perhaps I’m trying to save myself the time of trying out the sites one by one? But I bet lots of people do the same. Which is why those raters are doing an effective job.
in many google pages they state that they do not use third party agencies to employ staff, is it possible that they use Lionbridge, Leapforce and Butler Hill for this important position?
Jennifer,
As always another very informative post on the way Google really works. It is very interesting to see how these guys and gals work their magic. the ones doing 50 + pages an hours. Sheeesh, how can they even give any significant input on the value of the site in that time.
Normally we do not comment on ranking methods but I’ll explain a misconception: input from manual raters is used only in the rarest of cases when a non-brand cracks the top ten for high value money terms.
Beware the comment above: Thanks, Matt McGee!
Jennifer
thanks for the clarification matt.
It would be nice to get some confirmation about whether this is/was really Matt leaving a comment. The underscore in his name and the lack of an avatar makes me think it’s not. But I’ll welcome being wrong. 🙂
Confirmed He’s not Matt Cutts, thanks to Matt McGee on Twitter
Good catch Matt Mcgee
@YoungbloodJoe That’s a fake comment by someone impersonating me. I didn’t write that. cc @mattmcgee @jillwhalen @mark_barrera
For the millionth time, advertising on Adwords does not directly help your organic rankings.
someone may have mentioned this, but coming from a research perspective it would seem ludacris to pay for this immense amount of data and not use it. so i would tend to agree that in aggregate the raters ratings could be used as a signal in the algo.
that being said it seems like to improve on this system branding might be an option. people are more willing to let a brand they know of get away with things like Amazon.coms insane amount of related products on a page than a brand they dont know.
i just saw matts comment… =]
I have a friend who used to work for Google as an organic result rater, and I agree with all you said here.
You’re such a TEACHER, PotPieGirl.
Thanks for all these educational posts.
I will surely read the other posts here.
Thanks so much for the ultra long post! You’re awesome.
great article! Thanks. I had no idea there were raters looking at my pages.
@YoungbloodJoe It would be interesting if @potpiegirl would share the IP address that made that fake/impersonating comment. cc @mattmcgee
Jennifer, kindly forward me the IPs of all Google critics. Our engineers can quickly match their identity via analytics, adsense and Google + button logs, enabling us to manually inspect their sites.
Very interesting comment from Joe Youngblood catching Google and M Cutts in a lie, costly data is priceless and almost never thrown out. Google is particularly known for retention and heavy use of data.
It appears that options are very limited for average sites, the raters are biased against your not-famous sites even before they see them. Adwords to the rescue? If M Cutts is still posting here, would using Adwords help ranking directly or indirectly?
Thanks so much.
A few years ago, I was a military girlfriend who was looking for work and I remember the WAH forums talking about this and other opportunities. I don’t think I ever sat down long enough to even make the attempt to try and give that job a shot. Yes, it pays well if you want to sit at home instead of work in a local place in a rural area, but since learning about other ways to make money online – I’m preferential to letting the QRs work.
Overall, the lack of clarity is disturbing. I was absolutely appalled by what I learned from an AdSense publisher who wasn’t paid on time due to a glitch in AdSense, coupled with things like Google’s clear disconnect within several of its departments in terms of the advice it hands out.
Every company has its drawbacks and we’re all human, so I hope it works out, but overall I can’t say I’m surprised these QRs are so confused and assumptive with their tasks. Your insights are actionable – I’m grateful you break it down so sweetly. In essence ” Cover all bases”
Tedster, a senior member from Webmasterworld.com wrote that Google changes page titles to improve click through
“They’ve been working on their algorithm for when to change the page’s title and how, with a goal of improving click through.” Do you know if Google is making more money after these changes? It could mean better click through rate for the ever present Google ads, and less clicks for organic site results. My clicks are 37% lower for the past three months when Google started to change my titles.
Considering that he has had private talks with Matt Cutts at PubCon we should probably pay attention to his other comment:
“I never had any doubt – if an unfamiliar site starts ranking for a big money term it will get a manual check. That doesn’t mean it can’t rank ever, but if the manual raters question the ranking as well, then its time on top will be limited.”
As he indirectly says, Google wants brands, for brands are the life blood of advertising, leading me to believe that search is molded to support Adwords.
PotPieGirl, considering that you have the gift of simplifying things, can you write another blog post looking at Google’s income through the search rank changes?
Profit up, down, the same? And why.
Hi Elizabeth =)
That’s a pretty big topic, but I’ll look into it.
Thanks!
Jennifer
I was just thinking after reading this that the problem is that normal searchers are not getting paid to go to a website and that is because they are looking for something they want to buy, or get more information about. The searcher would evaluate a site based on if his/her needs were met.
So how objective will a google rater be if they are getting paid to evaluate a site? Especially if they do not have a vested interest in the site or the topic.
You can have your family members look at your sites and because they are not getting paid to look at it, there response will not be the same as a rater who has just looked at 30 sites before they got to yours.
I just think the fact that Google is paying people by the hour to do this work is not going to be as effective as if they just had interns doing it for free.
It looks as if the raters only have enough time to make their decision based on first impressions. Then the determining factor just may be the (niche or topic) the website is about an how competitive it is.
Its almost like asking your neighbor to tell you what he thinks of your house compared to another person down the street but you give him a $20 dollar bill before he gives you his answer.
Good points, Cybergrl.
I totally agree with human raters evaluating a site when they do not have a vested interest in the topic. However, due to that very fact, the first impression (ie, how the site LOOKS) counts even more, don’t you think? Same with family members – they may not care at all about the topic, but they can tell you what they think about the site and if they’d “trust” it or continue to read there.
It’s human nature to judge a song, a shirt, a house, a watch, a tv show, etc at first glance…same goes for web pages. The ol “book by its cover” thing.
Regardless, if you read my How Google Makes Algo Changes post, we see that Human Raters are just the start of testing these changes. The next step is to release these “new” results to a percentage of actual searchers – who are vested in the topics.
Thanks!!
Jennifer
Thanks for responding to my comment Jennifer, and I admire all the research you do, it is much appreciated.
Yes, exactly, how a site looks is going to be very important from this point forward. Webmasters will have to take time and jam pack their sites with enough information to make the visitors feel that they got their money’s worth so to speak.
But then on the other hand if I am coming to a site to buy something I do not want to go through tons of pages to find what I want. So I guess it is a thin line between providing enough value and getting the sale.
The sites (ease of use) and (relevancy) will be something to consider as well. I am glad that the algo changes will include actual searchers input because that will make the end results more accurate, hopefully.
>> how in the world can you really rate a web page in 2-3 minutes?
You’re all missing the point I believe. Raters are not there to rate a page or site. They are there to rate the search results, basically give their opinion on whether the results returned appear relevant. You don’t need to actually go to the pages, although that would help. This information is then taken to tweak the search algorithm.
Pierre, you are/were probably a rater.
IF you say a search result is relevant or irrelevant without looking to the whole page AND reading the content then i think you don’t really understand what searching the internet is about.
thank you good post, always thought that the evaluators is some sort of special elite programmers, is no ordinary people with interests that coincide with the average. classic form questionnaire. now 100% of sites needs to be done for people, not for search engines.
i have worked as a quality rater for Lionbridge. and a lot of what you say is wrong. you are also assuming that the document you saw is the only document we use. it is not. we have more than 50 different documents for different functions of the rating task. also, sites are not evaluated by hundreds of raters or it would be more obvious this happened because traffic reports would explode for a site being evaluated (think about it, really).
I want to understand a thing: how do you choose the keywords when you perform these searches ? The keywords are chosen by Google and provided you, for testing ? These keywords are high volume one ? Thank you !
Great post which informs us a lot on the methods and the human side of site evaluation, I learnt a lot. Congratulations on the deserved spotlight (and backlinks) your great post has earned, it gives us an example of how good research and useful content can be beneficial for all the community.
The people you describe above “Search Quality Raters” they are not alive anymore.
I guaranteed that there all cheat chatting is going on this forum website about. Google doe’t allow anyone to access there database & source.
Yes there there was a time when Google was having some people to rate search results but they are also not out-lander, all are belong to there staff members.
And,
Now days Google technology so mature to rate search result. so they don’t required any outside normal Google users to rate quality search result.
Thanks
“Great post which informs us a lot on the methods and the human side of site evaluation, I learnt a lot. Congratulations on the deserved spotlight (and backlinks) your great post has earned, it gives us an example of how good research and useful content can be beneficial for all the community.”
very good
I used to work as a Google Rater through WorkForceLogic. It was a fantastic job. My specific position paid $20 as I speak fluent German and did German specific rating. I was paid even more for “adult content” rating projects and I was often able to work nearly full-time on my own time doing it. After I took a full-time position, I ramped it down to 10 hours per week, which was the minimum at the time. Several weeks after doing that, they let me go which I was a little sad about but was best at the time. I find it ironic that years after this job (it was 06-07) I found myself doing SEO for a marketing agency.
@I was contacted by Google and had to stop sharing and linking to that document.
😉
I guess Google must have learnt something from China? Excuse us? That tells us ALL which direction the Google wagon rolls off to. Screw Google. these guys start to suck more and more and it’s about time they get pulled down of their high-arrogant sattle they are still sitting in.
A word to Google DIRECT… ever thought about how hard you may fell from so far high up?
😉
I have to add … it doesn’t really matter – Google’s search results still suck even more than ever before. Makes me wonder about that “professional” rater training these folks get… something like to retaliate, cheat, spy, lie, pretend and on top of all that … suck most and best. It appears Google must have hired that ugly sly face Mark Z.as their executive trainer.
This whole cyber game starts to get really boring. Leave it for those theory cloud ghost dreamers until someone pokes into their cloud and they are all vanished … just like real ghosts do 😉
I am surprised Google still uses so many human editors on the payroll. Seems like you could attain a wiki-like bevy of free editors.
Just to clarify one of your points on a post, the evaluators rankings don’t directly affect a ranking, even en-masse (Other than the porn flag). What they do affect is the algorithm. The benefit of SEE’s is that they tell Google how their changes to the algorithm perform in a huge way. So for instance if a change throws a spam site to the top of the rankings the SEE’s will quickly pick up on it and flag, so the programers know something is wrong in the Algorithm itself.
Thanks, this is very interesting. It isn’t that surprising but I didn’t know these details. I still don’t understand what the worry about how a rater rates your site is about. It seems to me Google is trying to tune its algorithm not identify bad sites. And if it does identify bad sites I can’t imagine it is really anything but the largest bad sites that would be impacted at all. But that is just my guess.
Is there some specific evidence (or guess) about direct impacts to sites out of the top 10,000 or so?
Thanks for sharing Jennifer. Had no idea that human evaluators existed for real. Going to click that Follow button so that I can come back again:)
Hi Jennifer,
just got the tip from SEOmoz, for this article. I am working in the german seo market and found on your blog tons of super helpful information.
Thank you so much for taking the time and share all these “inside” informations!
Best Regards from Germany!
Pascal
I read the 2007 and now the 2011 Google Raters Guide and it is good to read your post which throws even more light on the issues. What I get from the guide and your post is that perhaps these raters are employed to evaluate the algo changes, before and after so it can give Google a quick insight into if the change is possitive across the board. I like the thought that these raters don’t have a direct effect on rankings but only on the effectiveness of the algo changes. Maybe enough spam flags will get a website reviewed by some higher authority, other than the algorithm.
I will tell you my probelm? I used to be number1# for balcony clamps…in any google search engine in the whole world…truly in China, France, Germany…you name it I was there for about two years…about 5 months ago we changed our eshop provider… we noticed a drop in sales I then googled our key words and we have dropped off the planet…useing my SEO tools I cant even find us in the top 300 pages…I think i must have been visited by one of the quality raters…I have no idea why we have just vanished as the pages are farly informitive and I only use three key words and I dont over cram the text with the words or phases and i still have good ranking on my other products…the thing is how do I get them back (not google quality raters) I need to get my pages back to the top…
fantastic work and very interesting… thanks John
John,
If your rankings dropped right after you switched eshop providers, it probably had something to do with that. What is different between the two that could have caused a problem? Sometimes changing themes on a WordPress site will do the same thing. I had one site I switched to a different theme and the traffic dropped to about 50% of what it had been immediately. I switched back and within a few days it recovered.
thanks Beth, it’s not like that? I have over 1000 products and they are all intergrated now into out accounting system and picking…from placing an order om my site..everything from checking current stock to printing TNT lables (weights etc…) is all automatic, all my other product rank high…but this section dose not rank at all…it’s like it’s been removed from google…the same key words rank 1# on yahoo and Bing. I am running a check now on the pages…thanks for the advise…if I come across the problem or the cure I will let you know
Regards
John
here is my theory… the world is in recession..so maybe a lot of people or companies who use the Adwords are cutting back…I am about to pull all my budget as the cost per click have gone up 10 fold over the last three years and I am not getting the amount of traffic from them like i used too.
so maybe the good+++ pages that get a lot of the organic traffic and been down graded (by the human raters) so that searches are more likely to click on the paid adds…make money. some intersting facts I have found through years of monitoring my customers…the PPC customer does not stay on my site as long as the organic…the organic stay three time longer and look at between 6-10 pages. the PPC only look at 1-5 pages. so try to get the the top of organic as least on the 1st two pages…
I really think the best thing you can do is forget about all this and write real content and connect with other bloggers and people in your field.
Sam
The Yakezie Network
I have a new website and have been wondering about this exact subject since I heard somebody talking about this on the MozCon videos for this year. I feed alot better about getting my site evaluated by a human since from what I read I have nothing to worry about.
Thanks for the info, glad I found SeoMoz.
Jennifer,
I found this article via SEOmoz’s top 10 email list. I must say I have heard about this topic before but not to the extent of your post. I think that everyone will agree with me when I say that we are thankful for this post, great work. I will definitely star your blog and plus one it. 🙂
Also Oli made a great point, it makes much sense that it wouldn’t directly impact your organi c ranking, but rather add to the algo. Could you imagine if these testers had a direct impact on the search rankings? The whole system would be flawed. Anyways, great post I will be sharing it with my colleagues.
Jennifer,
First time to the post. Very informative. Your blog will become a regular for me. I love fresh, unique content. As a long time SEO guy, I can’t reiterate enough, add value, add something unique and you don’t have to worry about raters or any other tactics to improve serp rankings, traffic etc…
I really enjoyed reading this, I only wish I was able to grab a copy of the Quality Raters Handbook before you had to take it down. I was surprised to learn that there is such little oversight over the work being performed. One would think that Google and all of their resources could instead do something like this in-house, and increase the quality of the process (and the quality of their search product) as a result.
Very nice post. Thank You for making this information public, as it seems a lot of ppl wherent aware of this google “function”. In my opinion this is internal google competition between algo and humans.
I must have missed the Quality Raters Handbook 🙂 but this post was awesome. The most comprehensive I’ve ever read on Quality Raters.
Very informative post on Google raters! Also effective steps to follow. Being proactive and looking the business from your customer’s eye is really important.
Hi Jennifer,
Surely if Google spend all this money on building that level of data then it must affect the ranks, I would expect that (despite what google themselves say) you are right about a certain percentage of them marking a site as spam would have some kind of effect.
Thanks for the info about Quality raters…A new info for a new blogger like me..
As I understand it a Google rater will look at a website more closely when it reaches the first page and will determine if the site is providing quality content.
Hi, I need to dispel a misconception in this post. In the section “Can ONE Quality Rater Change the Ranking of a Url?” Scott Huffman is quoted saying that our search quality raters do not directly affect our rankings. That’s correct.
PotPieGirl goes on to speculate: “However, I do believe that if a certain percentage of raters mark one url as spam or non-relevant, that it does throw up some type of flag in the system that can cause something to happen to that url. Now I naturally do not KNOW this, but I get that sneaky feeling.”
That feeling is unfounded. Even if multiple search quality raters mark something as spam or non-relevant, that doesn’t affect a site’s rankings or throw up a flag in the url that would affect that url.
I’m confused; then what does it take for a website to be penalized when it goes through search quality raters? Is there a certain golden number of raters to mark it as spam for it to have any effect?
From my experience those that are in direct competition with google: travel sites, credit card sites, mortgage, loans, deal sites, q& a (Knol), pro-Apple blogs, po-Facebook sites, pro-Microsoft sites, news aggregators, shopping comparison sites, maps, local search, image search sites are almost assumed to be spam or thin content.
Jon, the search quality raters sit in the “evaluation” part of search quality and they assess whether a new potential search ranking algorithm is a good idea or not. When they rate something as spam or not, we use that data to answer questions like “If we launch algorithm A, will spam go up or not?” or “Has our quality/spam gone up recently?”
But the search quality raters are strictly “read-only”–they don’t directly affect our rankings in any way. If you think about it, you definitely wouldn’t want to spamfight on the same queries that you’re using to evaluate your quality: you’d get skewed quality metrics as a result.
To be clear, Google does reserve the right to take manual action on spam. But that action happens in the webspam team, which is completely separate from the evaluation team and the search quality raters.
A rater has to look at WHOIS among other things in the 150 page manual to see if the page is spam or not? LOL.
Off course Tim, that is highly relevant.
Also if you have affiliate links you are spamming, no doubt about that. Use Adsense so you don’t send so much spam in the world…
Google is abusing it’s search engine to push their products. They find all kind of ways to detect spam. Google ways.
“””Even if multiple search quality raters mark something as spam or non-relevant, that doesn’t affect a site’s rankings or throw up a flag in the url that would affect that url.”””
That tells us what is doesn’t do but not what it does do, if anything. Surely “search quality raters” have a purpose otherwise you wouldn’t employ them.
Are these Google raters manually checking Google Place listings as well?
Hi Matt
This is very interesting actually. If even several quality raters mark a site as “low quality” or even spam then this does not throw up a red flag at the least. Do I understand you correctly? And if so: Then what does the feedback from the quality raters do in relation to the ranking procedures?
It makes sense that if a large number of raters flag a site as spam that it would be penalised and taken out of the index, though look at my comment at the bottom of this page I beleive raters are not directly responsible for this kind of data evaluation, the amount of data online is far too large for this to be an efficient process.
I deduce from this that the process is to tweak the algorithms on a global level not to hammer down individual sites.
I guest that the quality testing is to analyze spammy sites to interpret how they are able to rank and ultimately how to stop them gaming the system. The finding of thousands perhaps millions of sites in individual niches are analyzed. The best course of action will then be plotted for each niche by G in the algorithms. my 2cents
To deduce you must have all facts and what you say can be done with much less expense.
A lawyer in a TC comment thread suggested that G cannot admit to manually adjust the rank in such a scale, Google always hides behind the algorithm. This is even more important when raters are given explicit guidelines to mark as spam clear Google competitors and small sites that may not have “trust” in the eyes of a random visitor.
Tim, yes, to deduce you must have all facts and for Google, ‘the facts’ are simply those that will most benefit their aims, not those of their users. . Of course they know that this will be noticed but they are too powerful to need to care.
For example, Google do not say their rankings are automated. This is the line they spin to their lawyers – well, at least in countries outside the US. Compare the argument they used as a defence in (Metropolitan Schools v Google (which has now been superseded): http://www.5rb.com/docs/Metropolitan%20v%20Google%20EadyJ%2016%20July%202009.pdf
with the argument in Kinderstart v Google: http://www.internetlibrary.com/pdf/Kinderstart-Google-ND-Ca.pdf
Google changed its documentation in direct response to the Kinderstart ruling. Apparently Mr cuts wrote: “Even though Google won the case, we tried to clarify where possible that although we employ algorithms in our rankings, ultimately we consider our search results to be our opinion”.
So in their opinion a website like ripoffreport.com can destroy businesses yet anyone who sues Google and writes about this is manually adjusted. No, this is not paranoia, my blog is listed as no 6 on Bing/Yahoo 9and appears on page 1 of my name SERPs) but cannot be found on Google and has a page rank of 0. There is nothing wrong with the blog from a SEO or a Google perspective (no keyword stuffing and other issues such as malware).
So if I may offer my opinion (and this is based upon the complete absence of any SEO ability, Google do what they want and rank according to their preferences – which may change on a daily basis and as a product of the internal split (between Cutts and the advertising head).
Given the above and the fact that Google profit from the wholesale decimation of small businesses by the website http://www.ripoffreport.com, I would not take the view of Mr Cutts as indicative of any Google authority. Bt hey that is just my opinion and I am currently suing Google (hence why my blog was removed from the SERPS).
Maybe my perspective that Cutts may have a brain but he has no soul (based upon his comments about Ripoff Report) is informed by my case. I am not and never was a business. I am just a consumer who took Google’s word and lost my career. However, at least 3 companies that Ripoff Report (and hence Google) have deemed to be verified safe’ over the past 12 months have been indicted by the FTC or successfully prosecuted by the ACCC. So Google is ripping off their users.
My case was recently picked up by the media and 1 major SEO supported my action. Many others may have been afraid to go against the Google god. In light of their arbitrary approach to ranking and their obvious lack of any soul (as cuts has demonstrated) I do not blame them. My blog provides an update on the case but we are on court next week for argument. Google have clearly ordered their lawyers to ‘deep pocket’ me and one has to question their rationale behind supporting a website that destroys small businesses on a global level.
Thank you for publishing my comment. I apologise for the typos but it is hard to edit when one is crying. I reiterate that it would have taken Google all of maybe 5 minutes to remove what they could verify were defamatory and false snippets published at the top of a search for my name. I pleaded with them over and over again. I lost a tenured position because my name was falsely indexed in terms such as ‘fraud’, ‘scam’ ‘ripoff’ etc.
This publication was a result of a revenge attack. It would have been inconsequential if the website removed it or if the website was not one of ‘high trust’ according to Google. However, require a payment of thousands of USD and this is evidenced by the more than 500 victims in comments in this petition.
http://www.gopetition.com/petitions/remove-rip-off-report-from-google/signatures.html
(mouse over ‘view’ to read the comments.
I will not go into further details here but it is all on my blog; http://drjaniceduffy.com/. However, it is not indexed in the SERPs for my name on Google. This is easy to prove on any freely invaluable checker for page rank.
The inane excuses of that guy Cutts are really insulting in light of the evidence. This applies not only to my case but of the research conducted by several researchers including Foundem
http://www.youtube.com/watch?v=BslAhJ5-C9g
The man has no soul but I cannot say too much about the legal strategy we will use except that Australian law does not bow to the CDA. I will instruct my lawyers to subpoena Cutts simply because he stated that he investigated Ripoff Report.
Moreover, I am not the only person outside the Us who has been decimated by the Google push to earn money from Ripoff report (they puts ads on the defamatory pages). Of course, Google has instructed their lawyers to try and ‘deep pocket’ my case but this will not occur. Thanks to the efforts of SEO writers over the past couple of years there is enough evidence to show that they are indeed liable under our law.
Have just spent an hour or so reading everything on your site. Obviously it is a painful situation and an even more painful litigation process, however, the quality content and its presentation puts many so-called authority sites to shame and is a credit to you. I’m sure the irony of Cutts posting here and studiously ignoring your comments was not lost on you or anyone else. As a frequent user of search I can assure you that what google says and what google does are poles apart. Cutts claims that “quality content” and “user experience” are key factors in rankings. That of course is utter BS. By way of example just take any IM product off Clickbank and do a product + review search. Most of the first page ranked sites will have a useless page with 50-100 words basically saying “I haven’t reviewed yet but come back again and one day it might be there” which hardly qualifies for quality content. Have tweeted every page on your site so hopefully that will get you a few more visitors. I will post on this topic on a couple of my sites soon with links to yours.
Best Wishes, Jonathon (across the Tasman in NZ)
Jonathon, Hi and thank you so much :). Your post really helped me as this has been a very long and lonely battle. A big thank you to Potpie Girl for publishing my comments. Many of the ‘name’ SEO sites have not published about this case. The exceptions are Frank on Searchengineland after the story hit the media here and SEOmoz (on the anatomy of a ripoff report thread). My comments on the latter are published under ‘interested aussie’ because I was writing under that name before I decided to come out of the closet in October.
You are my second knight in shining armour from NZ. Let me briefly tell you that in 2000 I was in China for a health conference. I gave away all my yuan the night before leaving and miscalculated the amount for departure tax. It is very hard to argue in Chinese but I gave it a go at the airport when trying to explain that I only had Australian currency and credit. The choices were yuan or USD – they wouldn’t accept AU currency then’ although it may be different now. The airport officials were not going to let me board without paying and the situation got a little heated.
A guy standing behind me in what was a long line that was rapidly increasing tapped me on the shoulder, showed me a NZ passport and popped the yuan I needed in my hand. It was only about $2 in AU/NZ but he did something proactive rather than stand there and complain that I was holding up the line (like other non Chinese in the line).
The NZ guy probably saved me from being arrested. I wasn’t going to budge from the line in the airport (as they wanted) for the sake of $2 and I am not going to give up my right to work because of a task that would take Google 5 minutes. We are in court for ‘argument’ on Monday. Google have moved for a strike out. This is just a ‘deep pocketing’ tactic and I have confidence in my legal team. We are asking the court to order them to file a defence.
The court dates take a huge amount out of me on an emotional level and I am stressed for days before. We probably won’t get a decision until February so after this hearing I will be able to focus on putting the rest of the content on the site. I have done a lot of research on the differences in law viz the global nature of the internet and want to be able to provide a resource for others in this position.
I looked at the times on the replies and Mr Cutts did, in fact ignore my posts. This is not surprising. However, he will not be allowed to ignore a subpoena and will have to testify since he commented about investigating ripoff report on his blog.
Hi Janice =)
I am also sorry to hear of your situation – I can only imagine the frustration. Being heard on this great big internet can be very hard – and part of the reason I chose to allow your comments.
Whether Matt did, or did not, ignore your comments is not something that should be handled here – and please hear me with the kindness I am offering. This is not the place for you and Matt to “have it out” or discuss your situation.
Matt came here to clarify a point made about the topic of this post – and I appreciate that. I appreciate all comments and activity here that brings more insight and clarity into the topic at hand.
I empathize with your situation and wanted to give you a chance to be heard if at all possible – and I think I did that. However, I wouldn’t expect Matt to comment on anything you’ve said here on this blog – especially if this is headed to court. Any decent lawyer would tell you to not communicate with that person without counsel, don’t you agree?
I wish you the very best!
Jennifer
~PotPieGirl
Oh yes I absolutely understand Potpiegirl. The issue is that google’s comments about their services are contradictory. I will leave it at that :).
Liked the post but it also raised many questions..
what if a url/website has been in spam and after reconsideration it became a seo friendly website ?
Potpie Girl, I will understand if you choose not to post this but since I posted on this thread my blog has been restored to the SERPS for my name. Google had removed it after they realised the content was concerned with my court action. I am sure of this because initially it was on p. 1 of the SERPs and then it was not anywhere (although it remained at the top of p. 1 on Bing/Yahoo).
Last night I did a name search and it was again on [age 1. I had not done any SEO. It just appeared back in the SERPS after the above posts. So I was manually adjusted – again!
Lucky you!
I’m affraid many people are not that lucky!
Well my opinion is that Google do not use the raters feedback directly. They will however use the average response of a rater as something that the algorithm should do automatically. So in a sense the role of a rater is very important in the rating of a website. If you ask me the guidelines are probably one of the most important documents that a professional SEO should have to hand. Interesting how they ask raters only to use firefox to rate sites… is that because of the addons or is there something more to it? I have my own theories and evidence but perhaps someone else can shed some light into it 😉
I would think they use the raters ratings to determine how real humans are reacting to the algo changes. Quantitative data is cute and all but it becomes far more powerful when qualitative data is added into the mix.
Google isn’t doing anything too crazy (IMO) they are just being good data scientist and getting both sides of the picture before iterating changes that affect the entire world.
I must write here, because Matt’s spamcomments generate good back links 🙂
BR
Peter
Guess you’re not familiar with the term no follow are you. 😉
Do you find it normal people place a nofollow tag on external links because maybe Google might find this link not appropriate?
I decide myself which link i trust or not (our which one earned his link through a good comment). Result: All outgoing links have zero value…
🙂
Maybe you want to try again with that, i can’t make head or tail of it.