And How Human Raters Affect Your Rankings
Ever have a web page that finally gets to Page 1 for your keyword – and then suddenly plummets to nowhere land? I’m about to explain to you a very possible reason why this happens. I’m also going to share with you something straight from Google that tells us exactly what manual reviewers are trained to look for.
When it comes to Google’s organic rankings, there is a LOT going on to determine where a page ranks…and if it STAYS ranking there. One of the biggest misconceptions is that ranking in Google is purely a “man vs machine” type game.
Grab something to drink and sit back because I am about to show you a LOT about how Google makes algorithm changes, how manual reviews work (and what Google “Raters” are trained to look for straight from their training handbook), and much more.
Here are some quick links to the info you’ll find on this page:
The Reason Behind This Post
How Google Algorithm Changes Happen
About Google Manual Reviews
What About Panda?
What About On-Page “Footprints”?
Who Does Google Want to Please?
How Do Google Raters Judge Web Pages?
How To Survive a Manual Review
“…Okay, correct me if I’m wrong, but when I read the page you mention, there’s no evidence at all that this particular list of Webmaster Guidelines is actually used by Google’s manual reviewers to review a site. In fact, that page speaks exclusively about algorithmic changes and “the ideas and research that drive the development of our algorithms”. It doesn’t even mention G’s manual review process….”
He is absolutely right. The page I linked to was not congruent with what I was stating in my post.
Thank you for pointing that out!
So let’s talk about how Google Algorithm changes happen.
There were over 500… yes, Five HUNDRED, changes to the Google algo in 2010. That’s a LOT of changing going on, isn’t it?
“…Based on all of this experimentation, evaluation and analysis, in 2010 we launched 516 improvements to search. “
I’m going explain how Google makes these algo changes to the very best of my ability. I have a video, I have pictures, AND I have words to help explain this process. We all learn differently, so there should be something for everyone
First, I want you to watch this short video where Google employees tell us how algorithm changes happen. It’s only about 4 minutes, but very enlightening..and important for understanding the rest of this post.
Interesting, isn’t it?
Now let me put what they just said into an infographic so you can see the process flow that leads to an algorithm change. This image is straight from Google found here (very interesting stuff found there, too).
Now, let’s put all that into words. Below is the process as written out by Google (my own comments follow after each highlighted quote)
“A typical algorithmic change begins as an idea from one of our engineers.”
Someone says, “Hey, this query space is horrible….I have an idea on how we can fix it with the algo”
“We then implement that idea on a test version of Google and generate before and after results pages.”
They implement that potential algo change into a test version of the Google search engine that they call their “sandbox”. They take a page on the current results and then a page of the test results (with the test algo involved) and prepare them for the next step.
“We typically present these before and after results pages to “raters,” people who are trained to evaluate search quality. “
Google shows these before and after results to PEOPLE. These people are called “Google Raters”. These Google Raters do not work AT Google… they are “normal” people like you and I most likely working from home or some other remote location.
What are they trained to DO? Well, I’ll get to that in a moment and you’ll know exactly what they are trained to look for…and do.
“Assuming the feedback is positive, we may run what’s called a “live experiment” where we try out the updated algorithm on a very small percentage of Google users, so we can see data on how people seem to be interacting with the new results. For example, do searchers click the new result #1 more often? If so, that’s generally a good sign.”
If the manual reviewers/Google raters like the new “test” results better than the current results, the algo change is tested again as a small sampling of live Google traffic is fed into the Google sandbox so they can test these potential changes on REAL users. This also can explain why sometimes you see totally different results from what you saw minutes ago.
Then, the launch committee analyzes the data from that sandbox test. If the results show that users clicked the top ranking web pages on the new algo, then the change is rolled out to the entire index.
At that point, an algorithm change has happened.
Ok, got that? Yep, it’s a lot, but I think it’s important to know and to understand.
Let’s move on.
Many of us might go through our days working online and simply play the “man vs machine” game. We very well might be affected by algorithm changes AFTER they are fully rolled out to the entire Google index, but we don’t suffer from manual reviews from Google Raters.
That’s also how many “crap” sites seem to slip through the cracks.
On the other hand, I have witnessed sites where I feel strongly that they were first hit by a manual “Rater” review….then the algo changed and they dropped to nowhere land in the rankings. Can I guarantee that? Of course not – no one on the “outside” can guarantee things like that about Google. But these weird “dances” that happen and then a major algo change is confirmed just strikes me as odd.
From everything I know (and you’ll know too in a moment), Google Raters are not trained to look at anything “off page” for your ranking web page. They are solely looking and rating the page itself.
Also, as stated by a former Google Rater -
“Yes they have their algorythms designed to get the best sites to the top, but when they do get to the top, they have to be reviewed with a human eye in order to make sure the site has quality. This is the reason the review panel exists and is the reason some of you go from page one to page 10 without warning.”
I want to draw attention to something said there (emphasis mine) -
“…when they DO get to the top, they have to be reviewed with a human eye in order to make sure the site has quality”.
So you work you tail off and FINALLY get to Page 1 on Google for your keyword – you better hope your site can hold up under a manual review by a team of Google Raters.
It’s not ONE Google Rater that determines your fate either – it’s a group of them all giving their unique rating based on Google’s requirements to determine spam and “utility” (ie, usefullness, relevance) of your ranking web page (and only your ranking web page) . Then the data from the WHOLE is analyzed and action is taken (if mandated).
How do I know this? Again, I’ll tell you in a moment.
Google Panda is a separate computer program/algorithm that is run every few weeks by Google. I think that the signals they (Google) get from various manual reviews help “feed the Panda” algo, but I do not think Panda itself is a manual action (make sense?).
Panda has been described as more of a ranking “signal” as opposed to an actual algorithm change. This means that when Panda is run it applies a “tag” to an offending site so that when the “normal” Google algo comes around, it uses that “tag” as a ranking signal (which basically tells the “normal” algo – “do NOT rank any page on this site well”).
So my theory is that Google finds an issue in the organic rankings of their index, comes up with an algo “tweak” to correct it, tests that algo change and hands it off to Google Raters to see if the “tweak” makes a positive difference.
The results of these “tweaks” are also analyzed, and if trends are spotted, they become part of the next Panda run.
That’s just me thinking out loud. It makes sense to ME, but no guarantee it is accurate.
It also makes sense to me why sites who all have a certain “footprint” tend to get penalized or de-indexed en masse – all in one fail swoop.
If a group of Google Raters all report a certain something about sites that are constantly rated poorly as low-quality or even spam content, it makes sense to me that Google would use that info to create a signal in their algorithm that could affect ALL sites with that particular signal – whether they are “offenders” or not.
That’s why I made that post warning others about a certain “footprint” a training program was putting on their themes for students. I was very, very willing to be wrong with that post, but I couldn’t stand the thought of potentially being RIGHT and never saying anything to help others protect their work. (note: that situation is over and was handled BEYOND graciously by the program owners.)
Ok, let’s move on…
We’ve talked about what Google’s REAL product is in a previous post (read: We the People, In Order To Form a More Perfect Google) – and it has everything to do with users (because that is where Google gets their real product – from those that use their search engine).
Recently, Eric Schmidt, Executive Chairman for Google, released his written remarks before appearing before the Senate Committe on The Judiciary Subcommittee on Antitrust, Competition Policy, and Consumer Rights on September 21, 2011(wow, that was a mouthful…lol).
In those written remarks, Schmidt states (page 7):
“First, we built search for users, not websites, and no matter what we do, there will always be some websites unhappy with where they rank. Search is subjective, and there’s no “correct” set of search results. Our scientific process is designed to provide the answers that consumers will find most useful.”
Google is for USERS – not websites. Google wants to offer the best to it’s USERS – not to each of us site owners.
If webmasters don’t like where they rank – tough. But if USERS don’t like the results they find – BIG problem.
Google NEEDS users.
sidenote: also in those remarks by Schmidt, he said (page 2):
“…Google’s founders, Stanford graduate students Larry Page and Sergey Brin, believed they could build a better mousetrap….”
Mousetrap? Uh, I don’t think I woulda said that. Isn’t that kinda what they are being investigated for? Just saying.
So Google makes it clear that they aim to please USERS and they employ HUMAN Raters to tell them what they find relevant, useful, and spam (among other things). PEOPLE feed the Google algorithm – it’s not quite man vs machine, is it?
Now, yes, Google trains these raters to determine spam, low-quality, etc based on GOOGLE’S guidelines, but it’s still PEOPLE doing it.
So, how are these Google Raters trained to determine what is spam, what is a “commercial intent” site, what is a “thin affiliate site” and other things that affect the rankings of all our web pages?
I’ve got a 125-page document straight from Google that tells us. How did I get it? Well I Google’d it, of course…lol! It’s right there in Google’s index – easy for anyone to find.
Click to read: 2011 Google Quality Raters Handbook
EDIT 10-18-2011 8:57 pm EST – I’m sorry everyone, but Google has contacted me and asked me to stop linking to this document.
important to note: this was published on March 30, 2011 – AFTER Panda came to be.
I’ve read through the whole thing a few times and I’ll tell ya what – if we can each train ourselves to be Google Raters, we will be able to put out the content that Google DOES want to rank. No need to game them, we simply just give them the quality they are looking for – which I believe is what Google wants, don’t you?
SEO is very important – it allows us to give the Google algo the signals it needs to determine what our web page is relevant to and should/could rank for.
But first impression, appearance, and perceived “utility” of your web page is equally important – and those things are judged by Google users just like you and me before they become part of any computer algorithm.
To sum up, the list of questions I linked to in yesterdays blog post is, to me, a simple way to ask ourselves the very detailed and specific things Google asks its Raters to do. That’s why I linked to that list of questions as opposed to linking to a 125-page document.
Google wants… no, NEEDS to please their users. “Quality” is determined by users regardless of what their perception of the search query intent is…and regardless of what they perceive a “quality result” to be.
As Schmidt said, quoted above – “Search is subjective, and there’s no “correct” set of search results.”
So very true…and every day – every second – new web pages are found and added to the Google index. Some pages might be perceived as “better” answers to Raters….others might be found to be “bad” answers. It’s an on-going thing. Nothing static about it.
One thing that we just might know for sure – once YOUR web page gets to Page 1 of your query space for a Google search, you very well might be judged by PEOPLE too.
Are you ready for that?
To please Google, we have to please PEOPLE.
Comments, questions, and all that are always welcome. I’ll answer the best I can, but needless to say, I don’t work for Google. I’m just one little woman working from her home in Georgia trying to make sense out of all this – and help others make sense out of it along the way.