But what was the number one listing on Tuesday and Wednesday has vanished by Friday:

What’s really remarkable is how thoroughly this site has vanished:

It’s usual for a site to have the home page as the first page in the sitelinks list. For the home page to vanish… well, looks like Google hates this site. And I wouldn’t bet on any rank being passed.

Unsurprisingly, none of the Google penalty checking websites that I could find suggest that there’s a problem with this site. After all, it has results in the listings. Just very low ranked and none of the top pages (like the home page) show up. Sure sign of a problem of some sort!

So, what happened? Well, the site was probably reported to Google, possibly even before I saw the solicitation. And Google acted, quite quickly.

If you have a Google Webmaster Tools account, you can use Google’s Undeclared Paid Backlinks report form to report sites that are using and offering undeclared paid backlinks. I don’t know what you do if you aren’t a registered webmaster – perhaps find one and ask them?

So, what happened to the companies mentioned in the story? I can’t find them, now, either. Certainly not on page one of results (I’ve used the non-personal search on our tool to view Google organic and paid search as if in another country)

Google really isn’t happy with undeclared paid links. At all.

What is the Plus One Button for websites?

It’s a small piece of Javascript to insert on web pages that are candidates for Google Search Results. On each page, you need to refer to the JavaScript file, and then invoke the script.

<script type=”text/javascript” src=”http://apis.google.com/js/plusone.js”></script>

<g:plusone></g:plusone>

As you can see from the brevity of this code, it has no idea *who* has dropped the link, other than someone with authority to modify the site. This is in strong contrast to Facebook, where the ID is an integral part of adding “Like” buttons. The implication is that Google doesn’t care who drops the code – they care who clicks on the button it displays, and probably, they care more about which page was clicked.

Click on the icon that the code displays, and your ID is added to the list of users that are interested in this page – and the icon changes from white to blue, to indicate that you have clicked it. If you “Plus One’d” a page, then it gets marked in the search results, as shown above. Note that the tooltip is not changed – when you +1 a result, the tooltip still shows that you should click to recommend the page, but that will actually remove the recommendation. Probably a transient oversight at Google, rather than intentional.

Here’s this very article, towards the bottom of Google search results for “Google plus one for websites”, before anybody I know had “plus one’d” the page:

You can see that I’ve +1′d the page. It shows up under the listing, below the URI, and it knows that I’ve voted – but gives no clue how many *others* have clicked. At the point that this was taken, there were a handful of +1s for the article that were at least an hour old.

It appears to take at least fifteen minutes for the “plus one” information to be updated in search results. I have to admit that I became bored checking – so somewhere after 30 minutes and before four hours, the search results had added a friend, who I’d ask to make a timestamp-noted entry. This is *much* slower than the update to the count of “plus ones” shown on the page – that count changed in seconds. I’ll guess that the delay before turning up in search results is to add the social graph data of who I know that has clicked, so my search results pages show data about people I know. I suspect that this process takes some time… so if you’ve just put up a post and emailed your mates to +1 it, don’t be surprised if you see on-page activity a long time (internet time) before the Search Results update to reflect that.

Amusingly, while I was checking to see who else that I know had used a plus one on an article about a plus one (interesting behavioural change – I went looking *much* further into results – will others look deeper to see if their friends have marked stuff, and will that be a transient behaviour until the SERPs behaviour becomes clearer?)… I found that the preview had updated to include the latest summary within the article and the clip showing the position. Here, recursively, is this article showing a clip of Google showing a preview of this article showing a clip of Google’s search results showing this article…

The implication is that Google snatches previews at the time of interest and marks them up, as a much faster process than the social data is updated. Now… Why is the social data apparently slowed so much, relative to the preview? I don’t know, and I’ll be thinking about that for a while.

What Failed Before, and Will The Plus One Also Fail?

Failures aren’t the worst thing that can happen. Failures tell you when you aren’t doing the right thing. So long as you don’t do something that kills the business, you can learn from failures. A failure to learn from the lessons you should be learning from a failure, now that is a real failing.

What has Google tried before, and why didn’t they work? And are there lessons from those attempts about what would make Google’s “Plus One” work, or be yet another fail marker?

Google’s SideWiki

Perhaps the most comprehensive attempt to solicit users’ direct feedback on a page or a business, is the Google SideWiki. This is barely known about, even in search engine marketing circles, much less by the wider public. It lets any user add an annotation to a web page. If the web master knows about the SideWiki, they can respond… But I can’t recall ever talking to a webmaster who knew about SideWiki before I mentioned it, much less ordinary web users.

The problems for SideWiki are quite large. There’s no on-page presence (if you have the toolbar or Chrome, there’s a minor marker in the decoration frame for a web page, but nothing in the area that users focus on). The SideWiki is intended to be present even for sites that don’t want to participate – which is why it has such a peripheral vision presence. It’s a parallel web space so only the search geeks or the deeply aggrieved find their way on to it. Parallel page specific annotations are a rich source of information for Google, with a user indication of liking or not liking the content, and it can provide a parallel stream of keywords and some clues as to the importance of a site or page (e.g. the Apple home page has a lot more SideWiki comments than the Twitter home page – working out why that is meaningful is left as an exercise for the reader! Try using the SideWiki for this page to discuss this? Your attempt to use it may partially explain why it has been so little used and known.)

Considering Google’s first rule in their Ten Things (“Focus on the User and all else will follow”), I’d have to call SideWiki a brave, but doomed, effort. Collecting additional data on page value is a great idea. But doing it in a way that users aren’t aware of, and can’t possibly comprehend, without a marketing information campaign to support it, and proper tools to engage with it for site owners and regular users – well, it’s destined to be roadkill. SideWiki is, I suspect, a victim of Google’s Engineering Lead business philosophy – a fallacy supported by the success of their first effort in this space, Google’s technologically and philosophically profound switch in ranking the results in search engines.

The “Engineering Lead Marketing Fallacy” is that if you make a great product, people will find and use it – but the reality is that if you have something good, you need to educate, inform and persuade users that it is good; this may be a rich vein for another article on another day, though; just think about Sun, until I write that article. I’m deeply familiar with this fallacy, having been a techie myself, before my Damascene conversion.

Google Stars

In Google Search Results, you can “star” a listing. This means that it is a page that you like. Starred listings are preferentially popped up to the top of your results pages. Like SideWiki, that is great feedback for Google on pages that work well for users. Pages that are well liked should get lots of stars – an easy way for Google to automatically find the hot stuff and the useful stuff. They are directly related to Google’s Bookmarks service – and social bookmarking is one of the unsung other keys of Social Media (it used to be sung, back when delicious was new).

Why doesn’t “Google Stars” work? When I talk to users they haven’t noticed the little faint outline of a star, or couldn’t be bothered to work out what it might mean. This star is an unusual icon, faint, and unexplained. Only the most curious users work out what it is for. And Google does their best to make sure that users *won’t* engage directly with the Google Star. That may be a surprising statement, but you have to consider the user experience (UX) when considering why the Star doesn’t and can’t work very well to deliver user feedback.

What’s Google’s proud claim and ambition (number 3 in the ten things)? That they deliver the best page of search results so you spend as little time as possible on Google. This does not encourage users to spend time investigating the details of search results. Any system that Google develops that requires time to parse the page, infer meaning and interact with the results, is counter to that (so far) successful formula of reducing the time spent looking at Google’s own search results rather than looking at the content that you wanted to find. Because of this emphasis on shifting people off the search results page as quickly as possible, Google Stars will be a minor “also ran” in terms of soliciting feedback, but probably slightly more effective than the almost invisible SideWiki.

As a subset of this mechanism, is Google’s “spam marking” annotation. Same problem. I find out that the page is spammy when I’ve been to it. When I use search (rather than being an SEO and staring at search results for ages to inder why Google does what it does), my attention is not on cleaning up after Google’s spam cleaning efforts, but on getting to the page that helps me do whatever it is I’m doing. So the user focus is not, at the time of looking at the page, focused on the question “is this spammy?”; anything as onerous as a check box, is too onerous to work with and is inappropriate within the normal context of a user interaction with search.

Effectively using the Star or Spam Marking within Google Search Results means having to return to Google, perhaps rekeying the search, and the user doing so while intending to mark a result as being annoying. Too much effort for most people. You’d have to *really* hate a web page to take that much effort. Or remember that if you are using Chrome, you can use the faint star at the right hand edge of the input box (URL/search field) to Bookmark/+1/Star the page you are on.

Google’s Search Wiki

Wikis are great, for certain classes of problem at least. The previous version of our company main site was written using a search engine optimised wiki we developed in 1994 and ran until a catastrophic incident in late 2010, but that’s a story for another time…. The point was that a Wiki collects lots of stuff, organise it and let users inside and outside the company work on it.

Google used to use a Wiki in search results. Google’s SearchWiki was effectively another parallel web – another Google system that had potentially one web page for every web page out there. This was before the success of Facebook, when MySpace was still a power online. SearchWiki looks like “Social”, smells a bit like “Social” – and it failed. There’s no sign of the user generated content attached to search listings any more. I think this experiment fell foul of the same problem that afflicts Google Stars – but even more so. The problem being that users on a search page don’t want to interact with the search page, they want to go to a useful web page. The Wiki compounded that problem by making the interaction into an essay. Users like simple UI choices – “do I like this page”, “do I like this company”, “do I never want to see this again”? Only a tiny fraction of users like writing essays. More prefer 140 character limits (c.f. Twitter) and more yet prefer just a “Like” button.

Google Plus One

There have been a few other attempts by Google to get users to interact about sites and web pages. They’ve all, to my knowledge, been variations of the above themes – hidden content in a parallel web or user engagement in search results pages. Note that user engagement does work when well done – Google’s own search results are a partially a consequence of user engagement with the content (title, snippet, URL and the actual site all play a part, as does the user interaction with all of those). The failure is not in the effort to engage users, but in the absence of the first rule (“Focus On The User”) in the interaction models – I believe these exercises have been focused on improving search results for all users, at the expense of focus on the individual user as contributor. Clicking a “Like” button, or a “Plus One” button, is a contribution – it’s just non-textual.

Google’s Plus One Button is different from all these previous efforts though. It’s on the web site, not the search results. It’s where users are focused when they are doing something. I think it is the first thing that Google has done, probably since they invented the whole idea of Page Rank, likely to actively engage with user opinion. This is a markedly different activity from any effort so far. But will it work?

So long as the interaction model is kept to a minimum (“I like this and would want to see it again” as a checkbox) and on the page that the user is considering, and if it turns up in search results (“you liked this page when you saw it”), it’s reasonably good.

The idea that I’ll know that people I know have marked a page – that’s good, too, and that puts it closer to the Social Media space. Knowing that someone I respect likes a page, helps me to decide that I want to visit that page too. This may have a profound effect on SEO – and given that user preferences are having increasing importance in ranking, who’d care to bet that this won’t also become yet another Google AdWords Extension – allowing you to advertise more visibly when you have a preferred page in the listings?

What will stop “Plus One” from working?

Several things can stop “Plus One” from working:

• A good distraction from FaceBook and LinkedIn or GroupOn. Distract people from paying attention to “Plus One” when it is new, so that webmasters focus on some other issue to deal with.
• Google’s Engineering Fallacy – just because this is probably the best interaction model so far devised for improving search engine results, doesn’t mean that it will win. It needs marketing and communication efforts. It needs Matt Cutts to write about it and encourage webmasters to embed it. It needs the Webmaster Tools guys to make space for it and report the results in WMT. It needs to actively let webmasters know when their sites are loved (and relatively unloved – like Google Analytics Benchmarking data). And most of all it needs to be social – I *care* when Bill Slawski, or RustyBrick (Barry Schwartz), or Kim Clinks like a web page. I don’t care when it’s some random Joe (unless there’s several thousand real random Joe’s, because that has its own message about a page).
• It needs to be non-spammy. Google has mechanisms to understand whether real users are interacting; the votes should probably use the same kinds of invalid click detection systems used in AdWords and Search Results. However, reporting to those to webmasters may need the same kind of obfuscation that conceals exactly which clicks were invalid, in AdWords, to continue the security-through-obscurity approach used in click fraud detection mechanisms.

At the moment, I don’t see how how I respond to a friend Plus-one-ing a page. So it is limited in terms of social media. I can like, or not. I can’t interact. Unlike Facebook, where I can comment on something that someone has found. That may yet prove to be the greatest weakness. Being able to see my friends liking something – that’s good. Being unable to respond and correspond, a core weakness.

And the name… I can easily grasp “liking” a page. But “Plus One-ing” it? Smells geeky, even nerdy. Hard to say. Hard to write. Hard to put into a grammatically correct page unless treated as an adjectival-noun thingie.

Is there any sign that Google Gets Marketing Communications?

A bit. But the most prominent usage of the “Plus 1″ graphic on the announcement page is just a link to the giant graphic, not a way to plus one the plus one announcement… That’s hidden with the other social markers below the article – and not even first in the list. Placement itself means something. Come on, Google guys – you rank stuff by importance. If you put “plus 1″ towards the end of the list of social markers, how important is it? Learn, for goodness’ sake, from your experience.

I’ll probably update this section as I see signs of life-after-engineering. When I see some type of audience communication rather than simply an announcement.

Summary

Good effort, but not enough, yet, to be completely convincing.

Possibly worth getting your braver clients to sign up and see what happens.

More important for Google’s de-spamming of search results than for social interaction.

May fail, because it is only partially social – no response mechanism to friends.

Where’s the social list drawn from? Ah. Now that’s an interesting question… for later.

Potentially disruptive – but not clearly so.

Could be emulated, perhaps better, by Bing using Facebook’s social graph and their Like button data. How? Well, if Bing did this, then made a “plus 1″ boost results for a while, “until the algorithm settles down”, then SEOs would want to get a Bing button everywhere and people to click on them – and that would probably help clean up Bing results, increase Bing usage, and is another reminder on web pages of Bing’s presence in some markets (still just an also-ran, here in the UK). So, disruptive but not exclusive… and open to a competitor taking it higher, further, faster.

So – what’s it like? See this…

… and do, please, click it. :)

The most common answer that people give to the question on keyword density, is that it used to be important, but that the main value of keyword density analysis nowadays is to avoid keyword stuffing or keyword spamming. Pretty much everyone says that it is important to mention the keyword at least once, preferably with some other related instances, and ideally use semantic markup (or titles, descriptions and headers) to emphasise the keyword.

Not one person has said that you can rank for a keyword without using it on the page – a so-called “Google Bombing” or “Link Bombing” exercise. In a link bombing exercise, you use anchor text to target a page that doesn’t mention the keyword you are focusing on. It was a nice demonstration of the power of backlink anchor text, that you could rank a page against its’ intentions. George Bush was targeted for “miserable failure”, for example.

Interestingly, after the attacks on George Bush’s reputation – but possibly not directly triggered by that, Google eventually implemented a fix to better synchronise page content and backlink anchor text. This fix tool appears to be something that is periodically run, so unexpected results will creep in from time, for up to a few months at a time. The fix that Google seems to have used, is that the targeted keyword should appear in whole or in part, on the targeted page. So George Bush’s CV at the Whitehouse appeared again for the search “miserable failure”, when a Whitehouse staffer referred to some Senate discussions as a “failure”. Close enough for Google to decide there was a match… until the tool was run again.

And it looks like it is getting time to run the tool again. Unless there’s something else going on, that is….

• Date: 2007/07/16
• Article: Google AdWords, Click Fraud and gclid

This article about Google, AdWords, Click Fraud and gclid usually shows up fairly high in searches for “gclid“. Every so often, I check that search, not just to see whether the article is still ranking, but to see if anyone else has written anything interesting about that tracking parameter recently. What did I see yesterday, when I looked? I was, FWIW, thinking about call tracking systems – not just vanity, honestly!

first page results, UK, for gclid - not mentioned in two of these pages at all

OK, so how many times is “gclid” mentioned on that Crucial Memory page? Never. What about the Tax Credits page? Nope. Not there, either.

So, why were these pages regarded as important for users to know about, when they search for gclid?

Interesting puzzle, isn’t it? If you can solve it, let me know… I just might have a job for you!

Oh, and see if you can work out what the connection is between SEO By The Sea’s recent blog article about dates in snippets, and this article. It’s a little test that I’m doing – but of what?

I have my doubts that the Sidewiki sidebar will be rendered on anything other than informational sites with UGC – which, by and large, will already have a mechanism for users to contribute and so will have no need for a Sidewiki. The fear amongst business site owners will be that users with a grudge will control the conversation. There’s plenty of evidence that suggests that users with a good experience of a business are much less likely to tell others, than users with a bad experience – in a ratio of about in in 12 (users with a bad experience are much more likely to tell others). So even if your business manages to annoy only a small fraction of users, they’ll be a lot more visible than the small fraction who are ecstatic about your business.

Unless the Sidewiki enables a clearly valuable business interaction – one that doesn’t leave site owners worrying that Google is controlling the agenda, and one that doesn’t overemphasise the negative commentary – then Sidewiki will remain a sideline. I think the idea of UGC paralleling the site content is probably the right sort of thing to be thinking about. I suspect that the current implementation and interaction model aren’t enough to satisfy business owners to make the Sidewiki an integral part of the site’s experience.

I’d fear it being added to, for example, the NHS web site. The amount of quackery and bogus advice that could be offered would make me nervous about the value.

So, for now, Sidewiki continues to be a technology in search of a solution. It appears to come from a philosophy of user interaction that isn’t matched by the real world, that users will contribute overwhelmingly. That may be true for sites engineered to make that happen (Hub Pages, Wikipedia, etc). But sources of tested information (NHS, NIH) and of commercial information (product details, sales, etc) can suffer in the face of unqualified, biased or defamatory comment. That doesn’t mean that Sidewiki is always useless, but does mean that the philosophy needs to be exposed as having a benefit to the site owner, many of whom will see nothing but risk when they look at the tool. Another case of Google mistaking technology for product – as with Wave.

And the blog announcement itself doesn’t do a lot to encourage confidence. What’s the first comment? A series of spammy links. How long would it be before some twit starts dropping spammy Sidewiki comments on sites, in order to benefit from their PageRank? Given the rest of the mythology surrounding SEO, it won’t be long after Sidewiki comments are seen, that there’s be someone selling Sidewiki spamming as a service, off the back of user ignorance that the Sidewiki domain is not that of the website…

The standard problems are somewhat less illuminating – banned sites form the majority of these, and the problems are usually associated with link spamming. Our experience of banned sites is usually a call from the new buyer of an established domain, who suddenly finds his site has been banned, as the previous owner boosted the site’s rank prior to sale with mass purchases of undeclared paid backlinks. Dealing with this is usually a matter of removing undeclared backlinks where possible and reworking the site’s paths to avoid URLs referenced by the spam, while designing and developing white hat link building strategies to avoid future problems. That resolution can be more complicated if the site has a mix of high quality backlinks and spammy backlinks to the same page…

The major class of problems that delivers serious insights are the things that web server developers do, and that CMS designers do. Microsoft’s IIS has been an especially rich source of insight. Previous articles on this blog have looked at ranking problems with case folding (case insensitive path names) and with IIS’s cookieless mode user tracking. In both these cases, resolving the problems correctly showed Google’s characteristic times to correct the index after crawling.

More recently, we’ve been getting stuck into some problems with the ways that Content Management Systems handle paths and page contents. Again, much like the IIS technical exercises, these page-template and CMS-specific issues create signatures on the site, that when corrected, tend to be corrected across the entire site at the same time. With a reasonably popular site, it is then possible to predict Googlebot’s significant re-crawl of content, and the delay between that re-crawl and the increase ranking of existing content.

Of course, re-ranking pages allows another set of tests that one couldn’t normally do on a user site. Clients are usually focused on getting back into the listings. With suitable corrections, the site can not only get back, but start to rank for keywords that were previously untouched. This exposes a new audience segment to the sites’ content. If the content is not addressed during the re-ranking exercise, then we sometimes get to see what happens when poor page content is exposed to the firehose of a major volume keyword – allowing the investigation of page content on user behaviour and the subsequent inferred response by Google when the content is judged inadequate by user behaviour.

Finally, one can occasionally find an example of a single off-site link that materially changes how the site ranks. This is usually discovered on small business sites, where some combination of offline connections or a lucky piece of PR manages to snag a high quality backlink to an inner page, with keyword loaded anchor text. The business owner may be puzzled by why one page ranks highly on a specific keyword, and the rest of the site is barely present. Identifying the link, or handful of links, involved and showing the business owner how much effect a high quality backlink can have, is usually an enjoyable experience.

These are valid ways to conduct scientific testing of ranking factors. Although it is much easier, and more predictable and repeatable to conduct tests on low search-volume keywords, Google’s behaviour is necessarily different on low volume and high volume searches. Much of SEO lore depends on testing on low search volume keywords. This gives undue weight to factors that are insignificant, or of low impact, on higher volume searches. It’s only technical failures on high volume sites that can yield the discovery of the importance of ranking factors for competitive terms. And since you can’t, in good conscience, go to the owner of a high ranking site and ask to sacrifice traffic for the sake of an experiment, technical failures on these sites tend to be the best way to collect the data – albeit sporadically.

There is an entire sub-set of scientific method associated with looking at rare examples of events that illuminate topics that are very difficult to experiment on. For example, almost anything involving a rare species of organism is non-destructive and observational, only. Much of astronomy depends on a rare event being observed, such as the impact of Shoemaker-Levy 9 on Jupiter. Most science classes teach the basic methods of science, which are easiest to demonstrate in class with repeatable experiments. However, this overemphasises the importance of reproducibility – observation of rare events is still a crucial scientific technique, albeit harder to properly explore in a science lesson in school. This is probably, also, one of the reasons that evolution is regarded with suspicion – the main investigative technique for evolution, as for astronomy, is observation, not reproducible test; when your understanding of science is driven from being able to reproduce tests, then observational science is downrated in importance and treated with suspicion.

In the SEO world, a reliance on reproducible testing will, I think, tend to overemphasise easily repeated techniques that work for low search volumes, at the expense of optimisation techniques that are more effective for highly competitive and higher search volume keywords (harder to repeatedly test).

Notice that recent articles are sometimes shown twice or more, with different URLs

Why are the articles shown twice or more? Because they have different URLs. They are the same page content, but on different paths. Duplicates. Notice that Matt’s blog is not penalised for these duplicates – because they aren’t what Google considers to be duplicate content. They are the same content, reached by different paths, and in this case the different paths are tracking parameters for Google Analytics. It’d be more than a little annoying if Google handled “duplicate” pages, caused by using their own tracking parameters, as if they were different resources!

Why aren’t other articles on the blog shown twice or more times? What magic makes it that two day old articles get three showings, but articles five days old and older show just once?

Interesting, isn’t it?

Now, instead of sorting by date order, sort by relevance. Let’s look for those duplications again:

The duplicate pages, with different URLs appear lowest for relevance.

So these duplicate pages of the last few days, appear as the lowest relevance pages in the last year of articles on Matt’s site. Yet they have the same text as the main article… or do they? Might there be something magic that the cache shows us?

The cached versions show us that they were taken up to a week previously – and reveal that these were found from Feedburner. Google is taking different URLs, determined by the tracking tags, and including them in results. The weight for the tagged pages is lower – Matt’s blog doesn’t refer to these URLs, and nor does anything else that Googlebot will crawl. So these additional pages rank lower, despite having the same content – the difference lies in the backlinks, and perhaps other factors.

But if there are no external backlinks to these articles, what makes them appear in results, at all, in any position? This, I think, is where we drag in another factor that seems to apply, especially to blogs. Trust. If a blog is well trusted by Google, then articles posted will tend to rank highly in search results, in as little as a few minutes for highly relevant searches (relevant, that is, to what Google thinks the article is about, which mostly means the title). Amongst the two hundred-or-so factors that Google is looking at, is whether they trust your results are likely to satisfy searchers. If your articles have enough hits from satisfied (long reading) users, you leap up the rankings within minutes of posting. Less trusted, lower weight postings won’t appear on page one, if they appear at all. That trust is extended to articles with no backlinks – these pages appear in results because they’re coming from a trusted site.

Why do these duplicate pages disappear, and when, and what does that tell us?

At some point in the last week, Google has removed the extra results for other page references with additional parameters. If we look again next week, we’ll see that the extra results for the current recent articles have also disappeared, but if Matt makes more posts, we may see those articles with extra tracking parameters for a brief period. Why do these “duplicate” pages disappear?

Well, one reason is that Google detests spam. A bad page of search results would be a page that contained different sites with the same article – because that wouldn’t reflect the diversity of opinion and solution on the web. Google would prefer to have ten different answers to the search query, than one answer on ten highly ranked sites. There’s a conscious effort at Google to compare articles.

How does Google identify that? Well, we can take some guesses by looking at the search results. Notice that bit that says “cached”? Taking several shots of a page (cached pages), over time, lets Google see that page content is evolving. User generated content accretes to a page – and that’s visible in successive cached snaphots.

Google can therefore see which parts of a page are static and which are likely to be UGC, even without any effort to understand the page structure. Matts’ articles get a lot of commentary, so it’d be pretty easy for Google to determine that a common core of the page is unchanging – and identical. How easy? Well, there have been tools to perform that kind of textual analysis with computers since at least the 1970′s to my certain knowledge – I used them back then. More recent techniques have used data compression techniques to compare compression rates of samples, using the highly optimised algorithms for good data compression – and Google has smart people who probably do more complex stuff than that.

Canonical Link References

One more datum to collect! Canonical Link References – does Matt’s blog use the canonical link ref to make sure that Google knows the best URL for this article?

The answer is very much affirmative – Matt so much likes the canonical link reference, it is added twice in the header! Hmm – some problem with conflicting plugins, perhaps? I don’t think this duplication of the canonical link reference is intentional. Fortunately they are the same, or we’d get into some discussions about whether Google believes the first or second canonical link reference!

What *should* the canonical link reference do for Google, when we see the variant forms with tracking tags? It should tell Google that the preferred form is the one without the tracking tags. We should end up with just the preferred form showing in search results. And that’s what we see – but the canonical link reference isn’t the only way that Google looks for probably duplicated data. You can tell, if you look for other blogs that use FeedBurner, but that don’t use canonical link references. Those blogs still get deduplicated listings – showing that other components to deduplicate are still working, but still take days to do so.

What does the transitory presence of these pages tell us?

Thinking solely about what Matt’s results are showing us – not taking any evidence from other experiments and tests into account…

I think the presence of these pages in the search results says that Google is reading FeedBurner feeds for Matt’s blog, and getting tagged data, tagged for Google Analytics. I’m guessing that the tags are not set as ignored in Matt’s Webmaster Console, so Google sees the tagged pages as different. Because users are not linking to them, and the blog itself doesn’t refer to them, the usage eventually dies out – where “eventually” means less than a week. And the canonical link reference probably also helps.

Note that the alternate pages *are* present. Not highly ranked, but present. So… the content is the same, the server is the same, but the links to these pages are different; the untagged page is referenced within the blog (by any category or blog tag and the archive), and after a day or so, there are probably some links to these articles, all probably pointing to the untagged page. So that tells us that backlinks are important – or there’d be no reason why these tagged pages shouldn’t rank as highly – the user experience is likely to be the same, the content was (at the time of first snapshot/cache) the same. So if backlinks within Matt’s blog weren’t important, then the pages should rank with equal weight… and they clearly don’t.

This observation also says something important to us – if we watch our web server log files carefully! You may have read some of my other articles here about web server log file analysis for search engine optimisation – I think the web server log files tell us important things about how Google perceives our sites. If we see Googlebot requesting a tagged resource, then that tells us something about what resources we have out there, our state of canonicalisation, and Google’s speed of change. We want to know the speed of change, because when we’re trying to improve performance, we’ll want to see when we might expect to see results, at earliest.

We don’t, unfortunately, have access to Matt’s web server log files… but you have access to yours… What do they tell you, under similar conditions? I may return to this topic, as I think the research is pretty interesting for what it tells us about how Google goes about making decisions.

Summary

Google sees the same blog article under different URLs over time. Google collapses the references to a single page URL – probably using intrinsic information (page content) and supplied information (canonical link references), and backlinks, and the Webmaster Tools mechanism to instruct Google to ignore certain parameters.

URLs for alternate page presentations get into the results for trusted sites, even if these URLs have no weighty backlinks. Not highly ranked, but they do appear. They appear quickly and disappear within a few days. This suggests that whatever it is at Google that evaluates, takes a few days to do so.

I don’t think this points to content being the key, but to content with backlinks and user-preferred results, as being the most important. And the presence of these duplicates is a hint that a site is trusted. Not a great hint, as if you don’t use FeedBurner or another similar tagging resource, you won’t see this effect.

I’ve not considered the backlinks to Matts blog in detail… there are a lot and the signal is messy. I’ve left that analysis out of this article. Besides, discovering the timeline of backlinks is itself a pretty tricky exercise, unless you, as the experimenter, control them; that’s definitely not the case for Matt’s blog!

What’s especially interesting is that an article, without significant backlinks to it yet, can appear high in search results, but that this tends to be for the canonical representation of the page, even though Google is probably learning of the page via FeedBurner/pingomatic notification (IOW, likely to be tagged). So the first mechanism that notifies Google of the URL, is probably *NOT* the canonical form – yet the canonical form is listed highest, within minutes. That suggests some fast processes at Google for identifying and evaluating a page’s relevance, and then some slower processes to determine whether the page can be justified for continuing presence.

And that behaviour of high ranking without external justification, in turn, has some implications for the weight that will flow from a blog article… It is initially likely to be low, and if the article “sticks” in the results because it helps users, then it gains some kind of value. Otherwise, the weight of the page will remain low – though, strictly, understanding that evolution of the weight of the article requires looking at the impact of links from articles in a blog. Another day, perhaps ;)

I hope you found this close look at search results amusing, if not educational.

Apart from the validation of Matt Cutts statements, there’s one result that appears blindingly obvious. Malformed URLs don’t pass anchor text – and by implication, weight. In the context of the example in the article, adding a space to a URL in the anchor, destroys the value. Googlebot changes spaces (which aren’t valid characters in a URL) into “%20″ symbols. In Dave Naylor’s article, that means that the Googlebot will do a DNS lookup for a domain that doesn’t and can’t exist – spaces are not allowed in domain names. If the URL in the anchor’s href had been a fully pathed URL, then a space would be added to the end and converted to a “%20″.

That full URL, with an appended “%20″ won’t be found on the site. It should appear, at some point, in web server log files as a 404 for a Googlebot visit. 404′s don’t pass weight. So why the surprise that a malformed URL would fail?

I think the real point, not cleanly spelled out in the article, is that web browsers don’t parse pages the way that search engine spiders parse pages. A browser will cope with the embedded space. That ability of a browser to infer the useful thing to do, doesn’t make the space into a valid character in URLs – not without being escaped, anyway. And the consequence of appending the space, will be that a web spider makes a request for a resource that will usually be 404′ed, unless the administrator has used Apache mod_speling or an equivalent typo-correction tool (which should yield a 301 redirect to the correct resource).

Attempting to infer the SEO value of browser interpreted behaviour, without understanding Googlebot behaviour, will create confusing and misleading problems.

I’ve fairly recently had discussions with a few web site managers who’d made what appears to me to be the most peculiar decision – to block Googlebot because of the traffic impact. This resonated with a previous short article that I’d posted, about a problem identified by a Google staffer who was running his own blog. He’d seen his blog dropped from search results and was looking for why that might be happening.

There’s certainly a potential problem – low bandwidth sites may suffer if Googlebot consumes the available bandwidth. But if you don’t have Googlebot crawling, then how are you going to appear, anyway?

You could use the Webmaster Tools to request that Google slows the crawl for your site. This should still result in having the crawling and indexing, and minimal damage to the traffic. But just disabling the crawl, by using robots.txt to block all crawling, or to block crawling of large sections of the site that should have user interaction, is probably a mistake.

There is also the legitimate concern that Googlebot’s visits might be draining server resources at peak traffic periods. That’s moderately difficult for non-technical site owners to work out. Google Analytics (and the other JavaScript page bug based web analytics packages, such as CoreMetrics, Omniture, Webtrends, etc) measure user visits, not Googlebot and other bot visits. Verifying that Googlebot isn’t interfering with and slowing down visitors, is pretty much impossible to understand without going to web server log file analysis.

Web Server Log File Analysis

I like web server log files. There’s things I can find out from them, in a few hours, that I simply can’t find from Google Analytics, CoreMetrics and Omniture. Look at this graph, for example. I’ve taken web server log files from a UK-targeted business, and extracted Google-inspired visits and Googlebot visits, by hour.

The graph shows that Googlebot is busiest when users are less present. That is, when Google can see visitors coming to the site, the crawl volume is reduced.

This pattern of making Googlebot most active when the site visits are least active, seems to be the most common pattern that I can see in clients’ web server logfiles. It makes a lot of sense for Google, too:

• Continuing visits by Googlebot allow them to check that the site is still working (preventing Google from delivering users to a 404′ed page)
• Site performance under load can be monitored (helping Googlebot tune crawling rates, and verifying that users are getting responses from the site, mostly)

Summary

Googlebot seems to be quite smart about when it visits sites. The more users that are being sent to a site in a given hour, the relatively lower rate that it crawls. So Googlebot should never get in the way of visitors, under normal conditions.

Simply disabling Googlebot looks like a weak way to go.

Following suggestions from the Google Webmaster Blog, if you have areas of the website that change at different speeds, you might want to validate multiple webmaster consoles for different sections of the site. That would allow setting different crawl rates. I’ve not tried this, yet… I don’t have a client for whom I want to restrict crawling speeds!

The searches are also amusing, because this blog has a comment policy. Searches looking for text indicating that this is a blog that accepts user generated content, will tend to find that policy page. In turn, that means that I get a lot of flattering comments about the article… posted to the comment policy. Bit of a tell that the commenter isn’t really engaged.

What’s also interesting is where people are coming from. Mostly China, India, the US.

So, is there anything that can be done, apart from using Akismet and other despamming tools, to help reduce blogspam? Might be worth changing the text, or possibly replacing the text with a graphic, to indicate where a comment should be left. That’d reduce the search volume coming to the blog from people intent on spamming. However, that’s essentially a defence based on security through obscurity – I’m going to experiment with changing that text and see what it does to searches leading to this blog. :)

I will, of course, continue to use the comment policy page as a great detector of spam. All those people telling me how great the article is, submitting to the comment policy page, have completely signalled their lack of interest in contributing! Like all these spammy comments, mostly targeting a few choice pages:

That sample is moderately representative – about half the spam submitted, is submitted to the comment policy page.

How Does Google Rank Results

I don’t know all the details. Rand doesn’t know all the details. Some guys at Google know a lot of the factors. Matt Cutts, Google’s head of the search quality team, claims over 200 factors go into ranking.

What we do know is that backlinks – credible links regarded by Google as likely for a search user to visit – are important. We know that anchor text is important. There’s some other factors that we know influence Google ranking.

What *else* do we know?

We (professional search engine optimisation people) know that on-page content is valuable. For low competition keywords – keywords where there aren’t a lot of links and anchor text, and hardly anyone searches – then page content is enough. Look at the example in the graphic below. There’s precisely one page on the internet, with that text for something that I can’t find on Google. When I wrote that, it was true; if you search now, you’ll find that page. Well, until some spoiler copies it elsewhere…

However, try putting the word “bad credit loan” on a page on a new web site with some other relevant and unique content, valuable to a user, and see how high you rank for the term. You can wait. And wait. And wait. You’re not going to show up on the first page of results, just by having a great page alone. It’s not just the content, it’s the backlinks that make the difference.

So we now know, as a result of this test, that while Google does pay attention to on-page factors, they also pay attention to backlinks. And in competitive spaces, *effective* backlinks count for more than the page content.

The important message to understand from this is that different factors apply under different conditions. Content alone won’t put you on page one. Backlinks alone won’t keep you there.

Click Through Rate and Bounce Rate

So, at some scale, do CTR (Click Through Rate) and Bounce Rate make any difference? I believe they do, and this blog is a testament to that. Look at this screenshot.

That’s a Google Analytics shot of the last 15 months activity for a specific page on the Merjis blog. It’s all about “gclid” – something you’ll probably care about if you do paid search and look in web server logfiles.

I’m using this blog as an example, because I’ve been using it for tests for years – I know how it works, and it isn’t confidential client data. I can reveal the usage, because I have my own reasons for running a blog, and few of them directly have anything to with making money.

Most other pages on this site get a profile like this other example:

This is pretty typical for a “newsy” blog article. Usage on the day that it is written, and a dribble thereafter. It then usually dries up after a few weeks, because the rank has decayed with time.

So why, with a higher bounce rate, does the older article do better than the newer article in rankings? If Bounce Rate is important, then surely the lower bounce rate in a newer article must mean that Google should drop the older article?

I suspect that Google doesn’t have a rigid number. They look at how well you do relative to other sites. And especially, they look to see whether search users search again for the same or very similar searches. Read that article on SideWiki, and it’s lightweight. No real information. No real recommendations. The long lived article on gclid has a much higher bounce rate and longer reading time. It’s the reading time that’s the clue. When you’ve read my article on gclid, you probably don’t want to read another article about gclid. It’s reasonably definitive.

Google sustains that old article in search results, despite its’ great age, and despite a high bounce rate, because those users who do read it, value it. It’s there, because it helps Google to deliver a page of search results that users value more than *without* that article present.

Uh – You Didn’t Mention CTR

Again, I don’t think it is actually CTR that Google is looking for. It is user satisfaction. So a high CTR, caused by a misleading piece of copy, won’t help. You have to deliver what you offer. Again, I don’t think that Google is measuring conversion, either. But a high CTR message with a high conversion rate, meaning that users are highly satisfied – that’s what Google wants you to make.

You won’t be directly rewarded for high CTR – but you can measure it (especially if you also run PPC and can get the impression rate). You won’t be rewarded directly by Google for high conversion rates. But Google does appear to prefer sites that answer the question posed by the search query. And the proxy that can be used by Webmasters, who don’t have access to Google’s richer data, is their own performance, as CTR and Conversion Rate. Increase those, and you are more likely to increase position.

Interaction of Factors

If you have a good site, with highly relevant content, you tend to get more links. So disentangling backlinks, and the immeasurable relative user satisfaction, is difficult. Pretty much the only way that I know it can be done, is when you have web sites with accidental misbehaviours that create the right conditions for a test. The technical problems that create the conditions are rare – and recreating them in a real website is likely to decrease the performance. It’s unlikely that anyone will give you the opportunity to mess up their site, just to prove what works.

However, if you want to go about it… Here’s what I think you’ll need:

• A visibly horrible page, with a low conversion – as your starting point
• Weak Title and Meta Description as a starting point
• A lot of visitors per day – it takes a long time to demonstrate, otherwise
• The ability to make sitewide link changes to the page under test
• Good backlinks – you’ll want to know that you *could* rank well on page one

Change the URL for your horrible page, sitewide. Wait for Google to find it and rank it again. Note the position. Watch the position fall over a period of a week or two (depending on visitor volume). Now improve the page, and switch the URL again and wait for Google to find and rank it. Then watch the rankings change and note which way they go. Now revert the page and switch URLs again, and this time change the Title and Meta Description. Now watch the ranking changes. Now fix up the page again and once more switch the URL and watch.

You should, IME, find that you achieve a higher long term position when you have a better title and description, and a higher converting page with a lower bounce rate. If you can explain why you *shouldn’t* get a higher position with a site that is better for users, I’d love to know the reasons. But don’t make your explanation involve “gaming” the system.

And, FWIW, I don’t believe that the Title and Description are important, as direct factors for SEO. You can rank perfectly well for keyword free pointless titles, and descriptions without keywords that are positively turgid and rambling. However, show the user that you are focused on solving their problem, and your CTR increases; and if you are focused on the user, you’ll probably have a reasonable landing page, which will engage and convert better. Google’s not going to reward you for a better snippet, directly, but for a better user experience. Your only measures though, will be what you can observe – CTR, Bounces, Conversions. If I could tell you to look at the “re-query rate”, I’d tell you to do so – instead, you’ll have to use the information you can get.

Implications For SEO

If a blog article can decay to little traffic in a few weeks, or sustain rankings for years, on the same blog, with the same blogging software, then the difference must be backlinks? Well, not substantially. Over the years, I’ve had more backlinks to newsy stories, but still this “gclid” article keeps ranking. And all the time, the other lighter weight articles just keep falling out of the listings.

There’s a few other similar articles on this blog that rank, and stay high for years and years and years. Non-competitive searches, but of long lasting traffic value. And the other sites that I’m competing with, for attention, are large forums. High weight. Much more frequently updated content. I’m *deliberately* not trying to place links for articles. Just letting what happens, happen – so I can understand why it happens. So there’s no contamination effects here with deliberate link placements.

What are the articles? They all tend to be like that gclid article. Something that is detailed, informative, and means that you can go away and do something. Useful articles, in other words. Harder to write than “straight news” articles, as you need unique content, written to address the audience. That’s part of my reason for writing – attempting to develop clearer communication.

The clear implication is, I think, that useful content matters. And how do we know it is useful? It’ll show up in search engine rankings, usablility data and other disturbingly hidden and arcane resource. Google will reward useful content with a better sustained rank – but won’t put you on page one just because you have a great article, unless you have some backlinks to create credibility.

But How?

Rand makes the point that data about use can be gamed. But so can backlinks. That’s the major part of undeclared paid backlinks, small world building, and other “black hat” techniques. We know that Google sees through most black hat techniques, given time.

We also know, or can find out about, Google’s interest in invalid impressions and invalid clicks. For example, invalid impressions are generated when search engine ranking tools are run – they reduce the effective CTR. Invalid clicks are generated when users double click, or are paid to click. Just as with paid search, these two types of invalid activity are measurable by Google. In fact, Google can measure a lot more than a webmaster can see.

We webmasters only get to see bounce rates and conversions. Google gets to look at whether users search again. Much more valuable. If you want to build the worlds’ best search engine, then you want to feature the results that tell you that you’ve got a winning page – pages where users don’t need to search any more. Results that have users positively selecting that site again, when they see it in listings. Webmasters just don’t have that detail, directly. We just don’t know if the other guy answers better – unless we expend effort to learn our customers’ minds and make sure we have the best answer.

User Experience

Google’s Ten Things lists, first, “Focus on the user”. The results from this blog, and from other client activities that I’m not going to reveal in any detail, are fairly clear. Content that Google can measure as being liked by users, rank better and longer than content that is spammy, tedious and weak. The factors that lead to better rankings will include appropriate Titles and Descriptions and engaging content. It has to be, or rule 1 is broken.

We know that Google has experience of measuring impressions and data to look for invalid data. We know that Google is pretty good at it – or there’d be more click fraud problems with AdWords. So, if it can be done, and it is an important indication of quality, why wouldn’t Google use searchers behaviour to modify results, not just personally, but across the index?

Why can’t you improve the results when you click on your own listings? Because it is identical behaviour to the banned AdSense practice of clicking on adverts on your own site. Detectable. Invalid. Not counted. And for reasons that I don’t want to go into, I believe the same will be true of botnets and eLance and Mechanical Turk attacks. There will be a signature associated with them, that doesn’t match normal user behaviour. The signatures can be spotted and countered, by assigning the activities as invalid – just as it is in AdWords. Since AdWords continues to run without being infested with click fraud to unusable levels, we have a working system, on a global scale, that shows that user behaviour can be extracted from noisy fraudulent behaviour.

It isn’t perfect, true, but it separates AdWords from being a system that solely acts to transfer advertising funds to thieves, into a system that, more often than not delivers prospective buyers to an advertiser’s site. It isn’t perfect, but it works well enough. AdWords only works because it identifies and categorises user behaviour.

User behaviour categorisation works in one system that Google has, worldwide, on a service with measurable economic value. Why wouldn’t it be usable in organic search results?

Conclusions

Failing to identify and understand user interests is an SEO mistake. These are reflected by (but are not completely explained by) CTR and Bounce Rates – because that’s about the best that Webmasters can get. Google doesn’t have to use those – they have better numbers that are more meaningful to user experience. But saying that “Google doesn’t use bounce rates” is not the same as saying “Google doesn’t take account of user behaviour”.

Unlike Rand, I believe that Google cares very deeply about the user experience, and that Google has very sophisticated technology, probably shared with the Google AdWords guys, to identify unusual search behaviours and exclude them from consideration.

Given enough data, probably gained from multivariate testing on all the different data centres, Google can identify whether users are more, or less, satisfied by different ordering in search results than a pure backlinks-plus-content model would give.

Small scale tests probably won’t show anything about user interaction – because the activity doesn’t have statistical significance or because the signature of strange search activity is too obvious. So, don’t try faking it – if you’ve read this far, you probably aren’t smart enough to outwit Google’s teams of click-fraud defence guys. They are really pretty good, as anyone with a rational assessment of AdWords click fraud levels will tell you. Not perfect, but good enough to make the effort of using AdWords worthwhile, rather than primarily a way of siphoning your advertising funds to fraudsters. :)

Why do I say “if you’ve read this far”? Because if you really knew how to hide click streams, you’d be doing it with AdSense. And you’d have stopped reading at that point – because you own the game already. If you can’t own that game, you can’t own the game of spoofing user behaviour in organic search – it is (not identical to, but close enough to) the same game. At the moment I don’t understand why you’d bother with SEO behavioural spoofing, if you’d gamed AdSense, because the revenue is a lot more direct… Maybe that’s why Rand hasn’t spoken with any black hatters that have cracked it?

And if Google can detect unusual impression and click data, then they can fulfil their primary mission, with respect to modifying organic rank based on real user data about preferences and satisfaction.