Notice that recent articles are sometimes shown twice or more, with different URLs

Why are the articles shown twice or more? Because they have different URLs. They are the same page content, but on different paths. Duplicates. Notice that Matt’s blog is not penalised for these duplicates – because they aren’t what Google considers to be duplicate content. They are the same content, reached by different paths, and in this case the different paths are tracking parameters for Google Analytics. It’d be more than a little annoying if Google handled “duplicate” pages, caused by using their own tracking parameters, as if they were different resources!

Why aren’t other articles on the blog shown twice or more times? What magic makes it that two day old articles get three showings, but articles five days old and older show just once?

Interesting, isn’t it?

Now, instead of sorting by date order, sort by relevance. Let’s look for those duplications again:

The duplicate pages, with different URLs appear lowest for relevance.

So these duplicate pages of the last few days, appear as the lowest relevance pages in the last year of articles on Matt’s site. Yet they have the same text as the main article… or do they? Might there be something magic that the cache shows us?

The cached versions show us that they were taken up to a week previously – and reveal that these were found from Feedburner. Google is taking different URLs, determined by the tracking tags, and including them in results. The weight for the tagged pages is lower – Matt’s blog doesn’t refer to these URLs, and nor does anything else that Googlebot will crawl. So these additional pages rank lower, despite having the same content – the difference lies in the backlinks, and perhaps other factors.

But if there are no external backlinks to these articles, what makes them appear in results, at all, in any position? This, I think, is where we drag in another factor that seems to apply, especially to blogs. Trust. If a blog is well trusted by Google, then articles posted will tend to rank highly in search results, in as little as a few minutes for highly relevant searches (relevant, that is, to what Google thinks the article is about, which mostly means the title). Amongst the two hundred-or-so factors that Google is looking at, is whether they trust your results are likely to satisfy searchers. If your articles have enough hits from satisfied (long reading) users, you leap up the rankings within minutes of posting. Less trusted, lower weight postings won’t appear on page one, if they appear at all. That trust is extended to articles with no backlinks – these pages appear in results because they’re coming from a trusted site.

Why do these duplicate pages disappear, and when, and what does that tell us?

At some point in the last week, Google has removed the extra results for other page references with additional parameters. If we look again next week, we’ll see that the extra results for the current recent articles have also disappeared, but if Matt makes more posts, we may see those articles with extra tracking parameters for a brief period. Why do these “duplicate” pages disappear?

Well, one reason is that Google detests spam. A bad page of search results would be a page that contained different sites with the same article – because that wouldn’t reflect the diversity of opinion and solution on the web. Google would prefer to have ten different answers to the search query, than one answer on ten highly ranked sites. There’s a conscious effort at Google to compare articles.

How does Google identify that? Well, we can take some guesses by looking at the search results. Notice that bit that says “cached”? Taking several shots of a page (cached pages), over time, lets Google see that page content is evolving. User generated content accretes to a page – and that’s visible in successive cached snaphots.

Google can therefore see which parts of a page are static and which are likely to be UGC, even without any effort to understand the page structure. Matts’ articles get a lot of commentary, so it’d be pretty easy for Google to determine that a common core of the page is unchanging – and identical. How easy? Well, there have been tools to perform that kind of textual analysis with computers since at least the 1970′s to my certain knowledge – I used them back then. More recent techniques have used data compression techniques to compare compression rates of samples, using the highly optimised algorithms for good data compression – and Google has smart people who probably do more complex stuff than that.

Canonical Link References

One more datum to collect! Canonical Link References – does Matt’s blog use the canonical link ref to make sure that Google knows the best URL for this article?

The answer is very much affirmative – Matt so much likes the canonical link reference, it is added twice in the header! Hmm – some problem with conflicting plugins, perhaps? I don’t think this duplication of the canonical link reference is intentional. Fortunately they are the same, or we’d get into some discussions about whether Google believes the first or second canonical link reference!

What *should* the canonical link reference do for Google, when we see the variant forms with tracking tags? It should tell Google that the preferred form is the one without the tracking tags. We should end up with just the preferred form showing in search results. And that’s what we see – but the canonical link reference isn’t the only way that Google looks for probably duplicated data. You can tell, if you look for other blogs that use FeedBurner, but that don’t use canonical link references. Those blogs still get deduplicated listings – showing that other components to deduplicate are still working, but still take days to do so.

What does the transitory presence of these pages tell us?

Thinking solely about what Matt’s results are showing us – not taking any evidence from other experiments and tests into account…

I think the presence of these pages in the search results says that Google is reading FeedBurner feeds for Matt’s blog, and getting tagged data, tagged for Google Analytics. I’m guessing that the tags are not set as ignored in Matt’s Webmaster Console, so Google sees the tagged pages as different. Because users are not linking to them, and the blog itself doesn’t refer to them, the usage eventually dies out – where “eventually” means less than a week. And the canonical link reference probably also helps.

Note that the alternate pages *are* present. Not highly ranked, but present. So… the content is the same, the server is the same, but the links to these pages are different; the untagged page is referenced within the blog (by any category or blog tag and the archive), and after a day or so, there are probably some links to these articles, all probably pointing to the untagged page. So that tells us that backlinks are important – or there’d be no reason why these tagged pages shouldn’t rank as highly – the user experience is likely to be the same, the content was (at the time of first snapshot/cache) the same. So if backlinks within Matt’s blog weren’t important, then the pages should rank with equal weight… and they clearly don’t.

This observation also says something important to us – if we watch our web server log files carefully! You may have read some of my other articles here about web server log file analysis for search engine optimisation – I think the web server log files tell us important things about how Google perceives our sites. If we see Googlebot requesting a tagged resource, then that tells us something about what resources we have out there, our state of canonicalisation, and Google’s speed of change. We want to know the speed of change, because when we’re trying to improve performance, we’ll want to see when we might expect to see results, at earliest.

We don’t, unfortunately, have access to Matt’s web server log files… but you have access to yours… What do they tell you, under similar conditions? I may return to this topic, as I think the research is pretty interesting for what it tells us about how Google goes about making decisions.

Summary

Google sees the same blog article under different URLs over time. Google collapses the references to a single page URL – probably using intrinsic information (page content) and supplied information (canonical link references), and backlinks, and the Webmaster Tools mechanism to instruct Google to ignore certain parameters.

URLs for alternate page presentations get into the results for trusted sites, even if these URLs have no weighty backlinks. Not highly ranked, but they do appear. They appear quickly and disappear within a few days. This suggests that whatever it is at Google that evaluates, takes a few days to do so.

I don’t think this points to content being the key, but to content with backlinks and user-preferred results, as being the most important. And the presence of these duplicates is a hint that a site is trusted. Not a great hint, as if you don’t use FeedBurner or another similar tagging resource, you won’t see this effect.

I’ve not considered the backlinks to Matts blog in detail… there are a lot and the signal is messy. I’ve left that analysis out of this article. Besides, discovering the timeline of backlinks is itself a pretty tricky exercise, unless you, as the experimenter, control them; that’s definitely not the case for Matt’s blog!

What’s especially interesting is that an article, without significant backlinks to it yet, can appear high in search results, but that this tends to be for the canonical representation of the page, even though Google is probably learning of the page via FeedBurner/pingomatic notification (IOW, likely to be tagged). So the first mechanism that notifies Google of the URL, is probably *NOT* the canonical form – yet the canonical form is listed highest, within minutes. That suggests some fast processes at Google for identifying and evaluating a page’s relevance, and then some slower processes to determine whether the page can be justified for continuing presence.

And that behaviour of high ranking without external justification, in turn, has some implications for the weight that will flow from a blog article… It is initially likely to be low, and if the article “sticks” in the results because it helps users, then it gains some kind of value. Otherwise, the weight of the page will remain low – though, strictly, understanding that evolution of the weight of the article requires looking at the impact of links from articles in a blog. Another day, perhaps ;)

I hope you found this close look at search results amusing, if not educational.

Apart from the validation of Matt Cutts statements, there’s one result that appears blindingly obvious. Malformed URLs don’t pass anchor text – and by implication, weight. In the context of the example in the article, adding a space to a URL in the anchor, destroys the value. Googlebot changes spaces (which aren’t valid characters in a URL) into “%20″ symbols. In Dave Naylor’s article, that means that the Googlebot will do a DNS lookup for a domain that doesn’t and can’t exist – spaces are not allowed in domain names. If the URL in the anchor’s href had been a fully pathed URL, then a space would be added to the end and converted to a “%20″.

That full URL, with an appended “%20″ won’t be found on the site. It should appear, at some point, in web server log files as a 404 for a Googlebot visit. 404′s don’t pass weight. So why the surprise that a malformed URL would fail?

I think the real point, not cleanly spelled out in the article, is that web browsers don’t parse pages the way that search engine spiders parse pages. A browser will cope with the embedded space. That ability of a browser to infer the useful thing to do, doesn’t make the space into a valid character in URLs – not without being escaped, anyway. And the consequence of appending the space, will be that a web spider makes a request for a resource that will usually be 404′ed, unless the administrator has used Apache mod_speling or an equivalent typo-correction tool (which should yield a 301 redirect to the correct resource).

Attempting to infer the SEO value of browser interpreted behaviour, without understanding Googlebot behaviour, will create confusing and misleading problems.

I’ve fairly recently had discussions with a few web site managers who’d made what appears to me to be the most peculiar decision – to block Googlebot because of the traffic impact. This resonated with a previous short article that I’d posted, about a problem identified by a Google staffer who was running his own blog. He’d seen his blog dropped from search results and was looking for why that might be happening.

There’s certainly a potential problem – low bandwidth sites may suffer if Googlebot consumes the available bandwidth. But if you don’t have Googlebot crawling, then how are you going to appear, anyway?

You could use the Webmaster Tools to request that Google slows the crawl for your site. This should still result in having the crawling and indexing, and minimal damage to the traffic. But just disabling the crawl, by using robots.txt to block all crawling, or to block crawling of large sections of the site that should have user interaction, is probably a mistake.

There is also the legitimate concern that Googlebot’s visits might be draining server resources at peak traffic periods. That’s moderately difficult for non-technical site owners to work out. Google Analytics (and the other JavaScript page bug based web analytics packages, such as CoreMetrics, Omniture, Webtrends, etc) measure user visits, not Googlebot and other bot visits. Verifying that Googlebot isn’t interfering with and slowing down visitors, is pretty much impossible to understand without going to web server log file analysis.

Web Server Log File Analysis

I like web server log files. There’s things I can find out from them, in a few hours, that I simply can’t find from Google Analytics, CoreMetrics and Omniture. Look at this graph, for example. I’ve taken web server log files from a UK-targeted business, and extracted Google-inspired visits and Googlebot visits, by hour.

The graph shows that Googlebot is busiest when users are less present. That is, when Google can see visitors coming to the site, the crawl volume is reduced.

This pattern of making Googlebot most active when the site visits are least active, seems to be the most common pattern that I can see in clients’ web server logfiles. It makes a lot of sense for Google, too:

• Continuing visits by Googlebot allow them to check that the site is still working (preventing Google from delivering users to a 404′ed page)
• Site performance under load can be monitored (helping Googlebot tune crawling rates, and verifying that users are getting responses from the site, mostly)

Summary

Googlebot seems to be quite smart about when it visits sites. The more users that are being sent to a site in a given hour, the relatively lower rate that it crawls. So Googlebot should never get in the way of visitors, under normal conditions.

Simply disabling Googlebot looks like a weak way to go.

Following suggestions from the Google Webmaster Blog, if you have areas of the website that change at different speeds, you might want to validate multiple webmaster consoles for different sections of the site. That would allow setting different crawl rates. I’ve not tried this, yet… I don’t have a client for whom I want to restrict crawling speeds!

You can see requests like “GET /(J(1)S(4dab…..))/” all over the log files. You can go to Yahoo!Site Explorer and find pages that are indexed with these cookieless mode paths. But I can’t find any of the big 3 robots (Google, Yahoo and MS) crawling those pages. Perhaps the reason is where the cookieless paths are embedded – ASP seems to embed these URLs in form requests.

method=”post” action=”/(Y(1)T(d4ab…”

At the moment my best guess is that something else is crawling, capturing the links and leaving them on a third party site, which the SE Spiders are then finding. At some point I’ll try to find out where that may be. It’s a lower priority than making sure those references point to the right page. A few odd spiders do seem to crawl the IIS cookieless paths – things like the “Majestic12” search engine spider.

Why would I want this listing of strange paths stopped? Because some of the pages that it links to are pages that are SEO targets, and this multiple/duplicate listing dilutes the weight given to the real path. Unless the bots are smart enough to know that this is really known as a shorter path name… But if the spiders are smart enough to avoid crawling them, why are the SE’s dumb enough to list them? Hmm. An interesting puzzle. maybe, (shudder), I’ll have to set up an IIS server and do some experiments.

Case folding continued

On the other hand, I do find spiders crawling case folded file names. Looking for “/default.aspx” and “/Default.aspx”, for example. You can see GoogleBot and Slurp happily grabbing exactly the same file under names that are identical because of IIS case folding.

Even worse, you can see some search queries turning up at both forms. That means that even Google can be confused when the same file is known under multiple names, because link naming is not consistent. Worse yet, while *you* may control linking on your own site, you can’t control it from all external sites.

I’m looking at an ISAPI Rewriter as the only way to deliver the technological structure for these IIS based clients, and to make sure that the right URL is optimised.

Case Folding Surprises

For the last three years, pretty much everyone we’ve worked with for SEO has used Apache. So all the tools we have are built around Apache, implicitly. It’s been quite an exercise to rework them to work with IIS. Intellectually, I’d known that case was important – I’ve been using UNIX systems since 1978 or so, and usually have been involved with Windows systems. I was shocked by the number of inbound links and search query references using a wide variety of case changes.

It’s made me even more determined to check inbound links and use mod_speling on standard Apache installations – it will easily catch and fix case problems.

I’m pretty sure that this wouldn’t be a problem if IIS was built on a case preserving filesystem. I’ve been backwards and forwards over the Microsoft SEO presentations on their blog and I can’t see any real reference ti this issue and how to resolve it. Perhaps MSN doesn’t suffer from it, so they don’t think they have to care? Maybe I glossed over a presentation or article that does explain the issue and how to resolve it. It seems like a pretty easy way to do some Google Bowling :(

Summary

Still no firm conclusions. IIS Cookieless mode is worrying, but I can’t find any evidence that it affects the Big 3.

IIS (and probably Mac OS X) case folding is a problem, at least for Google. It looks as though there may be different page ranks involved for different case variations of the same file. Maintaining internal link naming consistency probably helps, but you can’t stop external sites from linking to case variations and thereby possibly reducing PR from time to time. Either use a case respecting file system (Mac OS X allows that choice), or use one of the ISAPI rewriters to issue a 301 redirect from case variations to the correct file name.

What is the authoritative name of the web page?

Assume that your web site uses IIS and has a home page available as “/index.htm” You can then use the following as silent synonyms of the home page:

• /index.htm
• /Index.htm
• /INDEX.HTM
• /InDeX.hTm

Why is finding the same file under multiple names a problem?

Whether you use a web browser, or a search engine bot crawls your web site, each one of those URLs appears to be a different file, albeit with identical content. If you have optimised the content, then each is a candidate to be the answer for search users. So the spiders do need to track each file name variant, and check them, to see what has changed.

In each case you get an immediate web server response of “200″ – file found. Link love can be spent on a wide variety of paths that lead to the same place – but the spiders aren’t told that. There is another way to do this, which does not work, out of the box, on Linux and Apache, but is fairly easy to set up.

Web Brand Is All About Experience

If the file system does not fold case – that is, it treats upper and lower case letters as two distinct things – then a request for a file with mismatched case delivers a 404 – File Not Found. Now that’s a bad user interface experience. Brand is all about experience, so why punish your users with a 404 because they can’t remember what the capitalisation of your ProDuct (sic) is?

You need to find a way to deliver both the web page that the user wanted to get to, and also let the spiders know that there is one authoritative page – there may be a lot of different links to get there, but just one resource.

On Merjis.com we use a technique that helps the spider understand that we have one page, and that case changes are a link problem, not a server duplication. If you try to reach:

• http://merjis.com/contact
• http://merjis.com/Contact
• http://merjis.com/CoNtAcT
• http://merjis.com/CONTACT

then you should get to the same page, the Merjis contact information page. In terms of the user experience, this is just the same user experience as on IIS. But we issue a redirect on all the non-standard forms of the page name and IIS doesn’t. Spiders can see that only one page for “contact” exists on the Merjis site, even if it is accessible through many different URLs. This cuts down redundant crawls, focuses link love on a single page, doesn’t lose references from typographically challenged links, gets users to the page they want whatever the case of the URL they type, and is generally A Good Thing.

How To See A Redirect

If you don’t use a tool like “wget” or one of the Firefox HTTP inspection tools, your only real clue to our redirection is that whatever you did type in the URL bar, is replaced by our chosen URL for the resource. Between your input and our response, we added a redirect. Spiders will see the redirection, and only index one page and can pour all the link love on that page.

That’s completely unlike IIS standard behaviour. The default behaviour is to fold uppercase and lowercase. That means you see the URL that you typed. There’s no information that the file is a single file known by many names.

Spiders can’t guess that a single file is a single file – they only know what they are told. They get told what links exist in sitemaps and by other link references across the web. If a spidered site has references to mixed case versions of names, then the spiders will tromp madly off to each alternative case version of exactly the same file.

SEO Means Never Saying Sorry To Stupid Spiders

I’m of the opinion that helping the spiders to find the right information, helps SEO. Sending spiders to a dozen spelling variations of a path, doesn’t boost rank, unless the spiders are clever. Even without that, sending spiders to crawl redundant pages, when they could more frequently crawl real content, is a waste of the attention from search engines. If spiders were clever, they wouldn’t crawl redundant paths to the same content, repeatedly, across the whole server… So give them a hand to get to the right single file that should be taking all the page rank.

This default case folding behaviour means that IIS again contributes to spamming search engine indexes. Not so bad, except that it causes yet another problem.

My Web Analytics Don’t Fold Case

When you are trying to analyse what is happening to users, just one miskeyed filename can result in the analytics giving you multiple paths to, for example, conversion. Typically the JavaScript page bug reports the filename that was accessed – using the same capitalisation that the web server delivered. Why? Because on a large fraction of the other web servers, case does matter and “/index” is a different file from “/INDEX”.

So on an IIS delivered file, the same page can be known by a wide range of names that mean the same thing. But analytics packages don’t (usually) fold case – so each reference to a different capitalisation adds another meaningless node to journey analysis.

[ N.B. See Chris's comment below. I should quantify the assertion that Analytics don't fold case - of course, if any of them *do*, that's another problem... ]

The following deleted section is a bit rubbish. I failed to properly read and understand the robots.txt spec. I interpreted a line that meant all records in the file, to just mean the User-Agent line. robots.txt allows case folding – /MEMBERS and /members are identical according to the later spec; the earlier spec only clearly states that the User-Agent field ignores case – leaving the possibility of ignoring case or respecting it in a Disallow line.

However – I started this article because I found a clients private area on IIS had been crawled and indexed, despite being listed in robots.txt. I still need to describe that investigation – but this article is long enough.

And that’s not all. Oh no…

Google Spiders Content Disallowed In Robots.txt

If you have parts of the site that you don’t want to be in the index, you can use robots.txt to exclude those directories or applications.

Except… you can’t. Not sensibly, not with IIS out of the box.

Say that you have a directory called “/members” and you want only signed in members to see the content. You exclude the spiders with:

Disallow: /members

However… case folding… This directory is also accessible as “/MEMBERS” and that isn’t excluded in this customers’ Robots.txt. So your hidden content is now visible if just one link, somewhere on the internet, or even in your own site, uses a different capitalisation from that which has been put in the Robots.txt file.

Is this Google and Yahoo!’s problem to resolve? IMO, not really. If you choose to use a server that makes the same content available under a range of paths, it is up to you to protect those paths, not for spider developers to guess that you may have shot yourself in the foot.

OTOH, the SEs do themselves no favours for reducing spam in the indexes, and decreasing crawl volumes and bandwidth usage, by failing to recognise that IIS can serve the same page under a multiplicity of case variations. That’s a different problem – but solving one would solve the other. If the Server can be detected as using case folding then using a case independent match of Robots.txt paths would be a useful extension.

I can even imagine adding a new directive to Robots.txt to express that pathnames do not respect case.

Other Case Folding Systems

Well, Apple OS X. It may be my favoured desktop OS, but it has a default FS that is caseless:


$ echo boo > goose
$ cat GOOSE
boo
$


I haven’t tested – I have no SEO clients with Mac servers – but I suspect that Mac servers with a default FS run the same problem of a futile and useless Robots.txt protection.

IIRC, OS/2 aka “Warp” was used for some years as a web server and it used a case folding FS – so if you run one of those ancestral systems, watch out.

Webmasters

Your defence? Well, make IIS respect case in queries and do a proper redirect to the actual file.

That way, spiders could properly use Robots.txt and your hidden content wouldn’t be accessible. Try asking Microsoft about that configuration. Heh. Here’s Microsoft’s page about creating a Robots.txt – note the discussion about case folding? Oh, there *is* no mention of case folding? Hmm. Well, I think that’s a lesson in its own right.

Failing any rational advice from Microsoft, you could do a combinatorial madness on Robots.txt:


Disallow: /members
Disallow: /Members
Disallow: /mEmbers
...
Disallow: /MEmbers
...
Disallow: /MEMBERS

and so on. It’s easy. Yeah. Right.

Have I mentioned that I think IIS adds problems, rather than removing them?

It is all so much more complex if you have cookieless mode enabled for your ASP applications. Because the path given to the robots doesn’t match the path that is denied. Deny “/secret” and you get a path that starts “/(” and goes on to “))/secret”. Combine that with case folding and there is no end to index spamming.

And, of course, Robots are the main thing that need to read and respect Robots.txt.

Given a modicum of sense, this whole area could be made a lot simpler for system admins and webmasters. If an IP address and user agent doesn’t accept cookies, and asks for “robots.txt”, it is probably a robot. Stop sending cookieless tracking paths to that IP and UA.

Sticking In a Reverse Caching Proxy

If you are a technological sophisticate, then you could insert Apache with a rewriter and mod_speling, to get the benefits of case matching and redirection to the single real file instance. You’ll possibly see a slight average speed up for users, as unchanged content is delivereddirect from the Apache cache.

How to set up and configure one of these cute web servers is beyond the scope of this article, though.

Scale Of Problems

It’s quite nasty for people using hosted IIS, who have no significant control. I have no doubt that the SEs do duplicate detection. Matt Cutts has written that in-site duplication isn’t too awful a penalty – probably because they keep spidering IIS sites. OTOH, places like WebMasterWorld have a fair number of webmaster stories about having two or more case variations of the same page in the search engine indexes, with radically different page rank – and that depending upon what has been spidered recently, the position of the site will change radically.

There is a difference between ignoring duplicates and sending full credit for an inbound link to the “master” version of the duplicates. Some folk seem to think that there’s no great loss. I’m pretty conservative about this – why risk losing the benefits of inbound links, just because of case folding?

Case folding for private content is a problem. I’ve seen many complaints about Google revealing information concealed by Robots.txt, and I have strong suspicions that in most of these cases, the complainants were running IIS and had an undetectably case-mismatched link reference somewhere on the site, or had enabled Cookieless mode for ASP.

I can’t find any authoritative discussion about this issue, especially in Microsofts’ Knowledge Base, hence the posting here. Of course it may be that I detest MS so much that I haven’t spent enough time poking around their resources. That’d be a quite valid criticism of me and this article. :)

Other Solutions For Privacy

Given that Robots.txt is effectively useless faced with both Cookieless mode and case folding, you really, really need to get to grips with your friendly page-specific metatags “NOINDEX”, “NOFOLLOW” and NOARCHIVE”.

Any private content should at least have “NOINDEX”.

Arguably, terminal pages (action pages, etc) and private content, get “NOFOLLOW” directives.

Summary

IIS, by default, apparently opens up web sites to some Google Bowling activities. You can even bowl yourself out, if you use case variations in your own URLs.

Carefully review what you do have; probably the simplest suggestion is that you stick to all lower case in every link. Watch out for inbound links that use mixed case – those can scupper your rank.

Use the meta tag for robots to set NOINDEX and possibly NOFOLLOW on your private content on case-folding IIS systems to prevent your data leaking into the search engines.

Best bet – dump IIS. OK. That’s a biased view, and a very selective view. But if you want to rank highly in search engines, setting up and running a LAMP (Linux Apache MySQL PHP) or similar system is easy and cheap. While it does have comparable problems (unbranded 404′s etc) the solutions are also cheap and easy to set up. If I’m really blunt, I wouldn’t actually use PHP, either – the script language design makes it too easy to embed SQL, IME. I’d steer for Python, Ruby On Rails or OCaml… probably ;)

It’s better than finding your private data leaked, or that you’ve been scuppered by a near unfindable typo.

Clients that use IIS do seem to suffer from the strangest problems. I kept finding indexed pages, some ranking absurdly highly, with an path infix like (X(kjhgkjfkuyfku)). In other words, if I had a page like “http://merjis.com/login”, I’d find exactly the same content as “http://merjis.com/(X(stuff))/login, making it look as if I had lots of duplicate pages. Appended parameters as a session ID, I can understand. There’s even ways to cope with those.

The format of the infix was quite rigid – 23 characters starting and ending with a parenthesis, and embedding at least one more parenthetical group. I think the regex “\/\([A-Z]\([A-Za-z0-9]*\)\)\/” will match every example that I’ve seen. Oddly, you could navigate to them using a web browser and they worked, but even spidering the site and grepping the resulting mirror failed to show these strange paths in the HTML… so how did they get there?

Cookieless Session Tracking

This looks like an effort by Microsoft to allow tracking of people that don’t want to be tracked, or who might have, for example, an office based transparent proxy that blocks cookies. AFAICS, when IIS detects that a user doesn’t permit cookies, it starts sending unique paths. The use of the “referer_info” (sic) field allows tracking a single user across the site, looking at where they were and where they’ve gone to. The positive benefit is that users can be given access to stateful services (like logging in) without needing a cookie. It is, I think, effectively a session cookie, rather than a permanent cookie, as it can’t recognise you from previous sessions – your starting point will be the same as any other visitor.

This session tracking seems pretty barking to me. If a user prevents you from serving and saving a session cookie, and you then create a trackable session by using session keys in the URL, you have just violated the privacy that the user requested. If users try to use a use a resource that needs stateful dialogue – something that remembers whether you were logged in, for example – then I think that if the user disables cookies it is perfectly reasonable to tell them that this resource won’t work for any areas requiring an authenticated session, though the public areas may be freely roamed. If they want privacy in the session, then they can’t have access to anything that requires remembering from page view to page view, who they are.

Cookies and Search Engine Spiders

Robots don’t hold cookies. If faced with an IIS server configured to handle cookieless users, spiders end up being delivered with a false directory structure. Every new visit generates a new directory structure. You get excessive crawling and pages appear in the index as unique, when they are really duplicates caused by server behaviour.

Now, this too seems like a petty piece of madness. Given that some users want to maintain privacy to the extent that they do not even want session cookies, and this number is small on web servers that offer services involving a log in or other identification service, why would you make it more difficult for search engines to spider the site? I believe that you can get more users to a properly indexed site, than the number you’d lose from failing to handle uncookied users (unless you offer a specific service for the uncookied, of course).

Robots.txt to the rescue

Fortunately the workround for this problem seems small. In Robots.txt, add a line:

Disallow: *(


For all the spiders that I care about, that seems to prevent crawling the special tracking URL. The consequences of that… well, I’m not convinced it is entirely good. But it does stop silly URLs from being indexed after a single crawl.

Technically, this line says “disallow spidering for a URL starting with anything that has a ‘(‘ in it somewhere”. Although this client seems mostly to suffer from the infix immediately after the domain name, reading around the web suggests that the infix could be put at any directory level slash. Otherwise “Disallow: /(” would work and avoids the failure possibilities of the wildcard.

Spidering Improvements?

I can’t see why spiders should behave like this.

Having grabbed a server identity from the headers and behaviour, it should be possible to then strip out the session tracking from the path. I can’t currently think of a reason to *not* do this – unless you were really trying to sneakily discourage IIS administrators from using this tracking method.

I can think of anti-competitive ways to use the cookieless IIS behaviour. For example, find the user sitemap, point to it and let spiders follow the unique links to every path – then you can explode the crawl on each site by doubling it. And it accumulates – because every session appears to be valid for a long time, allowing repeat crawls and generating new unique paths with every reference.

You’d have thought that this explosion was worth trapping and stopping in the “fast crawl” discovery bots, so that the slower inspection spiders wouldn’t add these redundant pages to the indexes. The presence of these idiot links in link reports, and the multiple crawling of them by various bots, suggests that spiders are still stupid.

Webmasters?

Well, if you can’t avoid running IIS, take a good hard look at your web servers log files. Do you really have a useful volume of search from real users who are cookieless or have you just ramped up bandwidth so that bots can crawl redundant pages without adding any revenue? Is the majority usage of this feature an escalating collection of spiders? Might you get more users if link love wasn’t being directed to duplicate pages on redundant paths?

Put in the crawl reduction “Disallow: *(” and possibly a “Disallow: /(” lines to your Robots.txt and verify them with the Search Engine webmaster tools Robots.txt checkers. At least you’ll be focusing crawl on pages that should be indexed.

Ideally, turn off the cookieless mode. AFAICS, it is a breach of the privacy rights that users were trying to assert. If you offer a service that needs state, then when you detect that you can’t cookie, offer an apology that parts of the site are unusable. IMO, that’s perfectly acceptable for both users and designers and won’t lose any business that you couldn’t have gained.