'spiders' Category

• Google Hates Me, I’m Being Penalised. Or Not.

  Great story from a Google staffer about how his site started to disappear from rankings. I’ve seen clients lose *huge* chunks of traffic for very similar reasons. Sometimes the reason you don’t show isn’t for the obvious search engine optimisation reasons or that you’ve lost Google’s love. Sometimes there’s a simple technological explanation…

  February 25th, 2010

• Spiders, IIS Caseless, Cookieless and Search Engine Indexes.

  Digging into IIS web server log files is quite interesting. I’ve developed a number of in-house tools over the years that help understanding why web spiders go where they do. I’ve been reworking them from an Apache dominated view to include some of the things that IIS does.
  You can see requests like “GET /(J(1)S(4dab…..))/” [...]

  September 15th, 2008

• SEO, IIS case folding filenames, Spiders, Analytics, and Robots.Txt

  AFAICS, the best way to administer IIS for SEO purposes, seems to be to run screaming from the room and hide under a desk until you are allowed to use Apache. So many of the default behaviours create difficulties for users or SEO. Yes, I’ve been continuing to dig into web analytics and IIS web [...]

  August 20th, 2008

• IIS Cookieless Generates Spider Crawling Problems

  Another case of Web Server Log File Analysis on IIS being disturbed by bots, having the potential for SEO naughtiness and spamming the search engines. The problem is created by IIS’s cookieless model. The idea appears to be to present a unique string in the path so you can track sessions without needing a cookie. [...]

  August 18th, 2008