'spiders' Category

• SEO: Close Reading Of Search Results

  Looking closely at Google’s search results can be informative – at least, if you take some inductive leaps, and apply knowledge learned in other activities. Take a look at the graphic, showing these early April 2010 search results for Matt Cutts’ web site. Notice that the same articles appear several times, wIth slightly different URLs? [...]

  May 17th, 2010

• Non-news: Malformed URLs don’t pass Anchor Text.

  I’ve started another burst of postings about web server log file analysis and what it tells search engine optimisers about search engine spiders. Web spider behaviour often lies behind issues that I find on other blogs. For example, Dave Naylor has a couple of recent articles that are interesting. A good one to read is [...]

  April 9th, 2010

• Googlebot and Search Visitors

  I’ve been interested in the behaviour of Googlebot, the robot that Google uses to crawl the web, for years. It’s a topic that seems largely unaddressed by search engine optimisers, yet the behaviour of Googlebot should be extremely important. After all, uncrawled sites tend to have problems with ranking many pages – the best you [...]

  April 8th, 2010

• Google Hates Me, I’m Being Penalised. Or Not.

  Great story from a Google staffer about how his site started to disappear from rankings. I’ve seen clients lose *huge* chunks of traffic for very similar reasons. Sometimes the reason you don’t show isn’t for the obvious search engine optimisation reasons or that you’ve lost Google’s love. Sometimes there’s a simple technological explanation…

  February 25th, 2010

• Spiders, IIS Caseless, Cookieless and Search Engine Indexes.

  Digging into IIS web server log files is quite interesting. I’ve developed a number of in-house tools over the years that help understanding why web spiders go where they do. I’ve been reworking them from an Apache dominated view to include some of the things that IIS does.
  You can see requests like “GET /(J(1)S(4dab…..))/” [...]

  September 15th, 2008

• SEO, IIS case folding filenames, Spiders, Analytics, and Robots.Txt

  AFAICS, the best way to administer IIS for SEO purposes, seems to be to run screaming from the room and hide under a desk until you are allowed to use Apache. So many of the default behaviours create difficulties for users or SEO. Yes, I’ve been continuing to dig into web analytics and IIS web [...]

  August 20th, 2008

• IIS Cookieless Generates Spider Crawling Problems

  Another case of Web Server Log File Analysis on IIS being disturbed by bots, having the potential for SEO naughtiness and spamming the search engines. The problem is created by IIS’s cookieless model. The idea appears to be to present a unique string in the path so you can track sessions without needing a cookie. [...]

  August 18th, 2008