Blind Search Testing

Published on June 8th, 2009 by Jeremy Chatfield

Microsoft employee Michael Kordahi has a new and suddenly popular tool - Blind Search. It presents search results from the three search engines side by side, with branding removed. Until a few hours ago, you could see which search engine users were voting for - until some git decided to game the service.

As Michael himself says, this can’t be taken too seriously - the identities and goals of testers are varied and it is perfectly possible that the system may be easy to game, resulting in perverse conclusions. I’m not taking the stats collected for other testers as very serious - but I am extremely interested in my own preferences and the implications for my clients.

If you do a lot of search and search comparison, you can actually recognise the results pretty well from the length of Titles and Description and the way in which links are presented. I tried to use the tool “fairly” - that is, selecting the one that presented what I thought was the best set of results. I used searches like:

• gclid
• search engine optimisation
• seo
• google money
• google scam
• msn adcenter
• yahoo!search marketing
• google adwords
• adcenter
• overture
• adwords
matt cutts
• danny sullivan

and, of course, the names and products of many of my clients, because I’m very familiar with their pages and competitors, so I can assess with some expertise.

When I started using the tool, there were about 11,000 searches so far conducted. At that point Google was in the lead at about 40%-45%. During my testing, I saw the count crank up radically as new people discovered it. By the time I finished, there were about 15,000 search tests - and Google was still in the lead.

What surprised me? The amount of spam in various list results. I’m aware of Matt Cutts and his teams role at Google and that there are similar teams at the other SEs, of course, and I’m aware of many types of spamming. I normally use Google, so seeing the misindexed results on low search volume keywords on Yahoo and Bing, where spam and accidental inclusions seem to be more prevalent, was a reminder of *why* I often use Google.

The search “google cash” was interesting, with Google apparently having removed a lot of scammers listings that were left in Bing and Yahoo results. I’ll guess this is largely manual by Google - and somewhat of a dereliction by Yahoo and Bing, IMO. “Best Search Result” should also mean “least likely to lead to a ripoff”, not just “best indexed” - care about the impact on users should be significant. If someone promotes drinking Drano as a cure for the common cold, it shouldn’t be number 1, even if heavily linked and referenced.

“gclid” - an arcane topic about AdWords automated advert tagging and web analytics - where the non-Google systems showed quite a few listings in which the content was not about gclid, but there were embedded links including gclid. I infer that document scanning is somewhat better at Google, or user feedback on appropriate results is given more weight.

When I used my own name (vanity search) and variations, I was hard put to identify which search engine to pick. They had substantially similar contents, changing the order of entries rather than what was shown.

Yahoo showed some search results much better than I expected. Bing, too, every so often shone clearly better results.

This parallel testing allows much better comparison than the usual sequential testing I’ve done. I’m hoping that the Blind Search tool will be kept running and improved. This may be a nascent business for Michael - identifying the best search engine for different geotargets, languages, categories of search, perhaps by crowd-voting - there’s a career in that :)

What did I personally observe? About 45% of my votes were in favour of Google, roughly equal for Yahoo and Bing at 25% and 30%; so I prefer Google almost twice as much as Yahoo, and about 50% more than Bing. That roughly fits with my preconceptions of what I thought about the search engines. Bing is major leap over Live. But still has too much spam, and too many irrelevant results *for the small and specialised sample of searches that I used*.

Enhancements I’d Like To See

Use of local indexes. The current tool obviously uses the “.com” indexes. So any UK-only clients fared fairly badly, sometimes being pipped by US-specific responses, when I know they top the UK results.

Record the percentage for each SE *by date* so that emerging trends in improvements by SEs can be observed. Also graph the number of searches by date, so that some level of reliability can be assigned, or use error bars on the vote chart.

A better URL. :)

Personal stats. I consider myself a “fair” tester. I want to see what I assess as the best - my personal record of B vs G vs Y.

I’d also like to see what searches were done, and whether there are patterns in the searches such that some of the SE’s excel at, say, travel, or automotive, or financial service results.

Really interesting tool, Michael - thanks!