I think the reason is that site owners who “invest” in spammy paid link purchasing rarely deeply understand what they are buying. And link spammers don’t really care whether what they do is effective, so long as there are people prepared to buy, and so long as Google and Bing mistake the links as being valid in the early days. Businesses usually evaluate the impact of an activity fairly early – so if they are told that the search engine impact will be most visible a few weeks or months after starting, then that’s when they’ll measure. A spammy link buyer will keep buying for years, because the impact is positive at the time of the measurement – the value declines with time as Google and Bing detect the patterns of spamming.

The very worst link spammers will submit your site to places that are already known to the search engines as places for low quality links, and already offering no value. So the most value that you get from the service, is a list of places that won’t have any impact… Useful if you need to go back and clean up the spammy links, later.

Take, for example, this piece of spam, submitted to this blog:

It looks appropriate. It’s about H1N1 and it has been submitted as a comment to an article about H1N1. But the URL given is for a product, even though the name offered is not a keyword. Is there any way to tell that it is spam? We could search for a key piece of text that seems unlikely to be in other comments. And here’s the traces that this is a piece of spam:

We can see the same author ID, with the same comment in many blogs – 314 blogs identified as carrying that precise piece of text, presumably with the same link to the “Fish Oil FAQ”. It is definitely spam.

In what way are Google and Bing so stupid that they can’t detect the same piece of writing in comments, when they can tell that a site has two or three paths that lead to the same product, wrapped in a templated page? It doesn’t add up that Google and Bing would penalise a site owner for multiple paths to a product that customers buy, but don’t penalise spammy links. So, do the search engine penalise spam?

Why are pages containing spam reported in search results, if the content is treated as spam?

Search engines are looking at the overall quality of the site and its’ pages. Some spammy comments to a blog or a discussion forum won’t kill the pages’ value. If users are finding the whole page is useful, then the whole page isn’t deranked – unless the web spam teams decide that the only reason for the page is to host, or be target of, spammy links. So you can find spammy postings on pages that have weight. A few spammy links on an otherwise useful page, won’t kill the page. That’s why we can still find spammy comments – they are a part of a page that is valuable.

That doesn’t mean that the reverse is true – spammy comments can be found, but it doesn’t mean that the links in the spam carry any weight. If they did carry weight, then we should find at least 314 sites are offering weight to the Fish Oil FAQ. So… where’s the site in the listings?

Interestingly, you can’t find the site named in the spammy posting. Yup. All that spamming and link dropping has had no useful effect at all – just try the search for links for fishoilfaq.com. Which just goes to show that the technique is pointless – it doesn’t work.

Back To Duplicate Content

Google and Bing are tolerant of genuine duplication within a web site. The large search engines even have a mechanism to help webmasters to signal that they are aware of duplication in their sites, and have a preferred path to that resource – the canonical link ref. A signal agreed to and used by the major search engines. It’s been so successful that search engines are now respecting the canonical link reference tag across domains (with some limitations).

But, identical postings, across a range of blogs and discussion forums, with keyword laden author names? Somehow that pretty obvious technique is supposed to defeat the search engines with wicked cleverness? It does, for a while. Then the web spam teams notice, zero weight the spam, and decreases their trust in your business. And that’s why duplicated postings in user generated content don’t work – blog spamming is an ultimately sterile exercise. If you’re going to comment, comment because you are a part of the discussion. Be interesting enough, and people will write about you and what you’ve written, in their articles – just as I’ve written about Danny Sullivan, below.

This decay in the value of blog spam (and other types of undeclared paid links) is why we hear the repeated refrain from businesses that:

“we hired an SEO agency for link building, it made an impact at first, but since we terminated the contract there’s been no impact on our business”.

Spammy link building has a positive effect on visitor volumes in the first few months or even years. But after a while, the search engines downgrade it, and decrease trust in the sites that gave you links (because those sites host undeclared paid links) and in your site (as a business that buys undeclared paid links). The activities have less and less impact with increasing time, and it is harder for your business to make headway once the search engines suspect that you are focusing on spamming as a link building strategy. You can even find that an entire chunk of your website is not being given any credibility for inbound links.

Blog spam has an unpleasant impact on the blogs it is dumped on, too. Read Danny Sullivan’s article about the way that blog spam affected a nascent site that may been useful to a specific online segment. The site did have some spam defences in place, but doesn’t it seem just a tad nightmareish that a site offering some long lasting value is taken out of action through activities that ultimately have no or little value. The economic equation is imbalanced. When that happens, as Danny implies, there is time and space for ethics and morality to play a part. Law? I don’t hold out a lot of hope for that – all that a US based law would do is to drive the targets offshore, or to use anonymising proxies, etc. (See our ancient article about tracking the steps in a spamming effort, apparently by some Ukranians).

Summary

Spend less time worrying about duplication on your own site – use the canonical link reference to help yourself and the search engines. Read Google’s official description about the canonical link reference, and how they have coordinated with Bing and Yahoo to understand the tag.

Spend more time worrying about what your linking strategy is telling Google and Bing – are you telling the search engines that your business will lie and deceive? Here’s Bings statement about spam and what they do in response to detecting spammy links. You really want those outcomes? You really want to pay people to cause work for other site owners, that has no long term benefit and may have disastrous repercussions on your own site? And when you find that the search engines no longer trust you, then you’re going to face a higher bill to remove links – there’s automated link placement, but the technologies for link removal are largely manual, and hence more expensive.

It’s pretty easy to establish the practices that lead to long lasting, higher ranking web sites. Start engaging with your prospects and clients, or find another way to engage with an online audience – at this point in the search engine optimisation game, they don’t have to be the audience that you sell to!

Follow this on Buzz.

Let’s have a look at what Google Product Search is, and does, first… so we’ve got some context.

Google Product Search

It’s free. Anyone can sign up, and feed products. Well, almost anyone. Google has a restriction that there must be one supplier for a given product URL. So you can’t have lots of people all listing the same DVD on Amazon – Google doesn’t like spam, and multiple appearances of the same product from the same vendor at the same price, count as spam.

Within limits, then, Google Product Search is free and a wide range of businesses can use it. You just need to sign up for a Google Merchant Center account, and publish a feed. A feed can be as simple as a properly formatted CSV file (exportable from a spreadsheet, once per month) or as complex as a machine-readable and real-time updated XML file synchronised with your product inventory and pricing.

Product Search results – well, they don’t appear in every search, and they don’t alway appear as number one in the search listings. That last item is especially important and tells us, the rest of the world, something about how Google is treating Google Product Search.

Why is it important that Google Product Search is not always number one for product related searches? Anyone involved with search engines knows that number of clicks to a listing is *very strongly* correlated with the position on the page. A listing in position 1 can easily get ten times, or more, as many clicks as exactly the same listing at the bottom of the page. So for Google to list Google Product Search at positions *other than* position 1, tells us that Google is giving up a lot of influence. And remember, Google isn’t making money (none directly, and only indirectly some revenue) from Product Search listings.

The implication of the varying position of Product Search results is that Google is not giving undue weight to their own product. The reverse implication – that Google is artificially reducing the ranking of competing product listing services – is not addressed by that observation. Just seeing that Google doesn’t inflate it’s own rank in all cases, is not enough to say that a different service has not been penalised.

Are there any other vertical search engines in Google results?

If Foundem has a case, it is that they are being excluded, artificially from results. You don’t just get listed highly in search results because you exist. Well, not unless the search is pretty rare. If you’re talking about the highly competitive spaces in which searches relate to things that people might buy, the competition is intense. To show there, you need a lot of “link love”. You need a lot of people to link to your site, with the right words, and you need the site to be well designed, and you need to have *credible* links. If you don’t have all of that, you don’t stand much chance.

So, let’s assume that Foundem, being fairly new, and not having made much of a media impact before today, doesn’t have a lot of link love. Can we support their argument by showing that Google doesn’t highly rank vertical search engines?

Annoyingly for Foundem, there are vertical search products that show. Businesses like Trovit, with their used car facet-based searches, letting you slip a slider along the price range that interests you, does show up. And they show up for place reviews. And other stuff… It is arguably a better user experience than Google Product Search for used cars. Trovit have page one appearances for some products. I can see their impact on clients’ sites.

Google *does* allow vertical search specialists to appear in search results. That, for me, puts a dent in the argument that Google excludes vertical search specialists. If some do appear, and they collect clicks, in multiple categories of product, then Google is not completely removing them from listings. A dent, but not conclusive evidence.

Do Ciao and Foundem have a case?

I don’t think an appeal based on “we must be shown because we’re a competitor” has any real merit. I’m not a lawyer – so that’s not a commentary on whether the courts would find it to be true. But in terms of search engine results, if you aren’t sufficiently loved by users, to displace the other ten businesses that are shown, that’s not, IMO, a reason for a court to decide that you should be shown. Win the users’ love… and you should be listed.

But, since the algorithms are trade secrets, and the data collected is known only to Google, for an outsider to claim that they have been penalised by Google, is pretty difficult to discern, or prove.

Unless, of course, you have the backing of another search engine that produces broadly similar results, in which your site appears more highly ranked and in which you have no evidence to believe that the results are influenced. In other words, if you were financed by Bing (the only other serious volume search engine left in the game), then you might have strong evidence that you don’t have spammy links, spammy comments and other factors that would cause de-weighting.

Ciao is owned by Bing. Might there be some evidence? Or is this just Bing (Microsoft) playing games with Google. FUD (Fear, Uncertainty and Doubt) have always been strong components of Microsoft marketing techniques. I certainly don’t have any confidence that Microsoft is doing this because they love Egalité, Fraternité, Liberté and the European Way Of Life.

Foundem – it’s a small, relatively new player, in a crowded market. I suspect their only problem is the classic one – not enough users love them to help their site to rank highly. But I haven’t investigated their site or backlinks. That’s just a first pass impression with no evidence. I’ll gleefully admit I could be wrong – though I’d be very surprised to be badly wrong.

But the possibility of Google reducing the rank of a disfavoured competitor, or slightly inflating the rank of a preferred information supplier, is something that gets me twitching. There’s no oversight for Google. We have only their word that they are using what they publicly claim to be fair. If, in the depth of a recession, California based Google were to be weighting US businesses slightly higher in the UK than they should be, it would direct between tens of millions and billions of spend to US companies and away from British companies. Same for France, and Germany. Is Google making sure that the US economy brings in a little more business than it *should* do? I have no idea. I can’t measure it. I can’t tell. It does worry me. I worked in the USA for about a decade, and I know how insular US businesses can be. The rest of the world has fewer investors, fewer buyers and works in funny languages and currencies; it is easy to fall into the model of thinking about what works best in the US economy and rolling that out, worldwide.

So there *may* be some basis for European businesses to wonder if Google is deweighting their business. It may be deweighted because it doesn’t fit a market model that Americans understand. It may be deweighted because an American business is, even unconsciously, working for the end of a recession in their domestic market. It may even be a conscious effort. That really, really worries me – because I don’t think even a national government outside the USA, could get the data to investigate whether Google is deliberately, or unintentionally, diverting economically significant activity, artificially, to the USA.

Do Ciao and Foundem have a case? I don’t know. I’m looking forward to finding out, if it is released, just what evidence might make them think they do. I hope it’s a lot more compelling than “we want our share of the Google billions”, “Microsoft is using me as a sock puppet to attack Google” or “Google’s Buzz is creating a weakness in perception we can exploit”. I fear there may be some interesting biases in Google’s system that do favour US businesses – simply because I can’t see any way to prevent there from being such a bias, or to detect one if it were present. And *that* has implications for national security – the protection of the realm. Interesting times.

Updates

2010-02-27 – SearchEngineLand article about Microsoft’s involvement.

• Early warning of rank changes – helpful to know, and there’s a feedback form at the bottom of the page.
• Competitive response to Binghoo – telling the world they aren’t a relaxed incumbent
• Future indicator of AdWords policy and QS changes

Does this announcement imply any more than that? I don’t know yet. I’ll have to think about it a bit longer.

Geolocation

This test is only available in the US, making testing a bit of a pain. But for certain categories of search, local results should dominate national well ranked results. If you want to find a company that can fix a lawnmower, then finding a high ranked site that is 2000 miles away is probably much less useful than finding a low Page Rank site 20 miles away. While the internet has globalised some functions – purchasing products across international borders is now much easier, even if the legal framework hasn’t fully caught up with the capability – there’s still lots of local needs.

So I’m expecting to test local searches extensively… And I’ll be looking for how my US clients stack up in results and to see what happens to a few of my European clients to see what happens to their results.

AdWords Implications?

AdWords policies aren’t always the same as Google organic search policies. However, it is clear that Google often applies lessons from organic search to paid search. For example, organic search duplicate content rules have become stricter over the years and AdWords has more strictly interpreted duplicate advertising to the annoyance and frustration of many affiliates.

Quality Score is also clearly drawn from organic search patterns – the requirements that landing pages include text matching keywords and advert is well known by now. Whether that’s a valid mechanism or not is still, for me, up for some dispute – I’m pretty sure that I can design a highly converting page that would be given a very low landing page quality score, proving it popular with users and unpopular with Google’s current initial QS rating mechanisms.

I’ll be looking at whether changed organic search results might have any impact on Quality Score. That’ll have little immediate impact on current PPC strategies, I expect, but I often find that an exercise of testing improves understanding of the current system – so there will probably be accidental learnings that may change strategies for some clients.

First Impressions

This is the result of a small sample of twenty high volume searches for a specific client. Looks like important sites move up; heavily optimised sites by “unknowns” are moving down a little or are completely lost.

If continued in other tests, this may indicate a trend that Google has been following recently. The old “you were linked to lots so you must be important” rule has been replaced over the years with refinements like “but you were linked to by places that you linked to in massive link directories” and “but you were linked to massively from free for all directories”. I’m not sure what the basis is for Caffeine, but I’m guessing that other factors about the business are being taken more seriously. Possibly mentions on other key business web sites, and further de-ranking of social commentary sites often abused by SEO efforts?

Time to do some some more testing and see whether any statistical correlations can be made between mentions in different types of sites and the rank changes. That’ll be time consuming, but may shed some illumination for future strategy :)

As Michael himself says, this can’t be taken too seriously – the identities and goals of testers are varied and it is perfectly possible that the system may be easy to game, resulting in perverse conclusions. I’m not taking the stats collected for other testers as very serious – but I am extremely interested in my own preferences and the implications for my clients.

If you do a lot of search and search comparison, you can actually recognise the results pretty well from the length of Titles and Description and the way in which links are presented. I tried to use the tool “fairly” – that is, selecting the one that presented what I thought was the best set of results. I used searches like:

• gclid
• search engine optimisation
• seo
• google money
• google scam
• msn adcenter
• yahoo!search marketing
• google adwords
• adcenter
• overture
• adwords
matt cutts
• danny sullivan

and, of course, the names and products of many of my clients, because I’m very familiar with their pages and competitors, so I can assess with some expertise.

When I started using the tool, there were about 11,000 searches so far conducted. At that point Google was in the lead at about 40%-45%. During my testing, I saw the count crank up radically as new people discovered it. By the time I finished, there were about 15,000 search tests – and Google was still in the lead.

What surprised me? The amount of spam in various list results. I’m aware of Matt Cutts and his teams role at Google and that there are similar teams at the other SEs, of course, and I’m aware of many types of spamming. I normally use Google, so seeing the misindexed results on low search volume keywords on Yahoo and Bing, where spam and accidental inclusions seem to be more prevalent, was a reminder of *why* I often use Google.

The search “google cash” was interesting, with Google apparently having removed a lot of scammers listings that were left in Bing and Yahoo results. I’ll guess this is largely manual by Google – and somewhat of a dereliction by Yahoo and Bing, IMO. “Best Search Result” should also mean “least likely to lead to a ripoff”, not just “best indexed” – care about the impact on users should be significant. If someone promotes drinking Drano as a cure for the common cold, it shouldn’t be number 1, even if heavily linked and referenced.

“gclid” – an arcane topic about AdWords automated advert tagging and web analytics – where the non-Google systems showed quite a few listings in which the content was not about gclid, but there were embedded links including gclid. I infer that document scanning is somewhat better at Google, or user feedback on appropriate results is given more weight.

When I used my own name (vanity search) and variations, I was hard put to identify which search engine to pick. They had substantially similar contents, changing the order of entries rather than what was shown.

Yahoo showed some search results much better than I expected. Bing, too, every so often shone clearly better results.

This parallel testing allows much better comparison than the usual sequential testing I’ve done. I’m hoping that the Blind Search tool will be kept running and improved. This may be a nascent business for Michael – identifying the best search engine for different geotargets, languages, categories of search, perhaps by crowd-voting – there’s a career in that :)

What did I personally observe? About 45% of my votes were in favour of Google, roughly equal for Yahoo and Bing at 25% and 30%; so I prefer Google almost twice as much as Yahoo, and about 50% more than Bing. That roughly fits with my preconceptions of what I thought about the search engines. Bing is major leap over Live. But still has too much spam, and too many irrelevant results *for the small and specialised sample of searches that I used*.

Enhancements I’d Like To See

Use of local indexes. The current tool obviously uses the “.com” indexes. So any UK-only clients fared fairly badly, sometimes being pipped by US-specific responses, when I know they top the UK results.

Record the percentage for each SE *by date* so that emerging trends in improvements by SEs can be observed. Also graph the number of searches by date, so that some level of reliability can be assigned, or use error bars on the vote chart.

A better URL. :)

Personal stats. I consider myself a “fair” tester. I want to see what I assess as the best – my personal record of B vs G vs Y.

I’d also like to see what searches were done, and whether there are patterns in the searches such that some of the SE’s excel at, say, travel, or automotive, or financial service results.

Really interesting tool, Michael – thanks!

I’m used to the phenomenon of users ReTweeting, in which the original author is given attribution. Are there really so many Twitter users out there, with nothing original to say for themselves, and who don’t have the generosity of spirit to apportion authorship, or is there a sponsored campaign going on to post substantially identical messages *except for* the shortened URL?

And there’s more…

As I say, I’m a relatively new Twitter user, so it is possible that this morally dubious behaviour is common. I hope that there some kind of community sanction that can be taken against users who ReTweet without attribution – but I don’t see anything obvious short of claiming them as spammers.

As Twitter increases in importance, this bad practice of plagiarism seems likely to increase, and starts to decrease the social networking value of Twitter.

It may also point to how corporations are starting to use Twitter to manipulate opinion; trying to force particular topics onto the “breaking list”. If I’m behind the curve on this, who is covering this kind of stuff about Twitter use and abuse and the likely future threats to Twitter’s value? I’m certainly going to be a lot more reluctant to consider the breaking meme listing, knowing that it may be being manipulated without apparent oversight or action.

Some of these are clearly spamming – the two different accounts at the break in the screenshots are obviously related and identical postings – operated by a common source, I suspect. Some appear to be fairly normal accounts, though I didn’t go looking for other plagiarism, it is possible that they may have been systematically ReTweeting without attribution.

Summary

It appears that Twitter is definitely reaching an importance level that means we can expect results to become affected. I know I’ve seen manipulation of trends, but I’d understood until very recently that this was mostly larking about. It could have been test runs to exercise a promotional network and demonstrate to likely buyers that the service would work.

Attribution, always an important topic for search engines, is even more important for social networking. Its part of trusting your news sources. Seeing this has damaged, for me, trust in the Twitter service.

If you tend to the dark side, along with spamming Google, spamming and plagiarism on Twitter probably are part of your mix, or will be.

As an ordinary user, I can’t see any sanction other than to report spamming; but it looked to me as if many of these people were otherwise just ordinary joes. Account closure for minor bouts of overenthusiasm seems harsh.

Updates

Danny Sullivan finally catches up with ReTweeting, misattributed and malattributred Twitter spam. (I’m joking – I got there earlier, he got there in more depth – good article).