Google is destroying the web!

Adam Lasnik, Google’s missionary to the heathens, fired up to convince webmasters to use links only if they fit the citation model, wrote a few months ago that he and Matt joked that people are often bragging they have an undetectable technique to raise rank. The interview (second link in this paragraph, to Stone Temple) makes it clear that the focus of their effort is paid links on high PR sites. But the economic significance of Google has another effect: web design and web marketing decisions are influenced by the commercial reality that high rankings drive wealth.

How does this relate to the destruction of the web as we know it? Why is this any kind of a big deal? Today?

Today (well, strictly, yesterday) is the first time that spammed links submitted as comments to this blog have exceeded the daily spam detected in my email. That ratio is related to Google’s success and the way in which marketeers use their marketing mix, and why Google’s current ranking system is destroying the informational value of the web. I prepared this article back in May, as I saw the ratio gradually cranking up… It’s a sample of one blog and one email account, so the exact ratio is spurious as a worldwide marker, but is probably indicative of the way that the internet is evolving.

Huh? So what?

Your major reason to care is that this drives up the costs of internet marketing, making internet products more expensive for end users, but without increasing their benefit (either perceived or actual). That, in turn, slows the pace of online marketing expansion.

One way to understand the news that my blog spam rate exceeds my email spam rate, is to assume that this blog is becoming more widely cited. Well, that’s partially true, I suppose. I’ve a long way to go before this blog begins to attract a significant fraction of the traffic to Andy Beal, Matt Cutts or Danny Sullivan’s blogs. If I get this much spam, their blog spam lists must be truely awesome.

Of course, not all blog spam is oriented towards building page rank. Some of it is conscious advertising, hoping that users will follow links in the comments. However, the preponderance of blog spam that I see has author names, links with anchor text that look just like you’d expect from a SEO. SEO’s offer blog spamming as a service. And there’s advertised tools that spam blogs. They even offer to inject, just like spam email, high entropy text to evade spam filters.

Another way to read this spam ratio data point, is that businesses are finding blog spam to be a useful way to drive traffic. A high organic ranking can drive traffic for months, whereas one email campaign generates a limited duration boost to new visitors. If they were the same cost, which would you use? Which would your boss force you to choose?

The costs of generating email spam and blog spam are fairly similar. Both can use intensely automated systems for submission. The costs of submission are miniscule, for each spam item, but you need a lot of spam to make any money… so these costs ultimately do mount up, to a level that you notice.

Businesses do pay for meaningless activity. There are marketing managers who buy spammy SEO services without understanding what they’ve done. I have no doubt that a large portion of these paid activity links have absolutely no effect, or even in some cases a negative effect. Pretty much any blog spam submitted to this blog, for example, is wasted - except, I suppose, that I look at one or two of the links, usually via an anonymising service, to understand the technical and economic motivation for the spammers.

Some of the blog and other link spamming activity is clearly working, or believed to work. One consequence is that anything that looks like a form that might lead to web postings on blogs or discussion forums, is being stuffed with keywords and links and submitted. We’ve some clients, whose sites we manage, that have seen blog spam appearing in webmail inquiry forms, for the first time since we started managing their sites years ago, in the last few months. Forms previously only touched by prospects, are stuffed with MMORPG gold links, pharmaceuticals, cars and stuff in non-Roman characters sets, and then submitted to businesses unrelated to any of these activities, and who don’t publish the forms in any way.

This all suggests that the techniques are working, even with some failure rate through posting to blogs that don’t show spam, forums that moderate and forms that send data to internal mail users. Even though some people clearly are paying for essentially superstitious activities, there is too much link spamming for all of it to be ineffective.

Truth and Consequences

Long term, the effect is that link-based systems to determine rank are going to be as dead as the old in-page content ranking systems and the human moderated directory. These still exist as technologies, and they are still generating useful revenues for their businesses, but they aren’t where the majority of users are going. They are effectively “cash cows” - technological dead ends, being milked of the revenue they generate, but no longer in the growth phases.

Realistically, it took ten years for internet marketers, content management system designers and businesses to switch from stuffing meta tag keywords on pages and using human moderated directories, in order to accommodate an information rich, search engine friendly, link stuffed style. It’ll take at least ten years for this reshaping of “natural links” to decay back into a informational web, and that’s assuming that today we had, and were all using, an effective third generation search engine that didn’t use the informational web to reflect economic utility.

That search engine may be out there… and we just don’t know about it. Or it could be that Google has seen the threat to their business, and is actively working on it. They have enough smart people that this is feasible, and they’ve taken innovative steps in the past. Even though Google’s business is currently destroying link value, there’s a chance that they’ll also save the web.

Third Gen Search

The essence of Page Rank, as I’ve said in previous articles, is to find informational pages. This was useful in 1997. It was useful in 2000. But as the world’s economy shifts online, the search results for some subset (by volume) of searches, is primarily directly economic. That is, people do searches to find out stuff they need to know, but some searchers are looking for somewhere to buy from.

Informational page value and economic value to the searcher are distinctly different. The result is that second generation search systems using link citation will only find economically relevant sites, with low information content, if many of their marketing managers use techniques that destroy Google’s investment in the citation based system.

Simple Cases

Let’s take a simple example. I realise that detailed discussion about this example can blow the case out of the water, because real life is always more complex. However, this is representative of a class of sites that have problems with Google now, and who are most likely to be using link spam, in order to overcome informational bias in search results.

Let us suppose that you want to buy a DVD. You know the film you want (John Carpenter’s early SF film, “Black Star”). Many of you would be perfectly happy to buy from a site that simply told you it had the DVD in stock, that it shipped to the region you were in, and gave you some content that indicated it was a reputable trading business that wouldn’t just take your money without shipping anything. For the most part, you don’t need a review, don’t need customer recommendations (you know this film, you just want it on DVD). This minimalist site would be perfect for shoppers who searched for “buy DVD john carpenter black star”. Would it rank? Probably not. It’d be buried below a bunch of sites that had all the needed content, plus additional stuff. Additional stuff put there because information and economic utility have been confused. Additional stuff that costs money to do and drives up the cost of business (reducing margins, or increasing prices to end users).

The marketers response to low ranking? Hire a SEO to build a network of (aged) domains, and stuff them with links; make postings to marginally relevant forums; spam marginally relevant blogs; stuff forms on websites in the hopes that some of them result in page content; add reviews or other content to the web page to make the page more attractive to ranking systems, etc. None of this benefits users. None of it is free (it takes time, or rented software) to do.

Comparison Shopping

So, what about a comparison shopping site? Many shoppers like to use these, because they don’t have to open a zillion windows, with shipping information and availability and pricing presenting in a multiplicity of ways (some of which may require opening a single store in multiple windows). You may use a comparison shopping site when you know the products you are interested in, but don’t know what the best internet price is, or who has the item in stock now.

A minimalist comparison shopping site would list vendors, prices, geographical shipping criteria, availability, all on a single page. No reviews. Perhaps a star or popularity rating for vendors. Perhaps a “people who looked for this also considered” list of other products.

Again, this site is unlikely to rank. Indeed, with the informational citation model, this page would be very low ranked - the outbound links are often JavaScript (for tracking purposes) or obfuscated via transitory database reference, so the SE’s often won’t see links to highly relevant pages outbound. And, of course, comparison shopping sites are plausibly understood as a “secondary search site”. That is, this is a site that having once reached it, you have a further search to do (very common on the home pages of these sites, and even in department and category pages - search is a pretty well understood user interface model, for users).

Just as with the pure sales site, the pure comparison shopping site doesn’t need a huge informational burden. If it has any information, it is probably about the vendors, not the products. That’s what a user in the final phases of the buying process is looking for. “Who will sell me this item?” Not “Do I want to buy something like this?”.

In the informational model of search, for these sites to rank, they need to be different sites (with extra content, added at a cost, in order to satisfy the search engines) or to pay for links, or to survive only on paid search… Even though they arguably assist users to find what they want - the best price for the type of item they want to buy.

Economic Value of Link Spam

There’s an commercial imperative to link spam. That’s why my Akismet tab on this blog is stuffed with several hundred spammed comments each and every day. That has a cost to me, and to the rest of the internet - every blogger is facing a rising tide of link spam. Every business is beginning to see any web form, and support inquiry form, stuffed by spammed links… because current search models have confused information utility and economic utility.

So, we now get to the point… Google’s success in informational search is now creating the conditions in which quantities of poor quality information are being generated purely to offer rank (low value postings, link spam, SE bait). Links are established using a myriad different techniques, in order to promote an economically valuable page to the top of an information delivery service.

It means that Google is having more and more difficulty with finding the right content. It also means that potentially better search technologies are looking at an environment where the page content and links can’t be trusted. Page content was broken by page content spam in the late 90’s. Now links can’t be trusted without a huge burden of processing to dismiss possibly purchased votes. Pages have information on them that exists purely to attract search engines, and does nothing for the user experience. And you and I now have to spend economically valuable time dealing with blog spam, instead of earning money.

Better Mouse Trap?

Google has obviously been exploring how to move from the citation link model. I’ve seen some articles mentioning Google patents for other ranking techniques. As link spam comes to form a significant fraction of all links, Google and other SE’s will be forced to use new models to generate commercially and informationally useful rankings. These patents are clues as to the direction that Google is going.

Search marketers should be thinking now about what shape their marketing efforts will need to take, in order to position themselves for the next few years. Just as non-spammy link building exercises and user-valued content development have taken time, these new techniques will require resources, funding, strategy and tactical development… the early bird might not get the worm, but they’ll eventually be eating other people’s dinner.

Predicting these things is always nightmarish… but if you don’t start to work on the strategy, some accidental bystander gets rich. It’s actually part of Eric Beinhocker’s “The Origin of Wealth”, a book that is beginning to rank in my mind alongside Evans and Wurster’s “Blown to Bits”.

What will leading edge marketeers be doing next year, to get the jump on the changes? Interesting question… for another time. Oh, in reference to this article? They’ll still be linkbaiting. Gee, if you are here, these techniques are probably working.

Summary

Google’s use of informational content to determine page value to a user is a consequence of a specific belief about how users use search results. As users have become more sophisticated, content has evolved, and usage is altering to put increasingly large volumes of economic activity on the web. Although not directly cited here, much of the initial contact from web user to vendor is mediated by search. A search engine results system based purely on informational value and links does not necessarily satisfy searches for purchasing activity. The consequence is that organisations engage in techniques to artificially improve content and links in ways that are not always directly or even indirectly beneficial to users, or natural to the economic activity.

Recasting the economic value of a page as informational value, results in an increased burden of economic activity that is not directly related to the purchase, or even the same market, by larger numbers of users, to a population outside the search engines - site administration and web marketeers. Even if a solution were available now, in the form of a search engine that presented alternative results based on both informational and transactional interpretation of the query and of ranking, it might take a decade for the effects to be felt in Content Management Systems and other web site delivery knowledge.

Strictly, it isn’t Google destroying the web, but a myriad other players, who interpret the activities of Google and react individually to maximise their performance. This ought to be a classic study in microeconomics and of rational, partly rational and variously informed participants. This collection of users with conflicting goals and varying degrees of imperfection is why response doesn’t form an immediate equilibrium, nor do all participants cause direct economic effects - high value results, lack of information, mixed levels of competence and low penalty costs for damaging behaviour causes bystander effects, and will do so for a long time.

Material Disclosures

Some books are referenced above. These links are not paid for in cash, kind or reciprocal links, nor do we take an affiliate revenue stream for the recommendation. We do not have a relationship with the web sites or products mentioned above, and we derive no income or consciously sought page rank from referencing them.

We do international/multilingual paid search management and performance improvement, some internet strategy consulting and web site conversion improvement, with an emphasis on White Hat SEO for pages we build.

Updates

2007-08-02

Add link to Lambda The Ultimate in comment below, citing their article about spam and user comments about migrating to trust-rank and authentication based systems to help control it. This is the most advanced programming forum I know of. If these guys are headed there, then we probably all are.

Some text tidyup for clarity. Notably in the section about a minimalist site, I added some explicit detail about likely marketing responses to low rank.