Rev A: SEO, Game Theory and Intrinsically Corruptible Systems

I’m pretty sure that I can see a rising interest in, and reasons for, dislodging Google’s search and search advertising dominance. However, the causes are complex and the way in which it would happen are even more subject to unpredictable accidents. I’ve now written this article twice, with different perspectives… And I couldn’t think of why I shouldn’t publish both. So here’s Rev A.

Some days ago, Matt Cutts made a pretty bland posting offering the idea that Google is interested in collecting user reports of paid links. But, thud (that’s the other shoe dropping), Google’s attention to paid links is part of the end game. Page Rank was a great system. It worked much better than the in-page ranking systems and meta-tag ranking systems that came first, and it knocks human edited directories into a cocked hat, in terms of the speed with which content could be ranked and the implicit structure of information reworked in response to changing concepts.

But for all the benefits, Page Rank is still only a second generation web search technology and suffers from a critical weakness - at some point it derives rank from information purely held within the web, but there is no incentive to make honest pages and links. See comments in Matt’s column by BSolveIT and Dewald ama-Canuck, for similar arguments.

I occasionally give a lecture at a local university. Every time, I’ll be asked by someone about the Semantic Web, and whether they should be building products based on those ideas. I’m not a fan. I really like the idea of living in a future like that, I just don’t think it is realisable.

Why is Page Rank dead, and the Semantic Web stillborn?

OK, so maybe PageRank isn’t dead, but it is critically wounded. The core problem is that the idea was developed in an era when the link wasn’t currency. As soon as income became tied to PageRank, it was inevitable that the link would become a currency. But links are free (or nearly so) to the donor. So link spamming is a cheap technique - it’s literally printing money.

The consequence is that Google (or any other SE using purely internal information) will end up fighting a battle of filters. “If you do this, then I’ll filter out those results”. “Ah ha, but that now puts more weight on this type of link, so I’ll simply buy those links that appear to look like links you favour”.

Game theory has quite a lot to say about these sorts of system. They don’t work well, for either participant - more and more effort (cost) is spent on trying to beat the other partner. This is because the reward system is geared to punishing non-compliance. There is no reward system for being a good citizen (you don’t receive a benefit from only posting good links). The effect is that poor behaviour is rewarded by economic incentives and so is more encouraged, despite the increasing penalties when caught. And so far, SE spamming isn’t an illegal activity (unlike email spamming).

Ways to stop the corruption

As with email spam, the problem is that the costs of production are a miniscule fraction of the economic benefit. I get a few hundred spams per day, and a hundred or so emails (many from automated systems reporting their most recent activity). Most of my email spam reaches a spam filter and no further. Even if I see the miniscule fraction that make it through the spam filter, the obvious techniques used in addressing, and content that avoids the filter, allows me to tag them before opening. I see about five spam emails per month that sneaks through everything and is plausibly a possible real email.

I also lose several emails per month, to the spam filter, that should have been shown to me.

That’s a false positive rate of 1 in thousands and a false negative rate in the same order. Not bad. It’s all done using content measurement (nothing smart enough to look at relationships of other addressees, etc).

So the idea of using filters to clean up PageRank is not immediately obviously broken. If the spam rate can be dropped to 0.1%, then the lists of search engine results can be cleaned to the same level, surely?

Erm. No.

The problem is that spam email was never going to be the starting point for a huge volume of purchases. Currently the killer in this arena is search. If only 0.1% of links are used for building rank, it simply generates an intent to use ever more technology and more low cost economy labour to generate many, many links… Because the more varied you make the strategies, the more chance that one or more of the links will be unidentified. With the economic value of first page ranking being so high, the costs of performing aggressive link spamming become trivial. That, of course, leads to more incentives to identify link spam.

Ultimately, it is a Turing Test. “Is this page the result of a human interest or is a page built by a robot to simulate a real and interesting page?”. We’ll have AI’s judging AI’s, eventally, as filters give way to pattern recognition techniques and page generators turn from algorithmic simplicity to learning systems to rapidly research a subject and reword it as a high entropy page. This may be a way to accelerate the Rapture of the Nerds, but it isn’t a way to deliver high value search results that reflect human preoccupation.

I suspect that we will end up in a position where legitimate links, correctly built to identify useful resources, are penalised. When that happens, PageRank is completely dead.

How to escape the trap?

Diagnosing the problem and prescribing a way out of the predicament are completely different skills. There’s many ways to cure the problem - tackling the way that the current system offers only costs, creating a new system, etc.

We could also take a look at mature, or reasonably mature, disciplines that focus on trust. That’ll be information security and the security industry. What can we learn from them?

A web page is a document. InfoSec says that you can’t trust the document alone. Even if the document refers to a bunch of other documents, and there’s a set of documents that refer to the document you are considering… it could all be a fake. So you need to consider the sources - who told you about this document? Are they trusted?

Well, I think I’ve trodden this ground before.

However, if the problem is that free rankings generate cash (through sales for high ranking sites), resulting in organic listings being corrupted in a negative sum game… Is there a way to change the rules altogether?

Now, I’m definitely not claiming that this idea is fully worked out. I haven’t discussed this with anyone else, so the thing is probably more full of holes than a holy thing, right now…

If you were going to have to participate in paid link strategies (and paid postings and paid blogging and site sponsorships, etc) in order to be listed, you were committed to paying for meaningful traffic. So why not allow search engines to remove any commercial site for that search? That’s right. When AdWords adverts appear, the price paid is a consequence of the interest of the organisation’s preparedness to bid for your attention. It is competitive, not a zero sum game, because spammy adverts *cost* and so vendors will tend to clean them up.

So just lose any commercial sites. Any site that sells anything, any site with sponsored links on the page pointed to… Just lose it, if an advert appears.

When doing an informational research, you’ll get informational sites. When doing research into buying stuff, you get commercial responses from people who want to sell you relevant products. It’s just that the commercial results responses will be precisely that - adverts only.

It’s radical. It might even work. But unless people start trying some completely different systems, breaking the current models, any search engine will end up with spammed results. It’s inevitable, with the current setup.

Material Disclosure

We do some White Hat SEO. We have not (yet) executed any SEO that involved paying for a link, other than a paid, human moderated, directory service (e.g. Yahoo directory and similar) or equivalent (e.g. professional association member directory - where the payment may be intrinsic to the membership fee and an explicit benefit of membership).

We build sites, and have developed a Content Management System, that attempts to follow the Google Webmaster Guidelines. SEO is therefore something that we do, and pay attention to. Philosophically, we’re oriented with building sites that intrinsically interest and engage with users… but that’s inherently difficult for some businesses.

However, as time goes on, I find myself staring at results pages that are clearly the consequence of building rank by constructive means (e.g. buying low cost economy labour to contribute postings to forums and write articles, non-reciprocal link exchanges, etc). As I see how intrinsically corruptible the systems are, my resolve to wear a White Hat is becoming weakened… reinforced when clients really don’t care what is done, so long as they don’t get banned.

I am seriously considering turning to the Dark Side. Maybe…

We also do paid search. At this point, I feel my work on paid search is more likely to lead to relevant advertising than Black Hat SEO. Of course, Google’s eating my lunch on paid search too, but if you play in their playground, you have to expect they’ll write the rules.