Endgame for Organic Search?

I’ve been busy for the last few months, and suffering from a seriously unreliable ADSL connection. When I’ve had time online, I’ve been busy meeting customer commitments, too busy to update the blog. Time spent muttering about making things work leads to some strange insights. Keeping to the consensual view isn’t helped when I read stuff like Search Engine Watch’s news about the search landscape, or this synchronistic interview with Matt Cutts. So here’s my late predictions for what the coming year will hold.

The last few months have been mostly SEO Strategy work, with some chunks of Paid Search. It’s been, if not apocalyptic, then perhaps a Damascene conversion. I think I’ve seen the endgame of organic search as we know it. I’m pretty sure that I’m seeing, at least for those markets that Google dominates, the end of SEO as we know it. It could even leave us with the peculiar position that paid search ends up offering the best search results, unless there’s a dramatic change in the way that search engines build indexes.

A Brief History Of (Internet) Navigation

I’m collapsing a lot of detail here, in order to pursue the basic drives of the industry. It’s currently an arms war in which search engines deploy new strategies to find sites that users like, and search marketers find ways to use those mechanisms to get their client sites highly ranked, whether users want them there, or not. Feel free to argue the points at which I’ve elided history, but I’d rather that you focused on the escalation of the game and the likely endgame that results.

Where we are now - a practical example

Take a look at the page of search results for sunglasses, and consider whether most of those currently ranked, would be there without SEO at work - then consider why there are some sites in the search for sunglasses eye protection. That’s an easy example of the gaming process at work.

This fractional image of a highly ranked site under sunglasses, also appears under “sunglasses eye protection”. This is *the* page about eye protection, on the site; note the relative length of the text versus the much, much longer list of adverts on the left - what’s the motivation for this page? It’s a condensed summary of easy to collect advice. Hardly authoritative. Not exactly the resource you’d expect to find well up the first page for a search query that is probably intending to get some real information. A decision to rank here will have been taken, and links and copy tweaked through this site and a bunch of other places across the internet, to ensure this kid of result. A nice SEO job… except… what about the user experience? How have Google’s algorithms and human experts missed the large count of articles and postings that put this brief and not especially useful article here? The answer is probably “volume and variety”.

History repeating itself.

In 1994, the way to find resources on the internet was obvious. It was Yahoo. This was the monster directory of sites with the cutting edge, the tools, the buzz. There were also-run search engines, and by late 1995 we were looking at AltaVista, Lycos, HotBot and their peers, and wondering how they did it. We were also wondering just what algorithms could result in some of the disastrously useless indexes that were returned.

The answer was laughably simple. The Search Engine Spiders of that era crawled sites and read the pages and inferred what the page was about, trusting the page content. Then they fed back the list of pages. Search Engine Optimisation was an unknown discipline, but getting to the top basically meant using a clean information architecture. For example, I turned our database of product specs into flat files, correctly formatted, and we rocketed to the top of the listings for every product item and manufacturer. Within the search marketing community, pretty much anybody who wanted to get the relatively small volume of search users was using metatags, and on page information. You couldn’t reliably guess that the best site was the one at the top of the listings, because it was so easily gamed, but the value wasn’t high, except for new small businesses.

Of course that meant that by around 1996 the metatag was becoming fairly useless. The porn guys, usually the first to work out how to make money with edgy new techniques, had pages stuffed with metatags. Search engines once again started to return pointlessly useless web pages.

Next step was to use information about which page linked to which other page. The big bad monster in this was Google. Little by little this geek-favourite drew more and more people to use it. Then of course, people realised that you could tweak page rank. The era of the Google Bomb, or Link Bombing, began; it was accompanied by the development of sites to help build page rank, going by the names of “gateway pages” or “envelope pages”, etc. Search was now a big industry, eclipsing the directory star and putting Yahoo into a shadow from which it still suffers. Search Engine Optimisers started appearing and immediately started fractioning into two main camps - the Black Hats who “game” the search engines, working out what techniques the SE’s use that can be exploited for “unfair” ranking, and the White Hat SEO’s who attempted to boost ranking by delivering what the Search Engines, and possibly users, wanted.

So Google modified Page Rank - it changed the way that it weighted links. It’s been continuing to tweak the old idea of page rank. For example:

• zero-weighting reciprocal links.
• zero weighting link farms.
• zero weighting large volumes of links that suddenly appear.
• giving new sites a fallow time to establish their credentials - the sandbox.

All this tweaking of the original idea is making the once neat and clean idea of Page Rank a rather confused and cluttered object.

It’s got worse in the last couple of years. Blogs have risen. The authentic word of the masses. Or of, ohh, 0.1% of the masses, or thereabouts. This is the real stuff. This is what users want and are interested in. So Google now pays lots of attention to the blog. Every business has or aspires to a blog. People get paid for product placement in blogs.

I’m now seeing high-entropy (randomised text) blog pages, generated by machine, intended to convince the SE’s that this is rich, high quality information, with relevant links embedded.

These blogs won’t, mostly, appear in search results. They pass their weight to other pages. But it’s a new way to game Google. Filtering it will be hard. I suspect you need a new step in AI capability (natural language processing) to infer whether the content is semantically meaningful, rather than just using words out of the same conceptual grouping, with plausible grammar. It’s my understanding that this is still regarded as a hard problem in AI.

Endgame

Google can keep adding filters, but for every addition of a filter, there’s a new path that is optimised, and every newly optimised path to ranking offers the opportunity to game the system.

Problem is, I think we’re looking at the end of useful tweaks to the filters. Without a radical rework of the whole idea of page rank, this looks like the dawning of the end of page rank. Once I can rent a computer program that spews out spurious blog pages, with links to real sites, I’ve gamed Google, in this round.

Google’s got nowhere else to go. Not without transforming Page Rank, rather than fiddling with filters.

It’s the example that senior and systems programmers often talk about. If you want a performance improvement, you need a new algorithm, not to optimise the existing algorithm. Google needs to stop trusting corruptible web pages…

What’s Up, Doc?

There is a way to go, but it’s very Web 2. It’s the users. There’s nascent systems already out there, but most of them are susceptible to automated manipulation. I suspect that we need to build a web of trust.

Make it so that only trusted users can confer weight, and they can grant a trust relationship to third parties.

Then, a page rank system can work again, with the weight coming from trust relationships.

Can you build such a system from the current one? Hmm. Now there’s an idea.

Who currently owns systems that offer trust rank or could be reworked to offer trustrank? eBay and eBay’s Gumtree? Google’s Orkut? Amazon? Delicious? Technorati? Wikipedia? DMOZ? LinkedIn? FaceBook?

How do you feed back to the search engines that various pages are trusted by trusted users? Use page bugs (web beacons)? Browser mods? Simple frequency of use (rely on trusted users to avoid visiting pages they don’t like)?

How do you square this with privacy? Feeding back the data relies explicitly on knowing who is offering the opinion. If you don’t offer trust to the search engine, how can you trust what the search engine tells you? How do you tell the difference between a real user and a Turing pass-off, or even a robot-in-the-middle attack?

How do you monitor a system that is based on user trust? Can you cross check rank, and see whether rank derives from a single, perhaps corrupted connection? You only offer rank where there’s a web of trust? What if there are several suspicious entities in the web, can you identify those who offer untrusted pages? Is assessing someone as untrustworthy to trust page content the kind of thing that will wind you up in court for libel or slander or some other legal principle?

What about internet researchers? I visit a bucketload of pages that I wouldn’t send my friends to, or at least, not to look at the content, but how the content is structured to make the page a success or disaster. I wouldn’t want these pages to destroy my trust rank, or for my trust rank to be offered to the pages.

Trust the user… Somehow. Without upsetting people about their trust rank score, and without putting internet researchers into a grey limbo, just because they deliberately look at the strange corners of the web.

Is that it, for Organic Search then?

Well, there’s a bunch of other things… Video Search - try finding Eric Schmidt’s answer to a question about click fraud without viewing at least 30 minutes of conference that you don’t need to see. Intent - is this user trying to buy, or just do research? There’s a few other nasty messes in search. This’ll do for a posting, though!

Oh, and I made a reference to Paid Search presenting more honest/useful results. There’s strong measures in AdWords, and improving measures in Yahoo/Panama to make sure that advertisers only appear on relevant searches. As that profit pressure strengthens, there’s a commercial pressure to make sure that your adverts get shown on only the right search queries. SEO has the same economics as spam: the pressure is to spam as much as possible so that some of it will stick, because the costs are low and the return can be high. Paid search will not, of course, result in adverts for the research papers that I look up, whether technical, economic or marketing - unless Google embraces and extends its current social advertising program.

Next up - more on Google AdWords Macros and probably on the effects of losing the bulksheet download.

Updates

2007-03-20 - I finally noticed Matt Cutts interview in Read/Write Web, touching some of the same topics and coincidentally drawing the same analogy of page rank and trust…

2007-09-05 - added more detail (screenshot and description) of a nicely-SEO-ed, but probably inappropriately highly ranked “sunglass eye protection” example. Tweaked some text for readability, and added some more sites that might have the trust mechanisms in place to allow identity based relevance and citation.