i-Lab 'Web Presence' slides

I realise that a naked PowerPoint isn’t much use… But until I get around to shortening what ended up as a nearly two hour talk, with Q&A… here’s the slide set. It’s a QuickTime movie, click to advance, or use left and right cursor keys to move backwards and forwards.

As well as this, I promised the group that I’d publish some resources. Those’ll come in some further postings.

An Analogy

Recently, restaurant revenues have been generally in decline. Is this because people are finding restaurants misdirect them to food?

Or is it because of another, perhaps slightly more obvious, factor?

What about people optimising their budgets?

Is there any reason to think that businesses do not also look to their operations in a recessionary environment? Would the shareholders be out in lynching gangs if a business was to ignore a recession?

Why Is Advertising Really Decreasing?

It’s a recession.

That’s it. When the marketing budget goes down, what do you do? You focus money on the most effective items.

Paid Search typically is icing on the cake. Not always – it depends on the business and I can cite companies where this isn’t true, too – but we’re looking at industry wide factors. IME, Paid Search is most often used to add a chunk of business for an established site. They could drop the advertising, and still see leads and purchases. So when they are keen to save money, it is an easy line item to drop.

In a recession, users spend more time looking for bargains, and click more. Conversion rates typically decrease, and ROI decreases. What is the right response? Decrease spend, and improve the site.

Instead of spending more on advertising, companies will direct effort to Search Engine Optimisation. SEO is an investment. Over months, you keep working on improving relevance, increasing user satisfaction, helping users to link to your business, developing low cost business relationships, and perhaps using grey and blackhat techniques – depending on the type of business you run and the risk you will take.

You also invest in site improvements – for a large company, that is cheaper than paid search. By optimising the site you improve user satisfaction, you can improve search engine rank, and you can decrease the need for paid search.

Strategic Impact

Two deceptively simple-seeming numbers tell a strategic truth.

In January 2008, about 90% of our leads for new business were primarily people wanting AdWords management.

In January 2009, about 80% of our leads for new business are primarily people wanting SEO and Conversion Improvement assistance.

Most of our AdWords clients have reduced spend. A few have increased their spend. But nearly everybody looking for extra business is asking us to find more economical ways to win business. That means looking at other channels, such as FaceBook and Twitter. It means exploring the use of blogs, and discussion forums. It means asking for feedback on the site content. It means looking at what users do and don’t do, and working out how to provide them with the best answer. It means improving site navigation and usability. It means offering up to date content.

Misdirection and Satisficing

The *worst* part of the article was a complete misunderstanding of how search engines, especially Google, work.

Google makes money because it doesn’t focus on making money. It focuses on satisfying users. Because Google has a large number of users, it is more appealing to advertisers. If Google presented search engine results that *didn’t* satisfy users, then users would seek other search engines that offered a better result. Advertisers would seek other opportunities. That has a direct impact on revenue. How?

The more advertisers there are in an auction, the more Google makes in that auction. If just one company bids, and users like the advert in large quantities, the advertiser can pay $0.01 per click. If there are two advertisers and users think both of the adverts are pretty good, then the average cost per click is higher – how much higher depends on the bidding strategy of the two companies.

However, if only one company bids on a keyword, and search users hate the advert – either the CTR is low or the advert is misleading – then Google will disable the advert. With a sufficiently unpopular advert you can bid $100 and your advert won’t be shown, even if no other advertiser is advertising on that keyword.

This is called “satisficing“. Google has created an auction mechanism that allows businesses to compete with each other. It has also created a system that allows users to vote on the acceptability of adverts. Unacceptable adverts – things that users hate – will eventually disappear even if the business is prepared to pay outrageous amounts.

There is no misdirection in that system. None of the parties may be completely happy, but none are unwilling to be a part of the game. That’s satisficing in a nutshell.

Misdirection Can’t Satisfice

So a central plank of the argument, that advertisers misdirect, simply isn’t borne up. If you have any real experience of paid search, you’ll know that you can decrease your Average CPC on Google by delivering improved adverts, better landing pages, easier purchase paths, links to supporting information, better customer service – *NOT* just by bidding more. Bidding more, in general, reduces ROI. Bidding management as a sole strategy is a failed idea. It was true in 2002. It is obsolete as a strategy now, and has been obsoleted for years.

Ignore The Rest

If someone pontificates at such length about something they don’t understand, I can’t read the rest with any sense of belief.

I *do* know something about PPC. I’m an AdWords Help Forum Top Contributor – I’ve read *thousands* of advertisers questions about why their adverts don’t work. I have free tutorials on how to improve adverts – which people say work for them; all the tutorials focus on increasing search user satisfaction. I’ve spent, on behalf of paying clients, budgets of up to $500,000 per calendar month – each advert leading to a specific landing page that answers the search query, with up to 4000 unique adverts flying at the same time, to bring users to the highest satisfaction landing page.

I’ve advertised, on behalf of clients, on competitor names and seen low CTR and low conversion rates and reducing Quality Score. I *know* that Google pays attention in organic and paid search results, to user satisfaction.

I’ve seen too many examples of clients in distress, asking for help, where they haven’t understood that they can’t just pay to appear.

Summary

Paid Search is declining because you get one sale from about 1% of clicks; it costs for each of the hundred clicks (or whatever the rate is – we see between 1:5 and 1:500 for various clients). In a recession, you can save costs by reducing items that are per-sale. SEO is not a per sale factor, it is closer to a capital investment; you can weather an economic crisis by living off the organic search results or investing further during the downturn so that as the economy recovers, you are better positioned for increasing user interest – but you aren’t paying more while users shop around.

Almost every single client that we have is working on improving their site, and improving their SEO results as a consequence. A year ago, they’d have spent money to bring in leads, and spent less time improving the site. That’s why PPC spend is down. Not because users are misdirected, but because advertisers are actually focusing on improving performance in a downturn. It is often cheaper to address user needs than to advertise and an improved site is improved for some time – whereas a click is transient. Pay Per Click appeals less than Conversion Improvement and SEO. That’s it. That’s the story.

Pretty flaming obvious, IMO. Oh, gee, I wrote about 2009 strategic advertising responses months ago. In a prediction. Hoop de doo.

Update

Danny Sullivan versus Eric Clemons in TechCrunch.

• Google will struggle to retain revenues using a variety of techniques
• Searchers will spend more time browsing and convert after more clicks
• Online revenues will generally increase – but business margins will be squeezed
• Internet Theft Scandals – Click Fraud, Phishing and Account Theft

Details and the consequences? Read on…

Google Will Adjust To Preserve Revenues

The basic idea of paid search is simple. Searchers submit search queries, and advertisers pay to have their adverts shown to more or less interested searchers. This meshes with the purchasing process (buying model) in several ways. At the most basic, when someone searches for your unique trademarked business or product name, it is likely that this searcher is intending to buy something. More likely than if they’d typed a competitors name. When they search for something that describes the product category, they are less likely to buy – they are in an earlier stage of the buying process.

Auctions have some basic characteristics. As a broad generalisation, the more bidders there are in an auction, the higher the revenue for the sellers. So Google’s goals have to include increases in the number of bidders in an auction – Broad Match and adding ever more advertisers all help with this. Convincing advertisers to bid high – traffic scales heavily with position; Google needs to convince advertisers (or, rather, prevent them from becoming aware of the relationship) that position and click quality are not related, but that position and volume are related. Finally, Google needs more outlets, so that advertisers see more reasons (more impressions) to be part of the ecosystem; however, that largely means adding new, smaller volume, publishers – some will have niche specialist interests, but overall this encourages more Made For AdSense sites, which, IME, have an appallingly poor conversion rate and value (reducing click quality by adding more outlets).

You’d expect that in retaliation, advertisers would then want to stick to Exact Match and using only Google’s Search Pages. There’s other factors at play, though.

Users miskey – they can’t remember the name of the company properly. So someone looking for the travel company Thomas Cook might key “thomson cooke”, “tomas cuik’ and all sorts of other variations. This means that Exact Match isn’t enough to identify all people looking for the trademark or brand name. Broad match helps by allowing Google to send these near misses to the right advertiser. Google do a phenomenal job of matching; if an advertiser like Travelocity doesn’t use “cheap holidays” as a search term, then Google will match a high bidding Broad Match keyword to that search.

People will also type more specific searches, longer searches; most searches, by a long margin, have more than two words in the search query. Users will type “logitech mac support” or similar, to more rapidly jump to stuff they know exists somewhere in the vendors site. Some of these longer searches won’t lead to sales – as in this example, some searches are support inquiries. So Exact Match also fails as being too specific, because these search queries wouldn’t be matched. Phrase Match is useful to find unexpected variations and more specific searches – leading to the idea that you can help users to jump deeper into the site with the right combination of technology and advert. Again, typos and miscomprehension by users will mean that Broad Match captures additional searchers who intended to find the business, but would have failed with solely Exact or simple Phrase Matches in the campaign.

Google’s problem is that when someone has decided on “Honda”, or “iPhone”, the searcher is unlikely to be deflected by alternatives. The consequence is that there will be few competitors for a brand name. Brand names are usually the best return on investment – so the average costs per click will tend to decrease for brand names. This is part of what you’d expect from a basic analysis of the buying process – you generally type a specific company name after you have researched and typically just before buying. So there is an inbuilt pressure to decrease advertising on exact matched competitors name – resulting in lower revenues from trademark terms. That’s a problem for Google.

The earlier phase searches, which tend to be naturally higher in volume as people look for alternatives from which to choose, naturally imply a worse ROI – you need to pay for more clicks to support the research, and the conversion rate is lower because searchers are researching. This too tends to drive Average Cost Per Click down, in order to retain a positive return on investment.

Google needs to keep advertisers focused on Broad Match. This lets Google place unsold inventory, offer competitor adverts on trademark searches, and so on. The goal for Google is to increase the number of participants in the auction (helping increase average cost per click and hence inreased revenues), to find ways to increase advertising on unsold inventory and to affect the minimum bid.

Some techniques that are likely to be used by Google?

Expect to hear more about the difficulty of reaching interested searchers with organic search, now that personal search is becoming more deeply embedded. If advertisers think that they are missing audience because of personal search, that they can only reach with paid search, then there will be more advertisers and increased competition. This will be partially driven because of the effects of recession… in some market segments there will be reduced interest, and this will naturally translate as reduced search volumes, resulting in fears that personal search is sapping share of voice.

Expect to hear more about Quality Score changes, probably surrounding the calculation of the minimum cost to appear on a page. This is not the same as, but is confusingly similar in name to, the estimated cost to appear on the first page. The intention will probably be that if the CTR is lower than some target value derived from the rest of the network, that the minimum price is adjusted to meet some revenue goal – it won’t be written like that – I can’t guess how Google will describe the technique, other than that is likely to involve some description about “improving user search experience”. There will still have to be a way to allow $0.01/click – a feature that attracts many advertisers, few of whom have any real chance of achieving this… but it remains an important differentiator over Yahoo and the other competitors in this space. I think it is possible to offer both, because you only get $0.01 on high volume, high CTR keywords – IOW, not things that affect most advertisers.

Expect more dilution of “real search”. Google still wants additional outlets and will continue to re-present domain parks, “fake search”, selected content match outlets and other “opportunities” as if they were what naive advertisers expect from keyword search (that is, that the advert is shown directly in response to a real user search that leads to a directly relevant site – much as happens with organic search results). Google will want the increased impression volume and the *overlap* of keywords with different intent that allows a single outlet to have larger counts of competing advertisers.

Expect more messaging about the synergy of paid and organic search – that even when you have achieved page dominance for the targeted keyword, you should still be advertising as well topping organic rankings. This increases competition, ensures that there are at least ten plausible advertisers, etc. I expect the messaging around this to increase as advertising volume and value shrink.

Expect more overseas advertisers. Google doesn’t offer USD bidding to smaller overseas bidders. The exchange rate isn’t notified to overseas bidders. So, especially when currency exchange rates are in flux, Google can make margin on exchange rates. I expect that Google can easily and all-but-undetectably collect additional revenue on overseas bidders, by tweaking exchange rates – AFAICS, the auction is held in USD, so exchange rate changes are needed for anyone not using USD. This would be non-US income – no-one in the USA will blink at using exchange rates to improve revenues from non-US companies.

Radically – if Google were to remove Exact Match, they could make dramatic transformations of revenue expectation. Would they do this? I think they would, if they could claim that it improved the search users experience. If advertisers fail to appear on a substantial fraction of searches, especially when those clicks turn up in organic search clicks, it would allow Google to say that exact match was preventing searchers from seeing the results that they want to see. This would be a huge step for Google, and they’d probably need a lot of research. It’ll be piloted by some large accounts and evidence will probably be drawn from wide-category vendors, like eBay. It’ll turn out to be complete rubbish for narrowly focused vendors, but Google’s objectives are maintain share of search, and only secondarily to satisfy advertisers – the auction will take care of some that advertiser anxiety.

Increased Search Time & Reduced Conversion Rates

Most people are planning on reducing spend. They’ll do so by spending more carefully. They’ll look around more. They’ll be taking more personal recommendations. Trust in the stability of large organisations and well known brands will continue to be eroded – which means that the right businesses, with the right trust validations, can emerge from nowhere; just because your business has been running for 5, 10, 50 or 150 years is no guarantee that you haven’t recently based your business on false expectations of investments and earnings. A new business with the right accreditations can play on the same field as a 200 year old business, and can raise fear and doubt about the financial stability of well established players.

So, shoppers will be more wary. You’ll need to improve your sites to have online messages address current concerns. The fears, uncertainties and doubts that were addressed in marketing communications last year, won’t work so effectively this year; the concerns are different. Your customers will be concerned that even buying from a major brand will damage them, or is at least risky. Reassurance and validation will be important – especially with the media focusing on bad news. Bad news sells and media outlets will want more viewers. The at-risk media outlets, old broadcast media, seeing declining share of advertising budgets will want to stimulate sales through increasing interest in controversy, further weakening consumer confidence. Failure to address offline media reports online will impact conversion rates – though you will want to avoid directly feeding the controversy.

Minor brands should be able to gain, if they are cash flow positive or can find investors with imagination, foresight and cash. Guarantees and warranties, future-safe products and ways to reduce service costs for consumers; ways to manage domestic finances effectively; ways to cut back while still having luxuries.

The results of these will be to put some strange pressures on paid search. Companies that have previously survived on word of mouth and low or zero advertising, will need new customers. They’ll advertise. As new advertisers, they’ll make a lot of mistakes and won’t be willing to pay for expert assistance. That will increase claims of click fraud, and apply upward pressures on bids, sometimes from advertisers that shouldn’t be in the auction.

Existing advertisers will want to improve performance and will decrease budgets and target the spend on the most effective keywords and adverts. This will increase pressures on the paid search companies to dilute the inventory – the result is likely to be that overall clicks/conversion will continue to worsen, otherwise the paid search companies will report losses, and decreases in search volumes. It’s always bigger news if a big company loses, than a bunch of smaller businesses – so expect that larger businesses will use their asymmetric control of information to manage smaller customer businesses expectations and margins, through manipulating click quality. Expect the search engines to protect themselves at the expense of smaller advertisers.

ROI will generally worsen, but there will be islands where specific companies have struck the right messages that resonate with users. The result will be that paid search competition will continue to heat up. I expect that the average CPC will decline, the total value of paid search will decline, but some advertisers in various niches will have justifiably higher Average Cost Per Click and increased spending.

Affiliates

The affiliate industry will also become even more heated. Out of a job? Looking to make money with a low capital investment? Then you could become an affiliate…

However, who needs more novice affiliates? What does an influx of untrained, cash starved and at least initially ineffective affiliates do, when the real super affiliates (not the one man and a dog operations blogging from “super affiliates are us”, but the real, quiet, and highly effective super affiliates) already handle about 80% of affiliate traffic? The answer is that these new affiliates will provide free advertising for marginally effective businesses, by spending their own money essentially as an investment. Expect more companies to switch to affiliate advertising models to control their in-house marketing costs and to reduce spend on advertising agencies. Expect a lot more annoyed novice affiliates.

The affiliate industry will boom; but leave a lot of disillusioned “internet advertisers” in their wake. There will be some emergent new stars – not everyone who comes in will fail, and some existing leaders will cash out and move on. Expect some churn in the top affiliate products – but that’s pretty standard in the industry anyway.

I expect that the real super affiliates will be the same next year as this year – they have developed their techniques, and they are effective. They might retrench a little and change their focus on the businesses they are interested in, but they’ll survive and probably continue modest growth; their main obstacle to growth has historically been cash; 60 and 90 day payment cycles on sales conversions mean that working cash is tied up, effectively limiting them to “four to six inventory turns” a year. That reduces their rates of growth, and with banks not in lending mode, these guys are bottled up, at least in paid search.

Online Revenues Will Increase

Well, they will unless the internet theft scandals are outrageously large. In a search for the best value, comparison shopping online and internet supported purchases over the phone will become more important. Where businesses aren’t already selling online, there will be pressures now to do so, to reduce the costs of sales. The final result will be that more is sold online – even though total sales from all sources will decline. There may be some countries where there is a decline in online sales, but it will be less than the decline in total sales – the internet will be seen as one of the brighter spots. Just.

There’s only one reason for internet sales to increase. If executed effectively, you can reduce the costs of making the sale. That tends to be less and less true the lower the volume of sales; so expect continued outsourcing to places like Yahoo!Merchant Stores, where the infrastructure and development costs have already been put in place, and allows dropping the cost of sales. The interim step between often dreadful sites with no compelling call to action and a full online sales site, is a phone number; but placed on an inactive site that doesn’t reflect your best price and the unique value in buying from you rather than a competitor, this will have no effect. So expect the more savvy small business to look for CMS based websites where they can update their own content. I’m expecting that smarter SME’s will be investing more in templated CMS backed websites than the current static crop.

And metrics. Web analytics – free tools like Google Analytics – properly used, can reveal a lot. The problem for SMEs is setting up and understanding the data. There should be an increased demand for skills in setting up analytics, and interpreting what is happening.

Internet Theft Scandals

With money tight and some smart people out of a job… expect internet fraud to increase. With businesses looking for excuses to write off problem debts, expect more disclosure of problems, probably triggered by an inadvertently exposed online scam. This is obviously a tentative projection, as it depends on unforeseeable circumstances. However, this year is the first year since 1994 (when I started internet sales seriously), when I can see reasons to add excuses to the balance sheet, some understanding of the risks to businesses and that even corporations are affected by fraudsters, and a large value of online business that probably will increase (at least in relative terms), and an increase in technological capability by scammers and fraudsters. The combined pressures might make it attractive for the first time to attribute losses to technologically sophisticated thieves.

Online security for the average consumer has not improved in any seriously identifiable way since 1994, other than the progressive plugging of vulnerabilities in web browsers and servers. The introduction of Secure HTTP (https, or “secure server” technology) back then, provided users with a somewhat artificial degree of confidence. Many web sites offer access to financially significant resources with only an account name and password. Account names and passwords are insufficient for best security practices. Banks now often offer multiple levels of password and CAPTCHA, and may require authentication through encrypted PIN checkers – beginning to approach the holy trinity of security (“something that you are, something that you have and something that you know” – at least two of those are addressed by the better banks). However, there are still far too many ways to spend money online that are protected only by guessable account names and passwords, and by essentially unprotected mechanisms to send money to scammers.

There will be scandals about this. Probably soon. If there’s a sniff of a problem, then the offline media will pounce. These media need to pounce and will argue forcefully; their businesses are in decline, and they’ll need to savage the failures of online businesses to help protect their own. While this would be a significantly negative sum game for all online sales, the benefit of defection by an attacked business will be high; “We really made money, and we only show a loss because of internet theft we couldn’t control” will be an attractive excuse to some business, at some point. And then the skies will open and the extent of internet crime will be an issue.

That will damage sales. Hugely. Even if the effects are already part of current accounting systems, reported as part of business accounts and effectively managed by all online businesses, the *perception* of reductions in safety will be very damaging. I don’t see any sign of widespread adoption of even “best practice” account management – internet wide consistent login methods and messages, proper understanding of what a “secure server” means, and so on. While users are prepared to pay money to sites they can’t properly identify in return for promises to deliver products they don’t receive, across national borders (losing nationally accountable and interested police enforcement), this problem will continue to grow. The individual losses are small; the investigations complex. At some point, this will become a more serious problem to deal with than Enron and Madhoff. I think it’ll be this year.

Consequences

New advertisers will start by thinking about paid search – increasing pressures there. This will result in disappointment and claims of click fraud, especially if new advertisers discover that a click is not a click… different clicks have different values, even if Google conflates clicks from multiple sources. Expect a lot more noise and heat from new advertisers who feel they have been mislead.

Existing advertisers will mostly reduce and focus spend. AvCPC will have a downward pressure, counteracted by the SE’s including low value inventory – weakening conversion rates and hiding it in a generally slower market with inbuilt tendencies for longer latency and reduced likelihood of sales. ROI will get worse; the question is whether the business can take a few quarters of bad ROI in order to survive to an upturn. There should be increased attention to improving the web site, however, this will be diluted by the feelings of senior management that enough work has been done on the web site – “it’s a finished product” – and that there have been sales, so all that is needed is more visitors. IME, that’s usually wrong – there’s usually a lot of ways to make a site more effective in selling, even for long latency sales.

There will be a switch to increase SEO efforts. If purchasers will spend more time in research anyway, increasing the clicks and site visits per sale, then organic search results become more interesting. This will increase interest in spammy linking as people learn their way into search engine marketing. There will also be increased interest in affiliate advertising – if you aren’t skilled in internet advertising, then getting affiliates to advertise on your behalf will be interesting. However, this again should result in increased interest in web site design improvements; it won’t, because once a business have “put your brochure online” the naive perception is that there’ll be nothing else to do. There’s still far too many naive advertisers with ineffective web sites and a management team that focuses on volume of traffic, not the quality of that traffic and the quality of the response from the website.

There will be a decrease in search volumes for products. Despite the increased research for purchases, search volumes for commercial products will be negatively affected. That’s partially because the economy is slowing, but mostly because people are learning the online environment; they know where to go to get various things at decent prices, so they don’t have to do so much searching and researching – I haven’t heard anyone complain for the last 18 months that they are “useless at searching”. This year may be first year where the combined effects of knowing your internet neighbourhood and slowing commercial activity, actually dent the growth of both paid and organic search.

A significant internet theft scandal will cause a dramatic decrease in online consumer spending, if it bursts. Expect new web browser technology and new web site authentication services. At the moment, I’m pretty sure that the only way to tackle the identify theft problem is solutions that, as an industry, we’re nowhere near considering. The short term consequences will be to push the internet clock back towards use supporting research for purchases, not for actual online purchases.

CMS’s will become more important for SME’s. They’ll finally be able to build customised landing pages, important for improving the effectiveness of paid search.

Web analytics, neglected or misinterpreted, will increase in significance – and Google Analytics low price point makes it a very attractive platform to use, despite the poor general knowledge about how to best use it. The downside of “free” is that expectations of the value are low, and users are unwilling to pay a lot to understand the benefits; smarter businesses will invest in learning how to use GA effectively (IOW, not as it comes out of the box).

Blog spam is interesting for several other reasons, of course – not least being that there is a widely used technique to de-rate comments – the NOFOLLOW attribute for links. As with a bunch of other bloggers, I tend to prefer using spam rejection tools rather than nofollow. IMO, it’s better for users to see real comments than to (ineffectively) defuse spamming efforts with NOFOLLOW. CAPTCHA is moderately effective, but still places a burden on real users – and also still allows outsourcing spam generation to low cost economies.

As should be quite obvious (try to submit a comment) we don’t use CAPTCHA, so we get human and machine submitted spam. A lot of machine submitted spam. I normally only see it when I read the spam queue. Yes. I do that. OK, I’m wierd. I like to know what the buzz is. If big companies are sponsoring spam, knowingly or unwittingly. I discover all sorts of stuff – some of it seriously unpleasant.

I wondered whether there was a way to use machine submitted spam to measure blog rank. I couldn’t find anyone writing about measuring spam to individual blogs and the implications for the blog. The articles probably don’t have enough search rank to appear… I did find a recent article about spam and blogs and particularly about the value of closing comments to old articles. I’m not entirely convinced by the article – although there’s clearly a lot of research gone into it. I think there’s a key point of disagreement – Lorelle VanFossen asserts that blogs are found by robot crawlers, not by people using search engines. She says that blog spam doesn’t start until there is an inbound link. However, AFAICS, it’s the inbound link that causes search engines to rank blogs and allow the results to appear. It seems to me that the evidence for the link being involved is high, but I infer that the link gives pagerank… Something else I need to add to the research queue, I suppose.

Here’s an extract from an email that we’ve recently received, offering me low-cost-economy “link building”. Note the emphasis on relevance. IOW, this is built using search engine results, not webcrawling. This is typical of the unsolicited proposals that we get. These approaches clearly work or they wouldn’t be using this kind of text. They aren’t offering traffic directly, but rank, and they are clearly using search engine results to identify the targets.

My concern isn’t so much how spam starts, as the changes in volume. And that, I think, casts an incidental light on origins. If spam is a consequence of rich inbound links, then spam volume will tend to increase as a blog gains links. A model based on inbound links as the source would not tend to show decreases in volume. Yet… that can happen. Blog spam can decrease as well as increase – just like your other investments.

Why does spam volume change?

The point of spam is usually to improve the page rank of the targeted content, and partially so that readers will click on the links (direct traffic generation rather than indirect through rank increases). Spammers look for high page rank blogs that accept comments, especially when the area of interest is close to the spammy link to be planted.

Monitoring search engine queries that lead to this blog is often informative. Several times per day, I’ll see a search query like “internet marketing blog comments”, “shoe blog comment”, or “adwords blog add comment”. These guys are not looking for a substantive blog in order to join the community – they are looking for a blog that has reasonable page rank and offers the possibility that their spam will be shown. Because I’ve used shoes in examples of advertising in various articles over the years, this blog used to be quite highly ranked amongst shoe blogs. Amusing.

You can see this effect on other blogs that offer no moderation or no-despamming of submissions. Here’s a screen shot of part of a page on a fairly new blog that currently has low rank, and no significant protection:

So why does that blog page get no comments for weeks, and then suddenly have hundreds of spam comments? Why can some blogs survive for years with a handful of spammy comments and no identifiable protection, while other blogs get thousands of spam comments submitted, even though there are none or few visible on the blog?

Spam is non-linearly proportional to rank.

Once the observation has been made, I think there’s a pretty obvious interpretation:

The higher the page rank of a blog, the more that it will be spammed.

There is other evidence. An odd quirk of cut and paste in Google’s results used to lead to an embedded space in the search query. We’ve seen this signature every so often, in our web server logfile analysis. In addition to the obvious spammers looking for a blog to dribble on, there’s an even smarter crew doing product specific searches and then cutting and pasting to get to the site.

From observations of this blog and those of client accounts, the particular spam rejection technique seems irrelevant. Whether a “silent killer” like Akismet is used, or one of the CAPTCHA implementations, or the religious use of NOFOLLOW, spammers will try to submit spam. This implies that it is cheaper to submit spam to a lot of blogs, some of which are protected against it, than to develop the software that verifies whether the spam is accepted and published.

I had a look at web server logfiles, to see if there was any pattern between IP addresses that might indicate a separate spider followed later by a spammer. I can’t see one. I didn’t really expect to do so. Botnets, dynamic IP addresses, anonymising networks – the only reason to be closely correlated these days would be if you *wanted* be found. The nature of automation is such that there should be no real reason why a technique like measuring packet time of flight should work – because there’s no significant time dependencies involved. So I gave up on the idea of genuinely identifying and correlating the sources of spam…

The essence appears to be that spam targeted blogs are found, not by crawling the web, but by using search engines. This is fairly smart – because blogs that are higher in search engine results are obviously being crawled by the search engine spiders, and will contribute more to weighting, than a blog that isn’t even indexed by the search engines. Different spammers will have different search queries for finding blogs to target, but the overall effect will be that higher rank blogs will tend to attract more spammers than lower ranked blogs – because higher ranked blogs will appear in a wide range of results for different but related search queries – kind of like a Levenshtein distance of blogs. Higher ranked blogs will tend to appear on a wider range of searches, as well as being higher in each set of results – so there is an intrinsic non-linearity, where higher ranks attract significantly more spam than would be expected. There is a selection based on position – the higher the position the greater the likelihood of being a target – and on breadth of matching (a higher ranking blog will tend to be more authoritative about a wider range of stuff, with the way that Google ranking works). Hence a massively increased interest as blogs get higher ranked. Or a significant decrease in attention for minor slippages of rank.

Changes in Blog Rank

Changes in search engine rank appear to drive changes in volume of spam. The question is, what provokes changes in rank? This is a standard exercise in SEO, at least as I practice it.

I’ve used this blog to experiment on the factors that make a blog more, and less, interesting to users. I’ve also got experience of our clients blogs. This experimentation accounts, in part, for the frequency of postings here, and the variation in types and quality of article. A few articles (which I am in the process of marking up) are of dubious quality – the question in my mind was “will readers call me on it?”. Some articles are clearly antiquated – events have superceded them – what happens to those? Does the style of an article dictate the number and quality of inbound links? Which is the most popular page, and the page with the most and highest quality links?

Experimental Results

Really interesting results.

Linkbait articles, with or without logical flaws or internal inconsistency, are the most highly linked-to, but receive almost no traffic today, only a few months later.

A long, detailed, one year old, technical article about gclid, and another about adwords conversion tracking from two years ago, attract the most consistent day-in, day-out traffic. The third highly read, long duration article is about AdWords Editorial Review, but nearly all the people who read that are *not* coming from search engines, but from articles posted in the AdWords Help Forum. That’s probably because Editorial Review is still a pretty specialist term, much less likely to get attention than the problems it causes :)

Commenters pay no attention to the date of publication. An article that was correct at the time of publishing, may be superceded by events, and will then receive comments as if the article was current.

Embedded links in either the top or tail of an article, referring to updated information, are almost completely unused. The average time spent reading this blog is typically more than two minutes – some days it hits more than 6 minutes as the average time, and only drops significantly below two minutes when we’ve been Stumbled. In other words, the articles are read, in depth – but less than 5% of readers even follow a primary attribution. If I’ve written in response to another article, the source is rarely clicked on.

One of the least clicked links, that I notice, is the one in the gclid article, near the end, referencing Cut-Me-Own-Throat-Dibbler. So far as I can recall, it has been clicked three times in about 18 months, despite being in the most popular article. A few dozen people spend about 4 to 8 minutes each reading that article, each day, or about 10,000 readers spending a cumulative 1,000 hours of reading (an entire solid six weeks of reading). Three in 10,000 = 0.03%. Not a lot of onward clicking, is it? OTOH, the idea of writing an article that has cumulatively taken six weeks of reading is pretty awesome and makes me feel guilty about article content… Important to avoid wasting that much time!

Rank is not significantly affected by even quite extensive article rewriting. Some articles have been revised over the years to reflect changing information. They get remarkably consistent volumes and subjects of spamming, before and after rewrites. Comments don’t appear have much effect on spam volume either. Highly commented articles are no more likely, AFAICS, to attract spam, than uncommented articles. An article is more likely to get comments, if someone comments shortly after publication. That is partially because most articles have a short lifetime. Once the lifetime is over, they get a few visitors, who rarely comment.

The comment policy page is one of the least visited pages on the site. I’ve written all sorts of stuff on there, over the years, mostly for my own amusement. It has, at times, been a draconic rant against spammers with severe words about leaving meaningful comments.

Changes in the comment policy have no effect whatsoever on spam volume.

Changes in the wording around the comment box affect the volume and type of real comments – but do nothing to dissuade spammers.

Properly constructed outbound links within articles with a FOLLOW attribute (implied by default), to credible sources, won’t damage rank, and won’t cause many readers to leave the site. If the article is interesting, they’ll stay with you. If it isn’t, they’ll leave rather than click on a link in a useless article.

Summary

Monitoring spam levels is an unusual and intriguing way to detect changes in search engine behaviour and the overall rank of the blog.

Linkbait, unsurprisingly, still works to get inbound links. That’s still a great way to get rank. Be contentious. Be provocative. Standard journalistic conventions for dispute and disagreement… Doesn’t engender wisdom, peace and happiness, but does get you noticed. If the article that follows is sufficiently provocative, it’ll get lots of links within minutes.

Article topic and treatment has a huge impact on longevity of traffic. A comprehensive article that solves user problems, lasts for ages and may become permanently popular. A newsy article has a lifetime of hours to days and then gets random low traffic.

New articles do more for rank than refreshing old articles. Something in the range of one decent (as in “attracts inbound links”) article per week, will be just about enough to maintain rank. Two would be better. I infer and have observed that search engines will rapidly give rank to a new article, but do not do the same for revised content for an old article.

IME, popular articles covering technical issues should be corrected in place – even if it means significant rewriting. Strictly, I think that re-writing should force a re-dating of the article. Possibly it means that the article should be migrated to “static” content on the main website rather than being a blog article. I’m still muttering to myself about this conclusion, so the advice may change. The reason is that users don’t follow even very obvious and repeated links to later and better information.

Readers rarely call out the author, even on internal inconsistency. That’s actually worrying. If there’s an intent to convey factual accuracy, then the responsibility falls hard on the author. Blogs aren’t peer review…

Use plenty of links to authoritative references and similar articles. Isolating your blog doesn’t make it more authoritative. Referring to other blogs binds you in to the community and generates more inbound links, which brings you more readers than you lose to the links.

Most comments that pass spam filters will also pass any reasonable moderation – but occasionally some comments are clearly intended for the author of the article, or are amazingly off-topic (but not spam – requests for help with specific technical issues only peripherally related, or to do homework exercises, etc).

Very few readers and commenters look at the Comment Policy. Have one on your blog, but make it liberal. Most people are either spammers or really interested, interesting and helpful. I learn a lot by looking at the comments. I’m grateful for the insights you share with me, and exposure of the problems of interpretation that I inflict in the articles.

Use a welcoming message near your comment box, rather than some aggressive statement about spam. Assertive anti-spam statements will dissuade some real commenters and will do little or nothing to change submitted spam volume.

Monitor the searches that lead to your blog – changes in the nature of the searches may reflect changes in how the search engines perceive your content. I’ve found Lijit to be a helpful tool – more helpful than FeedBurner for understanding searches. If you visit this blog frequently – once every few weeks, or so – you can watch how the search cloud changes.

Consider joining the No NOFOLLOW crew. Nofollow hasn’t stopped blog spam, but it has provided a mechanism to penalise sites outside the trusted set. It reinforces isolation of high volume and high ranking sites from the rest and does almost nothing to stem the tide of blog spam. I wish Google had put their efforts into publicly accessible spam filters for blogs and web form submissions, not an attribute that creates extra work for little perceptible positive benefit. Actually, that’s not entirely true. I’m pretty sure the tag helps Google. They’ve recruited webmasters and blog writers to do their job for them… identifying the garbage.

Updates

2008-10-21 – edits for clarity and typos. Added new illustration of typical link spamming UCE.

I don’t have the statistical databases of Google, or even a small search engine. I have an acute awareness of how I treat search, three kids and a group of people who I watch using search to find things. I also have had access or have current access to huge web server log files with data from paid search and organic search (and walk ons aka “direct” and email, and so on).

I’ve also, periodically, had formal training in marketing – paid for by me. I *like* this stuff. It’s my entertainment. OK, that might sound wierd, but I get more of a kick from figuring out why some marketing trick works than I do watching a film. It is more deeply revealing of real human psychology and the stories of our lives. Back to the point…

So, what do I see when I watch people search? What do the log files say? What measurements are we missing, as an entire Search Engine Marketing industry, because our web analytics tools don’t provide them?

The Buying Process

I keep mentioning this, and each time a few more people seem to get it. The Buying Process is pretty important in offline, but often ignored online. I think that’s a subtle consequence of how the web analytics got going and the ease of making and understanding certain types of measurements…

The idea of the Buying Process is to identify the kinds of activity that a prospective buyer goes through. By targeting the right message to the right phase, you can influence the course and speed of progress through the buying process. The offline and online models differ, I believe. In particular, the nature of online purchasing injects an extra step, that isn’t articulated offline. Here’s the basic online buying process for a simple sale (one person, small enough value to allow personal decisions):

• Needs Awareness – sudden or slow dawning that they have an unfulfilled want
• Research – Find stuff out about the perceived need, and refine it
• Comparison – read reviews, and get together the data and emotions to make a decision
• Decision – put the whole thing together and make up your mind to go for the safest option
• Acquisition – this is frequently combined with the Decision, offline; how you go about getting what you want
• Post Purchase Evaluation – do you feel good about your decision? Support? Usage?

Prospective purchasers may slide backwards and forwards between these phases. So, for example, on choosing something and trying to buy it, it may not be available or costs more than you thought – so you end up being thrown back to an earlier phase.

Search queries seem to evolve. Different users have different strategies for searching, but one common search usage model seems to map to the buying process like this:

• Needs Awareness – irrelevant
• Research – typically repeated identical searches and looking at a range of organic search results
• Comparison – typically a wider range of different searches, honing in on particular features and benefits, or problems; can be Organic, or PPC
• Decision – usually irrelevant
• Acquisition – searches for a brand, web site, or specific product name and attribute – Both SEO and PPC can help (multiple messages, increased Share Of Voice, etc), often focus on adverts – PPC CTR can exceed 60%
• Post Purchase Eval – only relevant if there is a problem, a support group or online interaction – Organic is terrific for this

Organic search tends to be great for that early phase and the post purchase phases. Using paid search in these phases can result in a low conversion rate – and a signature is that you get repeat visits from the same visitor, on the same search. Ideally, for early phase searches, you’d use the organic search activity to get someone enrolled in a newsletter or other direct marketing activity – so you can send messages targeted at their activity, with their permission. You can then avoid search, thereafter, in some cases – not all, though.

When you get to the comparison steps, depending on the product type and the nature of the comparison, PPC may work better – because the precise message is more easily shaped. Organic can act as a support – so if you know that an organic result is present to take the other traffic, you can shape the advert copy to help attract buyers closer to conversion, or who need a message that the organic listing can’t properly support – a segmentation exercise, essentially. You may also be able to get an advert up when your SEO won’t let you show a reasonable ranking.

At Acquisition phases, you can do really well with PPC – or completely crash and burn. If someone has typed a brand name, they’ve probably made up their minds. Your chances of deflecting that purchase intent are going to be very low. Even if you can get a non-zero CTR, the conversion rate is low… unless you can shape a message to a competition-killing weakness and support that with the right Landing Page. Note that SEO generally finds it hard to rank at all on a competitors name… and Google’s usage of trademarks may make this difficult to do in certain countries, using PPC. It isn’t easy, or always possible. However, when the decision to acquire is actually a decision to research where to buy, then you get a serious chance.

Effectively, when someone decides that they want a holiday in Cancun or a specific model of car, they haven’t decided to buy… They’ve gone back to Research and Comparison phases, but they are now looking for *where* to buy, not *what* to buy – they are now more likely to be looking at paid search adverts, because those can carry, for example, current prices, specific messages that you can’t or shouldn’t carry in organic results (unless using Subscribed Links – but that’s another story).

Paid Search shines here – and has another trick that is hard to manage for Organic. Now, you still need the organic listing present – because you don’t want to lose the traffic doing early phase research. So the paid advert can focus on buying messages. “Book online now” – leave the organic listing to offer “Search/Browse online now”. The advert often needs to imply that you are ready and waiting to take the order and you have whatever it is that people are looking for and you’ve got.

You probably don’t want to advertise for post purchase evaluation – but you might in rare cases. If, for example, you have people proud to use your services and products, you might let them use icons or graphics to display their affinity. This supports hard-to-measure viral activities – and getting the right page, right message and rank with SEO is fairly uncertain. Placing an advert will let you support after sales activities that may be otherwise difficult to do. Similarly, if important events have just happened, then Paid Search can usually give you a head start on getting the new messages out.

And that, of course, is the final significant difference in the communication strategic strengths and weaknesses. If your tactical needs are for fast changes – new campaigns, promo prices – then Organic is a generally more sluggish responder. This, for example, makes PPC a suitable tool to conduct practical marketing research into consumer interest – something that you just can’t sensibly do with organic search. Note that with busy sites, you might be able to get the Snippet updated, almost in real time – that’s actually faster the editorial review, without asking for an expedited review.

PPC Tricks

Some things that help PPC, are not easily available in SEO. For example, day parting (where you can specify which set of adverts run at specific times of day), and geotargeting – ostensibly reaching only your desired audience.

Depending on the business, these may be more or less meaningful. For example, in some sectors you may be able to identify that businesses are the primary users of certain queries in the morning, and in the afternoon the audience switches to consumers. PPC lets’ you tune the adverts. However, other products may not have such an easily deconstructed appeal.

Geotargeting is less of an issue for SEO – because exposure to foreign geotargets is less of a problem. Google has made a minor pigs-ear of geotargeting – confusing where you want to show adverts and where the service area is. For example, if you are selling Plumbing Services in Florida, you probably don’t want jobs in Washington State. The query “plumber” may net all sorts of SEO that isn’t useful for you – but correct application of geotargeting could yield a local plumber. IME, geotargeting has been so poorly understood and exploited that it isn’t much of an advantage for PPC.

Measurements

I don’t know any current web analytics tool that reveals the evolution of search for a purchase, in any useful way, out of the box. This “missing tool” means that people are focused on the “first click that leads to a sale”, and this betrays the complexity of marketing messages. As purchases become more complex, the number and type of different messages increases, and the communication method to get the right message to the right user becomes increasingly important. But even quite simple sales often involve several searches that lead to the same site, often in the same session – or I wouldn’t see this search evolution in single vendors web server logs.

Latency is also often ignored. Well, actually, I can’t cite any Web Analytics package that offers latency of conversion as a standard measurement, but I’ve only properly used four packages in the last year. Anyway, by adjusting advert copy, I know that I can control latency – how long it takes between a click and a purchase. That’s because the Advert Copy is selecting people closer to buying, or in an earlier phase. Latency is significant – for cash flow – but also because of what your site offers. Some sites support long latency sales better than others, so the right site may actually achieve overall higher profits by obtaining long latency clicks. Other sites will be focused only on supporting final phase processes and can’t properly support a long latency sale activity.

Summary

Paid Search and Organic have a somewhat false competition. The nature of the two is different, and complementary. Failing to consider either would be a strategic marketing error. The reason is that the tools attack different messages and opportunities in the Buying Process. Failing to use one or the other would be like not-using email – it addresses a different message in a different way.

The way that we, as an industry, think about problems is partially a consequence of what we think we can observe. There’s a lot of underexploited information out there, and a lot of stats that would be useful to marketeers, that we aren’t getting and probably aren’t asking for, from our web analytics vendors.

“If You Build It, They Will Come”.

Annoyingly, this model usually fails. Let’s have a look at how people think about their web marketing and why they end up either ravingly happy or cynical depressives about their efforts. Web Marketing, or Internet Marketing, requires a different set of techniques than those that work offline – and it is a learnable skill.

We started Merjis in 2003 and in the course of that first year we did a survey of web sites. We picked 100 businesses in the same English county, Bedfordshire, with a website. We looked at companies with a size between 10 and 250 employees, with a turnover of less than £25M (about US$30M then, about US$50M today). We checked first to see if the business was focused on the company’s desire to talk about itself, or the prospects’ needs to find out about the company’s products and services; what the conversion path would be for identifiable prospects; and finally how the company ranked for searches of its own name and for identifiable products and services (with and without a geographical location in the search query).

It was a terrific and illuminating piece of work for us. 98 out of 100 companies shared a common problem. They had put up a site that was focused on what they wanted to talk about, and that failed to present their goods and services in a way that prospects would want to know.

For example, for an engineering company, should the home page offer an unavoidable Flash walk through of the new factory? Probably not… it takes 20 seconds for the walkthrough, for every new and returning visitor – so it should be an option for visitors. The same company could have instead explained their ability to turn ideas into products in shorter times, or at lower prices, or with higher quality – or whatever it is that their clients normally emphasise as the reason to continue doing business with them – playing to their strengths and triggering recognition in prospects of the value and customer focus of the company.

The other thing that became rapidly clear was that well over 90% had no significant presence in the search engines, for any search that should have lead to them. This was caused by a combination of poor site design, and a lack of “current engagement” – they were not visibly doing anything in their industry and had no visible customer endorsement or interaction, nor were they apparently involved in any projects. Some of the sites had clearly been developed in the dot.com bubble of the late 90′s and not updated in four years afterwards. A handful of the companies had paid to appear in various directories, such as Kelly Search, but none were using the staple directory of that period, Yahoo! – at $300, much less expensive than most of the UK based directories that we saw. All that, of course, meant that these sites had really, really low traffic. The few visitors that turned up on the site would not have seen a compelling reason to engage with the business, after seeing self-obsessed material about the MD’s life history or similar poorly focused material.

These businesses had thrown money at the web and pretty much without exception would regard the web as a pointless waste of money. Further investment in the web would be likely to be seen as throwing good money after bad. But we knew, from almost ten years of previous activity, that new niche businesses could be grown to several million turnover, in a few years. What was the basic problem these companies really faced, and what were the components that could be addressed?

Categories of Web Site

We attempted to classify web sites according to the purpose that the site should and did serve for the business. We recognised that some sites would have multiple competing categories – so we might end up with more than 100%, after the categorisation. We were able to recognise:

• Vanity Sites – built to flatter the CEO or Marketing Director, these sites do nothing useful for visitors outside the business – this is the completely dominant category that we found, at 98% of all sites in the survey.
• Validation Sites – Showed that the company was in business and active in a claimed area – about 80% of sites needed a validation role
• Sales Sites – had products for sale on the web, with a store and payment system – in this sample, a handful
• Lead Generators – typically for brokers and other organisations with complex products – about 35% of the sites had something too complex to sell online or regulatory controls
• Subscription Sites – news, information and advice sites, possibly to a small paying audience, with lots of hidden content – not even a handful of sites in the sample

Very small businesses – often management consultants in various specialities – may have no easily defined service, other than being a specialised consultant. They often can’t talk about what they’ve done for a specific client, so the best that most of them can offer is some description of what they do and their claimed client list. Their web site acts as a verification for offline discussions and business cards handed out at meetings.

With defined services and products, larger business need to focus on selling whatever they have, in a way that suits the needs of their customers. This may differ online from the sales and selling mechanism offline – and that can provoke problems. However, these businesses will tend to drive people to an online catalogue and online ordering system, or to an offline sales order entry system based on an online catalogue – these are both Sales Sites. Some of the sites in the sample had a split model – probably because they had not found a way to combine efforts cost effectively, in which the web site acted as a mechanism to request an offline catalogue – essentially functioning as a lead generator, though the business intrinsically needed an online catalogue.

Some sales are very complex, such as is typical of financial services products where a responsible broker may have to collect a wide range of conditional information that would be difficult to collect online. In these cases, the dominant online approach is to develop a lead generation site – this validates the business area for people in receipt of direct mail or other communications, and can be used to generate leads from walk on traffic. These sites have some specific problems – for which there are marketing answers, but the solutions are rarely offered by web site designers.

Subscription sites may appear to be rare businesses – but they are much more common than is usually recognised. Most newspapers and offline news publishing businesses have an online equivalent with a subscription mechanism – Harvard Business Review, The Economist, New Scientist and daily papers, for example. But this model also works for other silos or aggregations of users with specialised interests – dating sites being an interesting example, where the product for the subscription is a listing of categorised members, with member generated content. Don’t you love it when the product sells itself, and spends money to do so?

There’s a few other types of site, such as the “Art Installation”, where the objective is to exhibit largely online material, the “Game Site”, such as World Of WarCraft, or an “info site” such as Wikipaedia. Perhaps the most interesting of these “also rans” was represented by a single example in the survey – a mass product manufacturer; these guys have an intriguing problem – they want to be seen to offer advanced products, but do not sell directly to the public. The example that we saw also failed to offer a directory of retailers, and offered no phone number to find them, either. It would have been a major exercise in frustration for a prospective purchaser!

The rest of these site types were not represented in our sample – a consequence of the size of the sample, and the location. Had we been covering parts of Silicon Valley, the representative sample would have been quite different!

All this of course, is subject to some basic principles – known in Marketing as the Four P’s (or the Seven P’s, depending on what market and product you have). There will be some businesses for which the only sensible web site is a Validation Site – because even with a large volume of sales, the product should only be available through offline channels and the best that a web site could offer would be a location service to find a retailer or distributor.

Traffic, Conversion, Currency, and PR

After talking to a few of these businesses, it was clear that none of them had any idea how people would end up on their web sites. To some extent, this is understandable. Businesses of this size, at that point, did not have internet marketing specialists on staff. There may be only one person or even none, with an explicit marketing brief.

The problems are, of course, linked. Interesting and well designed web sites, useful to customers will generate a certain amount of traffic simply because they exist. Expending effort to drive visitors to the site, without creating compelling value for the visitor, will result in disappointment.

Web designers tended to hand over static sites, with no mechanism for the client company to update the content. Without visitors, the business would see no good reason to pay for the site to be upgraded.

Public Relations was also invisible for these businesses. Universally, they neglected to post press releases online, failed to note any successful projects to which they had contributed, or even flagging large sales. There’s a rule in marketing that one of the ways that customers recognise a safe business to buy from, is knowing that other people are doing so. Hence the importance of stuff like top 10 lists and testimonials – it says that you won’t be regarded as an idiot, because other people will be making or have already made the same choice.

Initial decisions about the choice of the site could create a negative feedback cycle in which further investment in the site would be seen as merely wasteful; and at the other extreme, a virtuous cycle could result in an upward progress and increasing business emphasis on internet activity.

Sites were not built to allow businesses to inexpensively change content – resulting in no evolution of content. Design was essentially based on “Your Brochure Online” – translating a printed brochure to an online presentation. Usability and accessibility were universally ignored or treated as an alternate text-only presentation. Content was self-obsessed rather than focusing on customers reasons to engage. No advertising could be found that lead to any of the sites, except via mostly UK based paid directories. PR was not used. No identifiable engagement in forums or other discussion channels. A handful of sites used any web analytics – identifiable by a JavaScript web beacon; we could not directly infer whether any businesses used web server log file based analytics. No companies in the survey were using a special phone number to indicate a query that had arisen from web contact.

I’ll survey these referenced techniques of web marketing in another article – but the key ideas to carry away are that the nature of your business, products and services will affect the ways in which you communicate; that the web site design itself affects and is influenced by, marketing strategy; that the web site itself is not the only online tactical effort to use.

A crucial question to be answered elsewhen: Five years on – has the state of play changed and in which ways?

Summary

Building successful online businesses involves different skills online from those offline.

Most businesses were building ineffective sites that dissuaded them from additional useful online efforts, and doing nothing else online that might generate traffic or provide evidence to guide improvement of effort.

The overwhelming majority of web sites in the survey sample, appear to have been bought because they flatter the sites’ owners, rather than because they did anything useful for prospective or returning customers.

Businesses will tend to divide into two main camps – those who stumbled on to or were smart enough to plan a growing online business; and those who bought a site that did nothing and regard the web stuff as hot air and a waste of money.

Internet marketing strategy and tactics were not a widespread skill in 2003. What will the 2008 survey show?

Updates

2008-03-11 Edits for clarity. Clearly flagged in the title that this covers the 2003 Survey. Technique usage description in the penultimate section has been extended.

If you are doing academic research into web spamming, or come from an enterprise involved with search engines, or content management, especially out of the user-interactive Web 2.0 space, we’ll gladly let you have the logfile trace and commentary. We don’t think that any law enforcement agency would do anything about this, but we’ll gladly pass the information on to anyone in law enforcement. We don’t think that sharing this information is in breach of any UK statute, such as the Data Protection Act, and we do think that the intent to deceive for purposes of financial gain negates ethical and moral obligations of privacy.

We wrote a Content Management System for Internet Marketing in 2004, for our clients to use – it’s designed to be a White Hat, search engine friendly CMS, with end user accessibility factors designed in. In one of the presentation modes, it is a Wiki, and it is as a Wiki that we saw the the attack. So this isn’t the same as blog spam, or spamming a discussion forum.

Is this particular spam activity massively significant? No. It’s not even the most common form of spam attack that we see, but it shares many features with other attacks. We hope that when you’ve read the analysis, you’ll understand a bit more about why people spam in the places that they do, how they spam, what you can do to defend your site(s) from spam, and to cast a little light on how this has become important and who the other significant actors are.

Outline of the activity

On the 9th August, we see a first usage, using a real email account. The future spammer makes a legitimate and mildly helpful change to the sandbox site. The following day, at roughly the same time of day, new but spammy links are added. On the third day, same sort of time, two active users from the same IP address make some spammy changes. And then the day after that, a flood of spammy links and text are submitted by yet another user.

A look at the anatomy of the changes gives some insight into how these guys think. On the first day, the arrival is from a site that lists CMS’s. The changes on the second day, when the first spammy links are added, start with a direct jump to the site. There is no referer_info – the spammers jump straight to the site. Is this browser bookmarks? Possibly – this spammer is using Opera, and we suspect that they are using the Speed Dial mechanism of Opera. IMO, Opera is a lot easier for fast local bookmarking than MSIE, FF, Safari or Camino (the other browsers I regularly use for various purposes).

The spammer then checks to see that what pages rank best, verifies the best search for that page, and submits some spammy links. How do we infer this? The search string to find the page on the site includes a space prefix for the search. When you copy/paste from Google search results, that initial space character is fairly hard to avoid. It could be another search engine with the same characteristics, but why research well ranked pages on something that isn’t a target? All of this circumstantial evidence means that we suspect that the search was performed on Google, but can offer no direct evidence for that – not having Google’s records, and all…

Spammer searches for ” Installing on PostgresQL 8″ – inferred from logfile entry for http://sandbox.merjis.com/_search?q=+Installing+on+PostgreSQL+8

After the first spammy submission, the spammers come back from the initial IP address, but get a new cookie. Hmm. Cleared the cache? New Browser? The User Agent signature looks the same as the previous sessions, so probably just cleared the cookies… In this session, it looks as though the spammer has performed concurrent activities to see what the site gets up to, and to test interaction with a second user editing content. So this is probably not one user, with two sessions, but two users.

We next see that the spammers are not coming from a consistent IP address. After the initial concurrent activity, we can see that the IP address changes. One of the users, but only one, is switching between IP addresses, independently of the activity. If this is really an asynchronous activity, this may form a recognisable signature. You’d need something other than the usual web analytics to detect this, though, as the switches between IP addresses are pretty quick. You’ll need something that thinks the cookie is more important than the originating IP address.

User sessions from a mobile wireless modem from someone in a car or on a train are a bit like this, but usually better localised to a network address range. Simply reacting to a changing IP address isn’t right.

Oh ho… later the same day at the same time we first saw the developer, we get the Ukraine IP address back with the initial cookie. So this suggests that we have at least two Ukranian users.

Then a little while later the third spammer starts up. Activity starts on the same IP address that was used before, but has a new cookie. Is this the same machine+user, but with a cleared cookie, or a different machine+user? Bit of weak design to keep starting at the same IP address. Oh, and some of the other IP addresses that it uses are also the same, or in the same IP address range, as the previous session, in the same order. That’s lame.

Fun! After pausing activity for a minute, looks like the second spammer is back, same IP address, but needs a fresh cookie – is this a *fourth* user? Ah… probably not – you need to do cookie clearing to allow logging in from different accounts. Does some work, and a minute later, back again from the same IP address as the last activity, but needs a fresh cookie again. Looks like testing of the login process. Each of these new cookie groupings is followed by a new login and registration of a new address. Oh, *not* testing cookies. That must have been done before. This is setting up for usage of multiple identities. So we capture the accounts registered for each of these logins. Ah ha, we see all three cookies, with a chain of deletion and re-issue, being used, over a period. So this looks like three machines, possibly three different users, at work, from behind the same NAT-ed firewall, and at least one of the users is using proxies, perhaps open proxies, to try and disguise the origins.

Once the setup of new accounts is done, the spammer is off on a trip around the site looking for any formatting rules and examples, and ways to trigger emails to users and mailing lists.

Then we get a burst of posting from multiple identities. Typically one posting from each identity, unsynchronised with the IP switching. So you could get two users on the same transient IP? Nope, looks like the switching is too fast for that. Looks like you might get a request to edit content from one IP address, and the spammed content is submitted from another IP address.

Looks like one person/browser does initial research and page identification, and one does user account authentication and the third adds spammy links? Yes, from the logs, the first user does little on the site other than identifying it and doing some test postings. The second user seems mostly to set up accounts, with some test postings. It’s the third cookied user/browser that does the spam postings, I think. Three machines or three users? Hard to tell.

IP Addresses

For the first few days, it is a consistent IP address that appears to be in Ukraine. Later we see new cookies issued to users who start on the Ukrainian IP address, but rapidly switch off to IP addresses around the world.

Even more interesting is that the software is switching the IP address during an activity. That is, it can start by navigating to a page in the CMS from one IP address, and edit a piece of spam from a second address. This implies that whatever switches the proxy is not coordinated with user activity.

Think… Tor? Yes! These IP addresses look to be Tor exit sites. Kerching. These guys are using Tor to try and disguise their origins, but aren’t particularly clever about it – or they’d have started using Tor, and probably visited a few innocuous sites first, to help disguise their origin. They are also pretty dumb about clearing cookies, or we wouldn’t have detected this. We did not do a DNS log cross check, but I expect that we’d have seen some DNS lookups associated with these accesses, probably directly from the spammers – they don’t appear to be using the latest best practice efforts in anonymity, only the most obvious tools, poorly understood.

User Identity

These spammers use multiple user identities. They log in as users from:

• Yahoo Mail
• Google Mail
• Microsoft’s Hotmail

They also register a few made up mail services, too, but of course can’t authenticate those, so don’t use them in the actual spam deposition. This usage of fake email addresses appears to be a check that the accounts are actually used for verification. I suppose we shouldn’t be surprised that the main email addresses used for generating web spam are all from free email services sponsored by companies with search engines. It is rather ironic, though. If the search engines want to control web spam, one thing they could do would be to better control opening new webmail accounts. Yeah, right.

One way to cause costs to spammers is to require additional verification steps. They’ll be more likely to use a service that doesn’t require verification by email. However, the more difficult you make interaction with a site, the less the interaction with users. This suggests, at least to me, that grades of identity are probably useful on the internet. A single user might want to maintain multiple grades of identity. The highest grade allows authenticated financial transfers – much like a bank. The lowest level grade is fully anonymised, protects identity above all, but has no trust – you wouldn’t let an authentication at that grade do anything on the site, except, perhaps, to browse.

Why so draconian on the authentication issue? Because trust is on both sides of the fence and trust is what allows positive sum games to develop. The more paranoid users who don’t want to allow a business any insight into how users use a site, deny the business information that potentially allows the site to be improved. Businesses don’t (IME) collect data to be used personally. They collect data so that they can see which pages work for users and which pages cause frustration. You don’t usually get that data from feedback forms, you get it from users abandoning the site on specific pages. It doesn’t matter who the user is, just that they have abandoned their session and don’t return. So cookies and authentication can help users to a better web experience – even though they can also be used for targeting adverts (umm, is that such a bad thing? Wouldn’t adverts be less annoying if they actually related better to your interests? How about using an AdBlocker rather than preventing tracking within a site?)

Here’s another intriguing number… How many different accounts do these spammers use? One identifiably real email address. And 13 other spam-validation addresses from search engine webmail services. So expect that any claims from the SE’s for user accounts are inflated by a large factor – it seems likely that most of them are bogus users… as if you hadn’t guessed that from the email spam you receive every day ;)

Automation and Botnets

When we first saw this, we suspected that it was a developer programming a bot to spam. We now think that the costs of this development and the relative inflexibility of software response, means that much spam is actually generated by humans. There certainly are bots active, but the analysed activity, and much of the activity on this blog, is human driven via real browsers, or astonishingly good mimics of those browsers.

Put it like this… When Indian SEO’s offer unique text links to your site, from topic relevant forums, blogs and social networking sites for $7.50/link, and article writers offer unique articles for $10.00/article, then the cost of developing content and links is so low that developing smart enough software to identify relevance, is too much. Even a smart AI programmer in a low cost economy can probably make a higher daily rate, in the short term, by personally spamming sites, than by writing software to do so.

This is because one of the costs of webspam, to a spammer, is making rejected postings. For example, this blog rejects about 99.9% of all postings, because most postings are spam. Most of the postings here also appear to be made by a series of bots. This is why they are rejected – they are insufficiently unique. Those that are unique are usually rejected by me or my peers here, because the content is dull, uninformative and irrelevant (“Your site design is good. Visit my site too.”, followed by a list of pills, porn, gambling, shoe and car links).

So a high quality spam effort won’t be automated, yet. It’ll be personal. Until the AI is improved. Work on beefing up AI and self-improving system continues, but until we make a jump, the cheapest way to infer meaning is to use low cost human effort. And humans leave characteristic signatures so far not emulated by bots – such as setting up Tor connections badly, and leaving cookie trails.

What Can CMS’s do to protect themselves?

Open proxies are a problem for many Information Security reasons. Given how much has been written about the evils and dangers of open proxies, you’d think system admins would have stopped using them. However, there’s still a bazillion of them, and plenty of people running probes to find them.

So, do yourself and the rest of the world a favour. Find out if your organisation is hosting an open proxy and see if there’s a way to make it more secure. Even having it authenticate against validated users would be better than leaving it completely open…

You might want to consider restrictions on users coming from free webmail services and using anonymising services or open proxies. What you can usefully do and what you can legitimately do, will depend on the environment, but this analysis suggests that teams work on spamming, or at least multiple browsers are used – meaning that the spam attack may come from a related source, but not always one on which you’ve left a cookie or could have left any trace or tracking id. Simple cookie tracking, or even Flash Cookie tracking aren’t going to be enough. The spammer may well come from a different address than the person that first identified the site. If you can, you want to head them off, early… but beware that making signup into an onerous burden may put off the customer. If you make the validation sufficiently difficult that you’d trust a financial transaction, *before* you allow access, then you won’t see much access.

One possible way through this may to be establish trust chains. That is, you only get to add content and links if you have been endorsed by someone else, who has been endorsed. Then, if you get spam injections, all the endorsements in that tree may become suspect. Trust chains are pretty rare on most sites – though places like Facebook, LinkedIn, Orkut are based on the friend of a friend type model, with introductions and described relationships.

Perhaps the owners of such sites could offer the internet a valuable service, extending their APIs to allow authentication. If you see an authenticated user, then you trust the posting more, even if looks spammy. If it actually is spammy, then either individually, or collaboratively, you could reduce trust. Of course, this is also hard to do – you can have malicious trust attacks… Take a look at Charles Stross’ stories of Macx, in Accelerando for an entertaining science fictional account.

I predict between 300:1 and 1,000:1 ratio of spam to real comments for this article, over the next few weeks. That’s how bad it is. After the first two months, the ratio will worsen, because older articles are mostly commented on by spammers.

What Can Search Engines Do?

Drat. You would ask that. This is really, really hard to manage, I think.

Since the SE’s are merely using content and have no access to the CMS web server log files, they are even less likely to spot dodgy content.

The main thing they could do now, would be to monitor changing web pages. In our case, we reverted these edits. The CMS has a full version control system, and we simply rolled back the edits. And then blocked the various real and fake accounts…

For a search engine, the signature should be that the pages were modified, saw new links, and then a little while later the pages were reverted and the links removed. If there’s been a spate of other activity for those links, then it may be worthwhile to slow the rate at which PR is given to these sites… Though that would make it likely for the malicious to end up paying spammers to negatively promote competitor sites… Very, very tricky.

What this means, I think, is that pure citation based models face an end-game… But I’ve already blogged endlessly about that. It means that organic indexes will become increasingly dominated by sites put there by increasingly complex software, and humans from low cost economies, rather than a consequence of human judgement. At least, until the SE’s work out how to compensate for this rather insidious mechanism.

The other thing is to use FaceBook and LinkedIn to offer authority chains. And lo and behold, we hear this week, that FaceBook and Google are in talks… admittedly the publicly stated focus is to allow FaceBook users to be listed on Google. But once that relationship is in place, there’s the opportunity for looking at TrustRank relationships, and business relationships. So once you say that you are related to other users and to a website, you establish a degree of trust – in a way that PageRank type algorithms can probably use.

As you might guess, it takes me weeks to write on of these postings, between other activities. So the Google and FaceBook stuff still isn’t fully resolved in my head. I’ll probably return to this relationship after I’ve thought about it a bit more. I actually started this piece almost a month ago, just after the spam attack and our defensive reaction, long before the G/FB relationship was public.

Summary

Spammers identify higher page rank sites, and then look for the better ranking pages. They identify how they can use the sites, then start adding spammy content, using a range of appropriate forms and multiple identities.

We inspected our web server log files around the time frame of this attack series, manually looking for re-use of identified and previously used Tor exit nodes and the signature of an IP address changing frequently with the same cookie, going back a day before, and up to the point at which access was effectively prevented for this spammer. The primary use of Tor, on this specific site, in this time range, was the described spam activity. We infer that you should expect spammers to try and cover their trails, and expect them to use free webmail email addresses.

Note that the first edit was a meaningful change. This means that a system admin looking for new users to behave badly, would see some helpful changes, and may assume that these users were legitimate. Only the later changes are spam – and the volume of spam is far higher, just a few days later, than the initial changes.

Webspam probably can’t be fully tackled without default installation of webspam tools, like Akismet, on all blogs and discussion forums, and the next step will be to add user authentication, so you know from other sites, whether this identity can be trusted to post sane and relevant information.

Things To Do

If you can demonstrate that you come from an organisation that develops CMS’s or web server log file analysis, or are an academic security researcher, we’ll share the web server log file data of the attacks.

Make sure that you and your organisation aren’t offering open proxies.

Make sure that you use content spam rejection tools, to ensure that your organisation is not hosting spammy links and articles that reflect poorly on your brand value. This will help your own site reputation, trust, whatever.

Join FaceBook, LinkedIn and other professional accreditation sites. If you can link your identity to a professional accreditation and a series of credible users, then there’s a reasonable chance that next generation search will find even your most inane postings to be profound and useful – more so than some anonymous dweeb who posts via anonymising services to offer badly spelled and gramatically incorrect links to replica watches. This is a pre-emptive recommendation – but, as I’ve said in earlier postings, I think we’re at a time on the Internet where economic needs of users and businesses are forcing a review of what constitutes a good link and a good search result.

Don’t accept links on social networks from people you don’t know and trust. This will be the fastest way to have your own reputation damaged, and the future value of your postings and links devalued. At present this may mean offending new a casual acquaintances – I know I feel guilty when I deny links to New Zealand based holiday promoters and South African photographers that I’ve never met… but FOAF spam will become an important tool for people intending to deceive.

Material Disclosure

We do mine anonymised user behaviour from web log file analysis to improve websites. So any opinions above about how harmless this activity is, to users, may be treated with suspicion or with endorsement, depending on the settings of your current memeplex. It comes down to Trust. Do you trust the bulk of postings that I’ve put on here, to reflect a truthful insight into the ways in which a marketing organisation *can* use data? Or am I really a lying and scheming bastard who is clearly depriving you of a right to spam us and others – a right that I regard as being the net equivalent of the right to causelessly shout “Fire” in a theatre? Trust and responsibility is a two way street – users and businesses need to be able to trust each other, in order to make the best use of the internet.

Updates

2007-09-09 Shava Nerad of The Tor Project pointed out that a sentence in the original summary could be read as implying that we were making the claim that the primary use of Tor across the entire Internet was to add blog spam. The summary was amended to make it more clear that this is a time limited analysis of a connected series of spam injections to one web server. We’re grateful to the Tor Project for the opportunity to improve the article.

2007-09-10 minor edits to correct spelling and improve clarity of some sentences, none intended to change the sense, except for one. Under the IP addresses heading was a sentence left over from the period when we thought the attack was part of a software development exercise, and it referred to the code doing something, when we now believe that the attack was purely human in origin.

2008-07-24 FaceBook now launches an online identity platform – single sign on, competing with Open Identity, Google’s own authentication services and the early-to-mass-market Microsoft Passport.

The auction still seems to be built on the generalised second price model. But there’s already a fudge factor for Minimum Cost Per Click (MinCPC) for the right hand side auction, based on undisclosed criteria, in addition to some documented or inferred criteria.

Google appear to be saying that the top position now only permits advertisers where the CTR is above some qualifying level, and above some minimum price for the bid.

Erm. Just how is this different from *anything* that Google does, now?

Experiments With The Auction

An example?

Create an AdGroup for some random text string. If you all do this together and start searching for the same string, you’ll create some interesting but unintended effects. So please choose one of the following, only if your imagination fails you:

• snazzled fribbles
• omnicognisant limpoids
• hyperlobic snozzles

Or, even better, combine those words plus something distinctive of your own (such as a colour) and make your own new AdGroup. Use the keyword you’ve picked in the advert. Use exact match only. Bid $0.01. Look at the AdGroup. Chances are that you’ve got a Minimum CPC close to $0.05, and your advert is inactive as your bid isn’t high enough. Now increase the bid to the MinCPC, and try some searches on Google.com (or your local Google server). You probably don’t appear. Increase the bid and you should find that at around $0.22 you’ll see your advert appear. Drop the bid, without clicking on the advert, and you disappear from searches again.

This shows that there is already an unpublished criterion for whether an advert appears, even on the right hand side, even if you meet the MinCPC. The default CTR for an advert appears to be around 2.5%. So Google is implicitly saying that you need to offer (not *pay*, but *offer*) a CPM of around $5.50.

Now, this number is a key number for Google. We can find it again, quite easily.

Pause the newly created AdGroup, and create another AdGroup for your selected keyword, but this time, do not use your keyword in the advert. Bid $0.01 again. Now look at the offered MinCPC in order to activate your advert. If your advert shares no similarity to the keyword, you should see a MinCPC of around $5.50.

If you do click on your keyword-matching advert, you should see a fairly rapidly decreasing CPC. The first click will probably pay Google around $0.11, and the second around $0.05 and after three or four clicks, you’ll probably be at around $0.04. The CTR is not relevant – this can be done with ten impressions or a hundred impressions and I’ve got roughly the same results, in the tests that I’ve done.

Result!

A few experiments yield some important parameters for Google.

You need to initially offer a CPM of $5.50. This is coincidentally the matching MinCPC required when your advert fails to match the keyword – and for which we can then infer that Google assumes a CTR of 0.1%.

Minimum *offered* CPM to appear – $5.50

Assumed CTR for an advert and matching keyword with no history – 2.5%

Assumed CTR for an advert and non-matching keyword with no history – 0.1%

Way cool, eh? But what has this got to do with the Top Position box?

Top Position

Until now, Google has applied a few known criteria for appearing in the top position.

Your adverts must have passed editorial review.

Well, this makes sense. These adverts are going to be given a premium position, and will be the most likely adverts to run on Google’s partner sites. So they *must* be reviewed and approved, or they can’t be published widely.

They must be high CTR

I’ve never got a clients advert into the top positions with a CTR below 7.5%. The actual value seems to depend on the keyword. Different keywords appear to have different threshholds. I’ve had some adverts with a 35% CTR (if run in position 1) that don’t appear, when the bid is too low.

Hang on… say that again

Yup, right now, if you have an advert that, when run in position 1, would have a significant CTR, it gets dropped from the top position box when you drop the bid.

In what way is this different from what Google is now saying?

They now say that you need to exceed a minimum CPC in your offered price to appear – just as now.

They say that they will only choose from higher CTR adverts – that appears to be functionally true now.

The difference is… what? They’ve always been able to pick who they want. Advertisers don’t have information about how they are picked.

The difference is, I suspect, that they’ve set a new CPM target for the top position box. This changes the required MinCPC before your advert appears there, and your offered price affects the required CPM target for your advert to be allowed to participate.

That’s all that this announcement boils down to.

For the implications – my previous article is still close enough, I think. This move will raise revenues for Google, improve partner revenues and allows Google to compete with MSN, by offering publishers more money from the top placed adverts. That will position Google for further growth by stealing more partners from other advertising channels.

I suspect that Google hasn’t been focused on user experience, or what this change does for advertisers, agencies, affiliates, etc. All that stuff is secondary, and the story about improving user experience is just smoke and mirrors. Maintaining the share price and analyst confidence. That’s what this is about.

Updates

2007-08-16 – Title changed “Higher CTR, Higher MinCPC, Does What?” didn’t point to the article being about Top Position.

As you might expect, a first pass analysis suggests that Google is using this to increase revenue streams. However, the consequential effects may be more subtle and may be the real motivation.

The key to the announcement is probably this key phrase about the top position minimum price from the FAQ:

Your actual CPC will continue to be determined by the auction, but subject to a minimum price for top spots.

In other words, this is yet another fudge factor for the still rather mysterious Quality Score. It lets Google put a thumb, hidden, on the scales for the auction, and adjust how much revenue is extracted.

Who will it affect?

This action should have a significant effect on a wide range of AdWords users, including advertisers, agencies, affiliates and publishers. Curiously, I can’t think of either positive or negative effects for users, but that’s partially because the chain of consequences is so complex. Working out whether this makes for a better or a worse user experience is particularly opaque. And I think that may be the biggest weakness.

Users

Google has mostly grown huge because, in 1997 and until relatively recently, it clearly offered a significantly better search experience; in the last few years the major competitors have offered broadly similar results. Google, in talks to advertisers, and in the practical advice of Googe Account Strategists, emphasises the user experience. Most things that Google does (like the motivation for the Quality Score) can be traced back to, at least in part, user experience. I can see how most of the things that are done will benefit users, even if advertisers are left confused and out of pocket.

This change? I simply can’t see any clear user experience gain. And that’s is really worrying. If the accountants have seized control, then expect further changes that do not help users, or even worse, inconvenience users, just because it yields a Google revenue optimisation.

Maybe I’m confused by the negative effects on many of my clients and psychologically I don’t want to see any benefits from this change… I just don’t see how this hidden factor results in adverts that better match user expectation. If anything, I think this increases the opportunities for high value advertisers who service smaller audience segments. That should result in a lower volume of clicks going to the highest positions. Revenue is sustained by manipulating the MinCPC, but that doesn’t appear to be a user benefit.

Advertisers

Advertisers who currently are in the magical top positions, and advertisers who want to be there, will be affected. Do these form any specific groups? From looking at our client base, we expect that the main effect will be on major consumer brands. If you have a major brand, then you may achieve the number one top position and pay only $0.01 per click, today. With Google’s thumb on the scale, they can increase the minimum cost per click, thereby increasing revenue.

This will in turn damage the overall account Return On Advertising Spend (ROAS). Many large consumer brands make their highest ROAS from their key brand terms. The consequence will be that keywords below ROAS targets will drag the account well below the target ROAS, and so these marginal keywords will no longer be bid by the major brands. That creates more opportunities for affiliates to find effective keywords, far from the main brand keywords. It should make broad match less attractive for major brands – because broad match tends to attract more marginal search queries.

One other way to look at this is that Google wants to take more share of major brand equity investment. It takes a lot of money to establish a brand. Getting users to search for a brand rather than a generic term costs that brand. When Google can raise their revenue by taxing usage of brand searches, then it imposes an additional brand cost, yields no additional revenue to the brand owner, but makes a pile of cash for Google. I’d be pretty damned cross with Google, if I owned a major consumer brand.

For non-brand queries, such as “mortgages”, “holidays”, etc? The effects for advertisers are likely to be complex. Right now, I expect that Google revenue will increase. This may dislodge some advertisers who, while getting high enough CTR were not getting enough sales to really justify the spend. These advertisers will disappear not just from the top positions, but from the search altogether. That will open more opportunities for advertisers who better match the search query with a web site that converts better. That is potentially better for users… if this chain of effects is what happens.

Agencies

Will need to rework expected revenue changes and adjust their mix to reflect the poorer ROAS from high volume, low AvCPC keywords. Since the precise effects are not yet known, this will raise client anxiety about meeting targets, and probably cause quite a bit of threshing.

Publishers

Publishers are in an interesting position. Many larger publishers of keyword search adverts take only the highest position (and hence highest revenue) adverts. This move will put more cash into the pockets of publishing partners. This could be seen as competitive positioning against Microsoft and Yahoo. By paying out more money to publishing partners, Google can buy their attention and fend off competitors who are willing to offer high revenue shares in order to woo the high volume publishers.

Part of this is probably a reflection of the way that the generalised second auction bidding system can be operated. There appear to be two main ways that this auction can be run:

• Once per day
• For every impression

If the auction is run once per day, then all advertisers take part, and their AvCPC will be a reflection of what they would pay, if every advertiser participated in every auction. Since many advertisers have budget limits or day parting constraints, this auction systemn results in higher paid prices.

If the auction is run for each impression, only for the advertisers that have funds for that impression, then the number of advertisers is smaller, and on the whole, the auction will result in a lower price.

Google appears to use an auction for every impression (with some data that is calculated daily). MSN appears to use an auction that is daily. I can’t decide what Yahoo’s auction is doing. It may be something else altogether.

By increasing the AvCPC for top positions, Google can present more revenue to high volume publishers. This may be Google’s real goal. Continuing to expand publishing networks and rewarding publishers by paying them more advertiser funds.

Affiliates

I can see advantages and problems for affiliates. I haven’t yet worked out whether the net effect is positive or negative. Much as with users, I can see some good and some bad stuff.

What can and should major brands do in response?

After a few hours of thinking about this, I suspect that the major response should be to review content match programs, and that this may be the secondary motivation for Google.

If your ROAS is being damaged by higher costs on your main terms, you should be looking for other low cost opportunities to make sales. Google has been pitching content match to major accounts over the last few months and incentivising agencies to make more from content match programs. Historically, large agencies have usually turned off content match for their clients, unless a major branding exercise has been under way, and the performance metric is eyeballs rather than sales.

By expanding content match advertising, major brands can decrease the average cost per click. Since Google appears to have been spending effort in cleaning up the content network, they probably feel that it merits renewed interest from advertisers.

Summary

I may entirely rewrite this article (well, I’ll probably write a new article, back refer to this, and put a prominent “Superceded” warning near the top).

I’m not at all convinced that this change helps users – and focusing on the user has always been Google’s main winner. Sacrificing, or at least having no clear benefit for, the user experience in order to generate revenue gains is counter to the culture that has brought Google to prominence.

I’m not feeling positive about this change. It looks like another way for Google to manipulate search behind the scenes, without any clear advertiser guidance, or attention to the benefits that advertisers can and should reap.

The only clear beneficiaries appear to be Google and larger publishers.

How does this relate to the destruction of the web as we know it? Why is this any kind of a big deal? Today?

Today (well, strictly, yesterday) is the first time that spammed links submitted as comments to this blog have exceeded the daily spam detected in my email. That ratio is related to Google’s success and the way in which marketeers use their marketing mix, and why Google’s current ranking system is destroying the informational value of the web. I prepared this article back in May, as I saw the ratio gradually cranking up… It’s a sample of one blog and one email account, so the exact ratio is spurious as a worldwide marker, but is probably indicative of the way that the internet is evolving.

Huh? So what?

Your major reason to care is that this drives up the costs of internet marketing, making internet products more expensive for end users, but without increasing their benefit (either perceived or actual). That, in turn, slows the pace of online marketing expansion.

One way to understand the news that my blog spam rate exceeds my email spam rate, is to assume that this blog is becoming more widely cited. Well, that’s partially true, I suppose. I’ve a long way to go before this blog begins to attract a significant fraction of the traffic to Andy Beal, Matt Cutts or Danny Sullivan‘s blogs. If I get this much spam, their blog spam lists must be truely awesome.

Of course, not all blog spam is oriented towards building page rank. Some of it is conscious advertising, hoping that users will follow links in the comments. However, the preponderance of blog spam that I see has author names, links with anchor text that look just like you’d expect from a SEO. SEO’s offer blog spamming as a service. And there’s advertised tools that spam blogs. They even offer to inject, just like spam email, high entropy text to evade spam filters.

Another way to read this spam ratio data point, is that businesses are finding blog spam to be a useful way to drive traffic. A high organic ranking can drive traffic for months, whereas one email campaign generates a limited duration boost to new visitors. If they were the same cost, which would you use? Which would your boss force you to choose?

The costs of generating email spam and blog spam are fairly similar. Both can use intensely automated systems for submission. The costs of submission are miniscule, for each spam item, but you need a lot of spam to make any money… so these costs ultimately do mount up, to a level that you notice.

Businesses do pay for meaningless activity. There are marketing managers who buy spammy SEO services without understanding what they’ve done. I have no doubt that a large portion of these paid activity links have absolutely no effect, or even in some cases a negative effect. Pretty much any blog spam submitted to this blog, for example, is wasted – except, I suppose, that I look at one or two of the links, usually via an anonymising service, to understand the technical and economic motivation for the spammers.

Some of the blog and other link spamming activity is clearly working, or believed to work. One consequence is that anything that looks like a form that might lead to web postings on blogs or discussion forums, is being stuffed with keywords and links and submitted. We’ve some clients, whose sites we manage, that have seen blog spam appearing in webmail inquiry forms, for the first time since we started managing their sites years ago, in the last few months. Forms previously only touched by prospects, are stuffed with MMORPG gold links, pharmaceuticals, cars and stuff in non-Roman characters sets, and then submitted to businesses unrelated to any of these activities, and who don’t publish the forms in any way.

This all suggests that the techniques are working, even with some failure rate through posting to blogs that don’t show spam, forums that moderate and forms that send data to internal mail users. Even though some people clearly are paying for essentially superstitious activities, there is too much link spamming for all of it to be ineffective.

Truth and Consequences

Long term, the effect is that link-based systems to determine rank are going to be as dead as the old in-page content ranking systems and the human moderated directory. These still exist as technologies, and they are still generating useful revenues for their businesses, but they aren’t where the majority of users are going. They are effectively “cash cows” – technological dead ends, being milked of the revenue they generate, but no longer in the growth phases.

Realistically, it took ten years for internet marketers, content management system designers and businesses to switch from stuffing meta tag keywords on pages and using human moderated directories, in order to accommodate an information rich, search engine friendly, link stuffed style. It’ll take at least ten years for this reshaping of “natural links” to decay back into a informational web, and that’s assuming that today we had, and were all using, an effective third generation search engine that didn’t use the informational web to reflect economic utility.

That search engine may be out there… and we just don’t know about it. Or it could be that Google has seen the threat to their business, and is actively working on it. They have enough smart people that this is feasible, and they’ve taken innovative steps in the past. Even though Google’s business is currently destroying link value, there’s a chance that they’ll also save the web.

Third Gen Search

The essence of Page Rank, as I’ve said in previous articles, is to find informational pages. This was useful in 1997. It was useful in 2000. But as the world’s economy shifts online, the search results for some subset (by volume) of searches, is primarily directly economic. That is, people do searches to find out stuff they need to know, but some searchers are looking for somewhere to buy from.

Informational page value and economic value to the searcher are distinctly different. The result is that second generation search systems using link citation will only find economically relevant sites, with low information content, if many of their marketing managers use techniques that destroy Google’s investment in the citation based system.

Simple Cases

Let’s take a simple example. I realise that detailed discussion about this example can blow the case out of the water, because real life is always more complex. However, this is representative of a class of sites that have problems with Google now, and who are most likely to be using link spam, in order to overcome informational bias in search results.

Let us suppose that you want to buy a DVD. You know the film you want (John Carpenter’s early SF film, “Black Star”). Many of you would be perfectly happy to buy from a site that simply told you it had the DVD in stock, that it shipped to the region you were in, and gave you some content that indicated it was a reputable trading business that wouldn’t just take your money without shipping anything. For the most part, you don’t need a review, don’t need customer recommendations (you know this film, you just want it on DVD). This minimalist site would be perfect for shoppers who searched for “buy DVD john carpenter black star”. Would it rank? Probably not. It’d be buried below a bunch of sites that had all the needed content, plus additional stuff. Additional stuff put there because information and economic utility have been confused. Additional stuff that costs money to do and drives up the cost of business (reducing margins, or increasing prices to end users).

The marketers response to low ranking? Hire a SEO to build a network of (aged) domains, and stuff them with links; make postings to marginally relevant forums; spam marginally relevant blogs; stuff forms on websites in the hopes that some of them result in page content; add reviews or other content to the web page to make the page more attractive to ranking systems, etc. None of this benefits users. None of it is free (it takes time, or rented software) to do.

Comparison Shopping

So, what about a comparison shopping site? Many shoppers like to use these, because they don’t have to open a zillion windows, with shipping information and availability and pricing presenting in a multiplicity of ways (some of which may require opening a single store in multiple windows). You may use a comparison shopping site when you know the products you are interested in, but don’t know what the best internet price is, or who has the item in stock now.

A minimalist comparison shopping site would list vendors, prices, geographical shipping criteria, availability, all on a single page. No reviews. Perhaps a star or popularity rating for vendors. Perhaps a “people who looked for this also considered” list of other products.

Again, this site is unlikely to rank. Indeed, with the informational citation model, this page would be very low ranked – the outbound links are often JavaScript (for tracking purposes) or obfuscated via transitory database reference, so the SE’s often won’t see links to highly relevant pages outbound. And, of course, comparison shopping sites are plausibly understood as a “secondary search site”. That is, this is a site that having once reached it, you have a further search to do (very common on the home pages of these sites, and even in department and category pages – search is a pretty well understood user interface model, for users).

Just as with the pure sales site, the pure comparison shopping site doesn’t need a huge informational burden. If it has any information, it is probably about the vendors, not the products. That’s what a user in the final phases of the buying process is looking for. “Who will sell me this item?” Not “Do I want to buy something like this?”.

In the informational model of search, for these sites to rank, they need to be different sites (with extra content, added at a cost, in order to satisfy the search engines) or to pay for links, or to survive only on paid search… Even though they arguably assist users to find what they want – the best price for the type of item they want to buy.

Economic Value of Link Spam

There’s an commercial imperative to link spam. That’s why my Akismet tab on this blog is stuffed with several hundred spammed comments each and every day. That has a cost to me, and to the rest of the internet – every blogger is facing a rising tide of link spam. Every business is beginning to see any web form, and support inquiry form, stuffed by spammed links… because current search models have confused information utility and economic utility.

So, we now get to the point… Google’s success in informational search is now creating the conditions in which quantities of poor quality information are being generated purely to offer rank (low value postings, link spam, SE bait). Links are established using a myriad different techniques, in order to promote an economically valuable page to the top of an information delivery service.

It means that Google is having more and more difficulty with finding the right content. It also means that potentially better search technologies are looking at an environment where the page content and links can’t be trusted. Page content was broken by page content spam in the late 90′s. Now links can’t be trusted without a huge burden of processing to dismiss possibly purchased votes. Pages have information on them that exists purely to attract search engines, and does nothing for the user experience. And you and I now have to spend economically valuable time dealing with blog spam, instead of earning money.

Better Mouse Trap?

Google has obviously been exploring how to move from the citation link model. I’ve seen some articles mentioning Google patents for other ranking techniques. As link spam comes to form a significant fraction of all links, Google and other SE’s will be forced to use new models to generate commercially and informationally useful rankings. These patents are clues as to the direction that Google is going.

Search marketers should be thinking now about what shape their marketing efforts will need to take, in order to position themselves for the next few years. Just as non-spammy link building exercises and user-valued content development have taken time, these new techniques will require resources, funding, strategy and tactical development… the early bird might not get the worm, but they’ll eventually be eating other people’s dinner.

Predicting these things is always nightmarish… but if you don’t start to work on the strategy, some accidental bystander gets rich. It’s actually part of Eric Beinhocker’s “The Origin of Wealth”, a book that is beginning to rank in my mind alongside Evans and Wurster’s “Blown to Bits”.

What will leading edge marketeers be doing next year, to get the jump on the changes? Interesting question… for another time. Oh, in reference to this article? They’ll still be linkbaiting. Gee, if you are here, these techniques are probably working.

Summary

Google’s use of informational content to determine page value to a user is a consequence of a specific belief about how users use search results. As users have become more sophisticated, content has evolved, and usage is altering to put increasingly large volumes of economic activity on the web. Although not directly cited here, much of the initial contact from web user to vendor is mediated by search. A search engine results system based purely on informational value and links does not necessarily satisfy searches for purchasing activity. The consequence is that organisations engage in techniques to artificially improve content and links in ways that are not always directly or even indirectly beneficial to users, or natural to the economic activity.

Recasting the economic value of a page as informational value, results in an increased burden of economic activity that is not directly related to the purchase, or even the same market, by larger numbers of users, to a population outside the search engines – site administration and web marketeers. Even if a solution were available now, in the form of a search engine that presented alternative results based on both informational and transactional interpretation of the query and of ranking, it might take a decade for the effects to be felt in Content Management Systems and other web site delivery knowledge.

Strictly, it isn’t Google destroying the web, but a myriad other players, who interpret the activities of Google and react individually to maximise their performance. This ought to be a classic study in microeconomics and of rational, partly rational and variously informed participants. This collection of users with conflicting goals and varying degrees of imperfection is why response doesn’t form an immediate equilibrium, nor do all participants cause direct economic effects – high value results, lack of information, mixed levels of competence and low penalty costs for damaging behaviour causes bystander effects, and will do so for a long time.

Material Disclosures

Some books are referenced above. These links are not paid for in cash, kind or reciprocal links, nor do we take an affiliate revenue stream for the recommendation. We do not have a relationship with the web sites or products mentioned above, and we derive no income or consciously sought page rank from referencing them.

We do international/multilingual paid search management and performance improvement, some internet strategy consulting and web site conversion improvement, with an emphasis on White Hat SEO for pages we build.

Updates

2007-08-02

Add link to Lambda The Ultimate in comment below, citing their article about spam and user comments about migrating to trust-rank and authentication based systems to help control it. This is the most advanced programming forum I know of. If these guys are headed there, then we probably all are.

Some text tidyup for clarity. Notably in the section about a minimalist site, I added some explicit detail about likely marketing responses to low rank.