Let’s have a look at what Google Product Search is, and does, first… so we’ve got some context.

Google Product Search

It’s free. Anyone can sign up, and feed products. Well, almost anyone. Google has a restriction that there must be one supplier for a given product URL. So you can’t have lots of people all listing the same DVD on Amazon – Google doesn’t like spam, and multiple appearances of the same product from the same vendor at the same price, count as spam.

Within limits, then, Google Product Search is free and a wide range of businesses can use it. You just need to sign up for a Google Merchant Center account, and publish a feed. A feed can be as simple as a properly formatted CSV file (exportable from a spreadsheet, once per month) or as complex as a machine-readable and real-time updated XML file synchronised with your product inventory and pricing.

Product Search results – well, they don’t appear in every search, and they don’t alway appear as number one in the search listings. That last item is especially important and tells us, the rest of the world, something about how Google is treating Google Product Search.

Why is it important that Google Product Search is not always number one for product related searches? Anyone involved with search engines knows that number of clicks to a listing is *very strongly* correlated with the position on the page. A listing in position 1 can easily get ten times, or more, as many clicks as exactly the same listing at the bottom of the page. So for Google to list Google Product Search at positions *other than* position 1, tells us that Google is giving up a lot of influence. And remember, Google isn’t making money (none directly, and only indirectly some revenue) from Product Search listings.

The implication of the varying position of Product Search results is that Google is not giving undue weight to their own product. The reverse implication – that Google is artificially reducing the ranking of competing product listing services – is not addressed by that observation. Just seeing that Google doesn’t inflate it’s own rank in all cases, is not enough to say that a different service has not been penalised.

Are there any other vertical search engines in Google results?

If Foundem has a case, it is that they are being excluded, artificially from results. You don’t just get listed highly in search results because you exist. Well, not unless the search is pretty rare. If you’re talking about the highly competitive spaces in which searches relate to things that people might buy, the competition is intense. To show there, you need a lot of “link love”. You need a lot of people to link to your site, with the right words, and you need the site to be well designed, and you need to have *credible* links. If you don’t have all of that, you don’t stand much chance.

So, let’s assume that Foundem, being fairly new, and not having made much of a media impact before today, doesn’t have a lot of link love. Can we support their argument by showing that Google doesn’t highly rank vertical search engines?

Annoyingly for Foundem, there are vertical search products that show. Businesses like Trovit, with their used car facet-based searches, letting you slip a slider along the price range that interests you, does show up. And they show up for place reviews. And other stuff… It is arguably a better user experience than Google Product Search for used cars. Trovit have page one appearances for some products. I can see their impact on clients’ sites.

Google *does* allow vertical search specialists to appear in search results. That, for me, puts a dent in the argument that Google excludes vertical search specialists. If some do appear, and they collect clicks, in multiple categories of product, then Google is not completely removing them from listings. A dent, but not conclusive evidence.

Do Ciao and Foundem have a case?

I don’t think an appeal based on “we must be shown because we’re a competitor” has any real merit. I’m not a lawyer – so that’s not a commentary on whether the courts would find it to be true. But in terms of search engine results, if you aren’t sufficiently loved by users, to displace the other ten businesses that are shown, that’s not, IMO, a reason for a court to decide that you should be shown. Win the users’ love… and you should be listed.

But, since the algorithms are trade secrets, and the data collected is known only to Google, for an outsider to claim that they have been penalised by Google, is pretty difficult to discern, or prove.

Unless, of course, you have the backing of another search engine that produces broadly similar results, in which your site appears more highly ranked and in which you have no evidence to believe that the results are influenced. In other words, if you were financed by Bing (the only other serious volume search engine left in the game), then you might have strong evidence that you don’t have spammy links, spammy comments and other factors that would cause de-weighting.

Ciao is owned by Bing. Might there be some evidence? Or is this just Bing (Microsoft) playing games with Google. FUD (Fear, Uncertainty and Doubt) have always been strong components of Microsoft marketing techniques. I certainly don’t have any confidence that Microsoft is doing this because they love Egalité, Fraternité, Liberté and the European Way Of Life.

Foundem – it’s a small, relatively new player, in a crowded market. I suspect their only problem is the classic one – not enough users love them to help their site to rank highly. But I haven’t investigated their site or backlinks. That’s just a first pass impression with no evidence. I’ll gleefully admit I could be wrong – though I’d be very surprised to be badly wrong.

But the possibility of Google reducing the rank of a disfavoured competitor, or slightly inflating the rank of a preferred information supplier, is something that gets me twitching. There’s no oversight for Google. We have only their word that they are using what they publicly claim to be fair. If, in the depth of a recession, California based Google were to be weighting US businesses slightly higher in the UK than they should be, it would direct between tens of millions and billions of spend to US companies and away from British companies. Same for France, and Germany. Is Google making sure that the US economy brings in a little more business than it *should* do? I have no idea. I can’t measure it. I can’t tell. It does worry me. I worked in the USA for about a decade, and I know how insular US businesses can be. The rest of the world has fewer investors, fewer buyers and works in funny languages and currencies; it is easy to fall into the model of thinking about what works best in the US economy and rolling that out, worldwide.

So there *may* be some basis for European businesses to wonder if Google is deweighting their business. It may be deweighted because it doesn’t fit a market model that Americans understand. It may be deweighted because an American business is, even unconsciously, working for the end of a recession in their domestic market. It may even be a conscious effort. That really, really worries me – because I don’t think even a national government outside the USA, could get the data to investigate whether Google is deliberately, or unintentionally, diverting economically significant activity, artificially, to the USA.

Do Ciao and Foundem have a case? I don’t know. I’m looking forward to finding out, if it is released, just what evidence might make them think they do. I hope it’s a lot more compelling than “we want our share of the Google billions”, “Microsoft is using me as a sock puppet to attack Google” or “Google’s Buzz is creating a weakness in perception we can exploit”. I fear there may be some interesting biases in Google’s system that do favour US businesses – simply because I can’t see any way to prevent there from being such a bias, or to detect one if it were present. And *that* has implications for national security – the protection of the realm. Interesting times.

Updates

2010-02-27 – SearchEngineLand article about Microsoft’s involvement.

I occasionally see PPC experts stating that QS isn’t as important as some other factor that they are promoting. This video makes it clear that QS has to be a central component of managing Return On Investment, and can even significantly affect the costs of an Awareness program – in which ROI is less important. The importance of the account’s accumulated Quality Score and long term response by users – longer term than CTR – can have huge impacts on paid price, which in turn benefits ROI. QS isn’t *only* CTR or *only* Landing Page factors.

Although this doesn’t explicitly describe the Content Network – AdSense placements – I’m pretty sure that there are differences. SmartPricing is not covered. In my experience whatever it is that SmartPricing does, can be counterintuitive to advertisers, sometimes with much more paid per click to high volume non-converting sources than to low volume repeat converting sources – I’d rather see a higher CPC to a site that sends my clients repeat converting traffic. Following Hal Varian’s suggestion at the end of the video, I’ll be asking for a clearer explanation of Content Network Pricing, especially with the new behavioural targeting system that has entered Beta.

• Google will struggle to retain revenues using a variety of techniques
• Searchers will spend more time browsing and convert after more clicks
• Online revenues will generally increase – but business margins will be squeezed
• Internet Theft Scandals – Click Fraud, Phishing and Account Theft

Details and the consequences? Read on…

Google Will Adjust To Preserve Revenues

The basic idea of paid search is simple. Searchers submit search queries, and advertisers pay to have their adverts shown to more or less interested searchers. This meshes with the purchasing process (buying model) in several ways. At the most basic, when someone searches for your unique trademarked business or product name, it is likely that this searcher is intending to buy something. More likely than if they’d typed a competitors name. When they search for something that describes the product category, they are less likely to buy – they are in an earlier stage of the buying process.

Auctions have some basic characteristics. As a broad generalisation, the more bidders there are in an auction, the higher the revenue for the sellers. So Google’s goals have to include increases in the number of bidders in an auction – Broad Match and adding ever more advertisers all help with this. Convincing advertisers to bid high – traffic scales heavily with position; Google needs to convince advertisers (or, rather, prevent them from becoming aware of the relationship) that position and click quality are not related, but that position and volume are related. Finally, Google needs more outlets, so that advertisers see more reasons (more impressions) to be part of the ecosystem; however, that largely means adding new, smaller volume, publishers – some will have niche specialist interests, but overall this encourages more Made For AdSense sites, which, IME, have an appallingly poor conversion rate and value (reducing click quality by adding more outlets).

You’d expect that in retaliation, advertisers would then want to stick to Exact Match and using only Google’s Search Pages. There’s other factors at play, though.

Users miskey – they can’t remember the name of the company properly. So someone looking for the travel company Thomas Cook might key “thomson cooke”, “tomas cuik’ and all sorts of other variations. This means that Exact Match isn’t enough to identify all people looking for the trademark or brand name. Broad match helps by allowing Google to send these near misses to the right advertiser. Google do a phenomenal job of matching; if an advertiser like Travelocity doesn’t use “cheap holidays” as a search term, then Google will match a high bidding Broad Match keyword to that search.

People will also type more specific searches, longer searches; most searches, by a long margin, have more than two words in the search query. Users will type “logitech mac support” or similar, to more rapidly jump to stuff they know exists somewhere in the vendors site. Some of these longer searches won’t lead to sales – as in this example, some searches are support inquiries. So Exact Match also fails as being too specific, because these search queries wouldn’t be matched. Phrase Match is useful to find unexpected variations and more specific searches – leading to the idea that you can help users to jump deeper into the site with the right combination of technology and advert. Again, typos and miscomprehension by users will mean that Broad Match captures additional searchers who intended to find the business, but would have failed with solely Exact or simple Phrase Matches in the campaign.

Google’s problem is that when someone has decided on “Honda”, or “iPhone”, the searcher is unlikely to be deflected by alternatives. The consequence is that there will be few competitors for a brand name. Brand names are usually the best return on investment – so the average costs per click will tend to decrease for brand names. This is part of what you’d expect from a basic analysis of the buying process – you generally type a specific company name after you have researched and typically just before buying. So there is an inbuilt pressure to decrease advertising on exact matched competitors name – resulting in lower revenues from trademark terms. That’s a problem for Google.

The earlier phase searches, which tend to be naturally higher in volume as people look for alternatives from which to choose, naturally imply a worse ROI – you need to pay for more clicks to support the research, and the conversion rate is lower because searchers are researching. This too tends to drive Average Cost Per Click down, in order to retain a positive return on investment.

Google needs to keep advertisers focused on Broad Match. This lets Google place unsold inventory, offer competitor adverts on trademark searches, and so on. The goal for Google is to increase the number of participants in the auction (helping increase average cost per click and hence inreased revenues), to find ways to increase advertising on unsold inventory and to affect the minimum bid.

Some techniques that are likely to be used by Google?

Expect to hear more about the difficulty of reaching interested searchers with organic search, now that personal search is becoming more deeply embedded. If advertisers think that they are missing audience because of personal search, that they can only reach with paid search, then there will be more advertisers and increased competition. This will be partially driven because of the effects of recession… in some market segments there will be reduced interest, and this will naturally translate as reduced search volumes, resulting in fears that personal search is sapping share of voice.

Expect to hear more about Quality Score changes, probably surrounding the calculation of the minimum cost to appear on a page. This is not the same as, but is confusingly similar in name to, the estimated cost to appear on the first page. The intention will probably be that if the CTR is lower than some target value derived from the rest of the network, that the minimum price is adjusted to meet some revenue goal – it won’t be written like that – I can’t guess how Google will describe the technique, other than that is likely to involve some description about “improving user search experience”. There will still have to be a way to allow $0.01/click – a feature that attracts many advertisers, few of whom have any real chance of achieving this… but it remains an important differentiator over Yahoo and the other competitors in this space. I think it is possible to offer both, because you only get $0.01 on high volume, high CTR keywords – IOW, not things that affect most advertisers.

Expect more dilution of “real search”. Google still wants additional outlets and will continue to re-present domain parks, “fake search”, selected content match outlets and other “opportunities” as if they were what naive advertisers expect from keyword search (that is, that the advert is shown directly in response to a real user search that leads to a directly relevant site – much as happens with organic search results). Google will want the increased impression volume and the *overlap* of keywords with different intent that allows a single outlet to have larger counts of competing advertisers.

Expect more messaging about the synergy of paid and organic search – that even when you have achieved page dominance for the targeted keyword, you should still be advertising as well topping organic rankings. This increases competition, ensures that there are at least ten plausible advertisers, etc. I expect the messaging around this to increase as advertising volume and value shrink.

Expect more overseas advertisers. Google doesn’t offer USD bidding to smaller overseas bidders. The exchange rate isn’t notified to overseas bidders. So, especially when currency exchange rates are in flux, Google can make margin on exchange rates. I expect that Google can easily and all-but-undetectably collect additional revenue on overseas bidders, by tweaking exchange rates – AFAICS, the auction is held in USD, so exchange rate changes are needed for anyone not using USD. This would be non-US income – no-one in the USA will blink at using exchange rates to improve revenues from non-US companies.

Radically – if Google were to remove Exact Match, they could make dramatic transformations of revenue expectation. Would they do this? I think they would, if they could claim that it improved the search users experience. If advertisers fail to appear on a substantial fraction of searches, especially when those clicks turn up in organic search clicks, it would allow Google to say that exact match was preventing searchers from seeing the results that they want to see. This would be a huge step for Google, and they’d probably need a lot of research. It’ll be piloted by some large accounts and evidence will probably be drawn from wide-category vendors, like eBay. It’ll turn out to be complete rubbish for narrowly focused vendors, but Google’s objectives are maintain share of search, and only secondarily to satisfy advertisers – the auction will take care of some that advertiser anxiety.

Increased Search Time & Reduced Conversion Rates

Most people are planning on reducing spend. They’ll do so by spending more carefully. They’ll look around more. They’ll be taking more personal recommendations. Trust in the stability of large organisations and well known brands will continue to be eroded – which means that the right businesses, with the right trust validations, can emerge from nowhere; just because your business has been running for 5, 10, 50 or 150 years is no guarantee that you haven’t recently based your business on false expectations of investments and earnings. A new business with the right accreditations can play on the same field as a 200 year old business, and can raise fear and doubt about the financial stability of well established players.

So, shoppers will be more wary. You’ll need to improve your sites to have online messages address current concerns. The fears, uncertainties and doubts that were addressed in marketing communications last year, won’t work so effectively this year; the concerns are different. Your customers will be concerned that even buying from a major brand will damage them, or is at least risky. Reassurance and validation will be important – especially with the media focusing on bad news. Bad news sells and media outlets will want more viewers. The at-risk media outlets, old broadcast media, seeing declining share of advertising budgets will want to stimulate sales through increasing interest in controversy, further weakening consumer confidence. Failure to address offline media reports online will impact conversion rates – though you will want to avoid directly feeding the controversy.

Minor brands should be able to gain, if they are cash flow positive or can find investors with imagination, foresight and cash. Guarantees and warranties, future-safe products and ways to reduce service costs for consumers; ways to manage domestic finances effectively; ways to cut back while still having luxuries.

The results of these will be to put some strange pressures on paid search. Companies that have previously survived on word of mouth and low or zero advertising, will need new customers. They’ll advertise. As new advertisers, they’ll make a lot of mistakes and won’t be willing to pay for expert assistance. That will increase claims of click fraud, and apply upward pressures on bids, sometimes from advertisers that shouldn’t be in the auction.

Existing advertisers will want to improve performance and will decrease budgets and target the spend on the most effective keywords and adverts. This will increase pressures on the paid search companies to dilute the inventory – the result is likely to be that overall clicks/conversion will continue to worsen, otherwise the paid search companies will report losses, and decreases in search volumes. It’s always bigger news if a big company loses, than a bunch of smaller businesses – so expect that larger businesses will use their asymmetric control of information to manage smaller customer businesses expectations and margins, through manipulating click quality. Expect the search engines to protect themselves at the expense of smaller advertisers.

ROI will generally worsen, but there will be islands where specific companies have struck the right messages that resonate with users. The result will be that paid search competition will continue to heat up. I expect that the average CPC will decline, the total value of paid search will decline, but some advertisers in various niches will have justifiably higher Average Cost Per Click and increased spending.

Affiliates

The affiliate industry will also become even more heated. Out of a job? Looking to make money with a low capital investment? Then you could become an affiliate…

However, who needs more novice affiliates? What does an influx of untrained, cash starved and at least initially ineffective affiliates do, when the real super affiliates (not the one man and a dog operations blogging from “super affiliates are us”, but the real, quiet, and highly effective super affiliates) already handle about 80% of affiliate traffic? The answer is that these new affiliates will provide free advertising for marginally effective businesses, by spending their own money essentially as an investment. Expect more companies to switch to affiliate advertising models to control their in-house marketing costs and to reduce spend on advertising agencies. Expect a lot more annoyed novice affiliates.

The affiliate industry will boom; but leave a lot of disillusioned “internet advertisers” in their wake. There will be some emergent new stars – not everyone who comes in will fail, and some existing leaders will cash out and move on. Expect some churn in the top affiliate products – but that’s pretty standard in the industry anyway.

I expect that the real super affiliates will be the same next year as this year – they have developed their techniques, and they are effective. They might retrench a little and change their focus on the businesses they are interested in, but they’ll survive and probably continue modest growth; their main obstacle to growth has historically been cash; 60 and 90 day payment cycles on sales conversions mean that working cash is tied up, effectively limiting them to “four to six inventory turns” a year. That reduces their rates of growth, and with banks not in lending mode, these guys are bottled up, at least in paid search.

Online Revenues Will Increase

Well, they will unless the internet theft scandals are outrageously large. In a search for the best value, comparison shopping online and internet supported purchases over the phone will become more important. Where businesses aren’t already selling online, there will be pressures now to do so, to reduce the costs of sales. The final result will be that more is sold online – even though total sales from all sources will decline. There may be some countries where there is a decline in online sales, but it will be less than the decline in total sales – the internet will be seen as one of the brighter spots. Just.

There’s only one reason for internet sales to increase. If executed effectively, you can reduce the costs of making the sale. That tends to be less and less true the lower the volume of sales; so expect continued outsourcing to places like Yahoo!Merchant Stores, where the infrastructure and development costs have already been put in place, and allows dropping the cost of sales. The interim step between often dreadful sites with no compelling call to action and a full online sales site, is a phone number; but placed on an inactive site that doesn’t reflect your best price and the unique value in buying from you rather than a competitor, this will have no effect. So expect the more savvy small business to look for CMS based websites where they can update their own content. I’m expecting that smarter SME’s will be investing more in templated CMS backed websites than the current static crop.

And metrics. Web analytics – free tools like Google Analytics – properly used, can reveal a lot. The problem for SMEs is setting up and understanding the data. There should be an increased demand for skills in setting up analytics, and interpreting what is happening.

Internet Theft Scandals

With money tight and some smart people out of a job… expect internet fraud to increase. With businesses looking for excuses to write off problem debts, expect more disclosure of problems, probably triggered by an inadvertently exposed online scam. This is obviously a tentative projection, as it depends on unforeseeable circumstances. However, this year is the first year since 1994 (when I started internet sales seriously), when I can see reasons to add excuses to the balance sheet, some understanding of the risks to businesses and that even corporations are affected by fraudsters, and a large value of online business that probably will increase (at least in relative terms), and an increase in technological capability by scammers and fraudsters. The combined pressures might make it attractive for the first time to attribute losses to technologically sophisticated thieves.

Online security for the average consumer has not improved in any seriously identifiable way since 1994, other than the progressive plugging of vulnerabilities in web browsers and servers. The introduction of Secure HTTP (https, or “secure server” technology) back then, provided users with a somewhat artificial degree of confidence. Many web sites offer access to financially significant resources with only an account name and password. Account names and passwords are insufficient for best security practices. Banks now often offer multiple levels of password and CAPTCHA, and may require authentication through encrypted PIN checkers – beginning to approach the holy trinity of security (“something that you are, something that you have and something that you know” – at least two of those are addressed by the better banks). However, there are still far too many ways to spend money online that are protected only by guessable account names and passwords, and by essentially unprotected mechanisms to send money to scammers.

There will be scandals about this. Probably soon. If there’s a sniff of a problem, then the offline media will pounce. These media need to pounce and will argue forcefully; their businesses are in decline, and they’ll need to savage the failures of online businesses to help protect their own. While this would be a significantly negative sum game for all online sales, the benefit of defection by an attacked business will be high; “We really made money, and we only show a loss because of internet theft we couldn’t control” will be an attractive excuse to some business, at some point. And then the skies will open and the extent of internet crime will be an issue.

That will damage sales. Hugely. Even if the effects are already part of current accounting systems, reported as part of business accounts and effectively managed by all online businesses, the *perception* of reductions in safety will be very damaging. I don’t see any sign of widespread adoption of even “best practice” account management – internet wide consistent login methods and messages, proper understanding of what a “secure server” means, and so on. While users are prepared to pay money to sites they can’t properly identify in return for promises to deliver products they don’t receive, across national borders (losing nationally accountable and interested police enforcement), this problem will continue to grow. The individual losses are small; the investigations complex. At some point, this will become a more serious problem to deal with than Enron and Madhoff. I think it’ll be this year.

Consequences

New advertisers will start by thinking about paid search – increasing pressures there. This will result in disappointment and claims of click fraud, especially if new advertisers discover that a click is not a click… different clicks have different values, even if Google conflates clicks from multiple sources. Expect a lot more noise and heat from new advertisers who feel they have been mislead.

Existing advertisers will mostly reduce and focus spend. AvCPC will have a downward pressure, counteracted by the SE’s including low value inventory – weakening conversion rates and hiding it in a generally slower market with inbuilt tendencies for longer latency and reduced likelihood of sales. ROI will get worse; the question is whether the business can take a few quarters of bad ROI in order to survive to an upturn. There should be increased attention to improving the web site, however, this will be diluted by the feelings of senior management that enough work has been done on the web site – “it’s a finished product” – and that there have been sales, so all that is needed is more visitors. IME, that’s usually wrong – there’s usually a lot of ways to make a site more effective in selling, even for long latency sales.

There will be a switch to increase SEO efforts. If purchasers will spend more time in research anyway, increasing the clicks and site visits per sale, then organic search results become more interesting. This will increase interest in spammy linking as people learn their way into search engine marketing. There will also be increased interest in affiliate advertising – if you aren’t skilled in internet advertising, then getting affiliates to advertise on your behalf will be interesting. However, this again should result in increased interest in web site design improvements; it won’t, because once a business have “put your brochure online” the naive perception is that there’ll be nothing else to do. There’s still far too many naive advertisers with ineffective web sites and a management team that focuses on volume of traffic, not the quality of that traffic and the quality of the response from the website.

There will be a decrease in search volumes for products. Despite the increased research for purchases, search volumes for commercial products will be negatively affected. That’s partially because the economy is slowing, but mostly because people are learning the online environment; they know where to go to get various things at decent prices, so they don’t have to do so much searching and researching – I haven’t heard anyone complain for the last 18 months that they are “useless at searching”. This year may be first year where the combined effects of knowing your internet neighbourhood and slowing commercial activity, actually dent the growth of both paid and organic search.

A significant internet theft scandal will cause a dramatic decrease in online consumer spending, if it bursts. Expect new web browser technology and new web site authentication services. At the moment, I’m pretty sure that the only way to tackle the identify theft problem is solutions that, as an industry, we’re nowhere near considering. The short term consequences will be to push the internet clock back towards use supporting research for purchases, not for actual online purchases.

CMS’s will become more important for SME’s. They’ll finally be able to build customised landing pages, important for improving the effectiveness of paid search.

Web analytics, neglected or misinterpreted, will increase in significance – and Google Analytics low price point makes it a very attractive platform to use, despite the poor general knowledge about how to best use it. The downside of “free” is that expectations of the value are low, and users are unwilling to pay a lot to understand the benefits; smarter businesses will invest in learning how to use GA effectively (IOW, not as it comes out of the box).

Weaknesses in the paper

There are a few embedded misperceptions in the “AdFraud Anecdotes” PDF.

I’ll probably add to this list as I think more about it.

• Advertisers do not react rationally. That’s partially a consequence of information holding asymmetry.
• Google controls Broad Match and Auction Quorum Sizes to improve Google’s returns – and hides the crap search queries in reports as “18,0000 other unique searches”
• Google is in complete control of the quality of matching, by default. Google controls click quality, advertisers choose bids and can forgo volume by selecting phrase match and exact match, or accept Google’s decisions about matching. Most of the variation in CPA & ROI that I find can be directly attributed to Google changing the nature of search queries that are being matched.
• Advertisers are remarkably reluctant to measure – because for a small business the costs of developing the understanding of measurement is expensive; so is hiring in the talent to understand the data. I infer that small businesses have a higher percentage of undetected and undetectable click fraud and greater difficulty in establishing a “useful for management purposes” CPA (internet noise makes data collection long period – I may write up the details of this).
• The paper addresses large advertisers quite well. Typical of economists and governments – but single large entities are *not* representative of the larger numbers of smaller businesses – scale changes impact.

Strengths of the paper

This shows that some people within Google are clearly understanding the role that Google has.

It is interesting that the thinking and insights exposed in this paper *haven’t* percolated to the AdWords Sales Teams, at least in the UK. I’ve recently had a rather unsatisfactory meeting with a major account team, who said that Google has no insight into conversion… Despite our mutual client having AdWords Conversion Tracking enabled for several years. If ROI was important to Google, you’d have thought they’d have noticed, and been rather more interested in how recent change have affected ROI. Wouldn’t you? Instead, I got the usual lecture about increasing spend… instead of them giving me techniques to manage AdWords to improve ROI in the face of Google’s efforts to undermine that. I may write about that more, when I can work out how to disentangle any explanation from any specific clients’ data.

The PDF provides some rather Delphic oracular premonitions of the recent user interface changes, for example, separating metrics between Google Search and the Search Partners and the Content Network; allowing (some) control over domain parks, 404 page advertising, etc. Careful reading is required and may be rewarded.

I may make some predictions about likely further AdWords UI changes, when I’ve had a bit more of a think… Frankly, stumbling on this has been a bit of a shock…

Material Disclosure: I am not, and have not been, employed by any search engine companies. My 30 year old degree is in Cybernetics Science, and my main academic research involved dynamic thermal models of houses and measuring low energy housing performance. Anything that I’ve got right about economics is through The Economist, New Scientist, and a few influential books – see the bibliography in the next article in this series. I’m pretty happy with this analysis – it affects how I perceive the search engines, but YMMV. This kind of publishing is the closest that I get to a peer review of my thinking. Well, that and paid search results…

Simple Auctions

One of the most basic auction mechanisms is that every participant offers a bid and the highest bidder wins. Another favourite, sometimes used for government contract bidding, is when all participants offer a sealed bid, and the second lowest bid is the winner. The rules of the auction determine who is the winner – and those rules therefore influence strategy. In the first example, you’d need to bid the most to win; that strategy would lose you the winning position in the second type of auction. This coarse example should make it clear that understanding the model that the auction uses will affect bidding strategy.

The original Yahoo!Search Marketing auction, when it was GoTo.com and Overture and prior to the rollout of the still fairly new Panama system, used a slight variation of the simple “highest bidder wins” auction. Bids were ranked and the advertising slot you got was based on the rank of your bid. For example, if you bid the second highest amount, then you showed in slot 2. You could consider it an auction for each slot, where the topmost bidder wins each time, and is then removed from the auction for lower places.

This model, or variants of it, tends to be used in the second and third tier paid search vehicles. It is simple, easy to understand and has several crucial flaws:

• Simple auctions don’t maximise revenue (bid x impression volume per position x advert CTR, is not optimised)
• It distracts advertisers from focusing on relevance in favour of technical bid twiddling.
• Search Engines can’t broaden the keyword, which reduces revenue

There’s one more twist to a standard auction that is common in paid search. The offer that “you pay only one penny more per click than the advertiser below you is bidding.” If you are number one with a bid of $10.00, and number 2 is offering $0.23, then you pay $0.24 per click, not $10.00 – at least, not until advertiser two notices what they can do to your advertising budget with a simple bid change… This “penny more per click” offer also has an interesting effect on auctions and revenue.

Revenue Optimisation And Simple Auctions

Let’s illustrate an example. We’ll pretend to look at a generic keyword like “holiday”. The number one advertiser is a new and little known business. They get a CTR of 2% because they have focused on flogging the tagline for the business and have failed to mention the word “holiday” in the advert. Instead their message is something about brighter futures and feeling less bloated; totally incomprehensible as a holiday advert. The number 2 advertiser has an 8% CTR. Their advert mentions “holiday” and they have a well known brand name. If the order of the two adverts was swapped, even though the second advertiser is bidding less, then the publisher would make more money – because searchers will further increase the click rate for that more relevant second advert. Even if the previously number 2 advertiser pays no more per click than they were paying in position 2, to gain position 1, the higher CTR advert in this position will yield a higher revenue for the publisher. This is most easily understood as thinking that the CTR for the advert will increase when moved to position 1. More money makes for happier shareholders and bigger bonuses.

There is an additional incentive to consider a different auction method, for the search engine. It even helps advertisers, too. Strictly, it helps aware advertisers. If you have any clue what messaging means, and can divine the searchers’ intent, then you can build a better advert than your less aware competitors. If you can deliver adverts that attract more clicks per page, then you have arguably improved the search engine experience for users. That should make the search engine into a more popular property because users can know that adverts as well as organic results are pretty well optimised.

Now, it isn’t always true that higher CTR means higher revenue for the advertiser – I’ll address that elsewhere. It’s a subtle point, but quite important for certain types of business.

It’s the Relevance, Dummy

The old Overture bidding model spawned a collection of companies offering automated bidding tools. With 96 chances to update the bid per day, you could work out what bidding patterns your competitors used and do things like targeting large gaps, and bidding high under competitors, so that their adverts cost more, and so on. The emphasis was not on servicing the search users, but on technological twiddling.

I’d typically find, on taking over an Overture account, that the adverts were all very bland, very generic, with low CTR – but someone had invested a lot of time in bid management. Completely the wrong focus for both advertisers and publishers.

Open Auctions

Some auctions are open – all the participants can see what the currently winning bid is. This is typical of animal, car and property auctions. When there is only one item to be won, you only need to know the topmost bid to know the winner. However, paid search advertising is subtly different. Instead of wanting a single winner, the goal of the auction is to find multiple advertisers and rank them by some criteria. The simplest extension of that is to take the “winner takes all” auction and extend it. If there are three advertising slots, then the top three bidders take the slots, in the order of the bids.

However, that model has a weakness. It assumes that the bid and the value to the advert publisher are the same thing. They aren’t, in many practical cases. To illustrate this, look at the numbers below:


Bid: 1.00
CTR: 1%
Revenue: 1.00 per 100 impressions or a CPM of 10.00

Bid: 0.5
CTR: 2.5%
Revenue: 1.25 per 100 impressions or a CPM of 12.50


The second advert, although offering a lower bid, has a higher CTR and is therefore a better value to the publisher. It is also, interestingly, more likely to be considered valuable by users of search advertising – more of them click the second advert than the first, suggesting that it has a higher perceived relevance.

By making the auction open, you can make some assumptions about competitor adverts and bid higher. However, rationality often flees at this point. For example, brand owners may insist to their agency or staff that they appear first in results. If this strategy can be identified then competitors can drive up the cost to the advertiser at little expense to themselves. Most people won’t click on competitor adverts and they are even less likely to do so if the advert is deliberately made unattractive.

This example shows how a business can drive up advertising costs of a competitor at no significant cost to themselves. An open auction is not what you could call a fair system. It leaves advertisers feeling dissatisfaction. On the other hand, this effect is advantageous to the publisher, because the CTR on the first advertiser is likely to be reasonably high, and their bid has been forced up. Brand terms aren’t the only searches that yield conversions, and they arguably cannibalise organic search – choosing to use or not use a simple auction just because of the effect on Brand keywords would be perverse. There are better reasons to consider changing to another model – as evidenced by none of the top three continuing to use a simple, open, auction.

Ignore The Man Behind The Curtain

AdWords says that it is a Pay Per Click advertising system. But is it?

You certainly appear to pay with every click in the keyword advertising campaigns. If one click costs $0.10, then two clicks costs $0.20 (or so). The value that you pay is quantised at the penny interval for each click. So it sure looks as though you are buying clicks.

Let me give you a disturbing vision… Google is actually operating a CPM bidding system. Yup. Your bid position is a consequence of the value you give to Google, per thousand impressions – but quantised to a penny per click.

By ranking adverts according to the CPM, Google can overcome many of the defects of simpler bidding systems:

• Search users get (mostly) more relevant adverts
• Revenue is optimised
• Keywords can be matched to a wide range of search queries

There are some consequences that drop out and further subtleties in choice of the auction mechanics that affect the revenue that can be extracted.

In the next article, I’ll look at what happens with a CPM model and quantised CPC payments, what the “Quality Score” is likely to mean for auctions, and various other implications such as the assumed CTR before you have a statistically relevant volume of impressions.

Updates

2008-07-29 – The Register Describes AdWords AutoMatch – partially pre-empting the next part of this series. Minor edits for clarity.

2009/02 I can no longer detect Search History Permutation using the diagnostic tests that I previously used – I believe that this is no longer operating, or it has become more subtle in its effects.

Original Article

I was wrong – Search History Permutation probably isn’t the sole cause of problems. Amongst my clients, I can see Search History Permutation (where the words in the current and previous search query are combined to generate new searches, and adverts are conscripted that the searcher wasn’t lookiing for). However, other clients are also having new problems, not obviously caused by Search History Permutation. I don’t, yet, understand what else is going on. But let’s at least get the Search History Permutation out of the way.

One client has had Google Analytics running for a few years – so I can do some basic time series analysis. If I do a cross correlation on impressions, click, visitors and conversion rates, they are consistent from Feb 2006 until approximately March 1st this year. Now, that’s *after* the first point at which I became aware of Search History Permutation. So there may be something else going on, too.

Search History Permutation Signals

If you have been hit by this, then Search Queries will start turning up in web analytics, that have one or two words that you’d normally expect, and some other odd words that you might not expect, or combinations of words that you would expect to see in different keywords. For example, if you are advertising using “free dentistry” and “dentist poughkeepsie”, you might see search queries for “free dentist”, “free poughkeepsie”, “dentist dentistry” and “dentistry poughkeepsie”, as well. Some of these are similar to the sorts of expansion that Google does already with Broad Match. If you used “free dentistry” and saw “free dentist”, that is exactly the sort of expansion that Broad Match is already capable of delivering. However, “free poughkeepsie” is *not* normal – well, not for a dentists keywords, anyway.

How frequently you get hit by Search History Permutation will depend on the precise mix of keywords that you have in the account. If you happen to have keywords where searchers will typically make consecutive searches that each include a word from your keywords, then you might get dragged into unexpected auctions to advertise for searches that you don’t expect to appear in.

The frequency appears to be highly variable – but only for certain types of account. What distinguishes those accounts? They are high bidders, with niche keywords. When searchers have a set of searches combined, then the rare niche keywords are triggered at an abnormally high frequency. Because the bids are high, these adverts are conscripted to take part in auctions where the real search is very different from the targeted keywords.

In accounts with low bids – perhaps because, like travel, the industry has a lot of browsers and relatively few bookers on any one day – the effects are less easy to see, because most of the combinations that will be generated, end up either with no advertiser (“cheap family” – taken from consecutive searches for “cheap holiday” and “family holiday”), or with plausible Broad Match relevance (“holiday holiday” would be a plausible synthesised search, which would trigger Phrase Matched and Broad Match holiday adverts).

So the signal will be, in many accounts, subtle and all but invisible. This means that regular activities, such as adjusting bids to keep inside ROI or profit targets, may conceal the effects. There will be a small proportion of advertisers for whom the new Search History Permuted Searches form a substantial fraction of their impressions. These are the advertisers who suffer – but they need those peculiar combination of search queries to trigger them.

Looking in detail at some mass market and lower bidding clients, suggests that the frequency of unusual search queries is about 1 in 30 to 1 in 60 (about 3% to 1.5%). This is hard to distinguish though, because I normally run some Broad Match keywords in these accounts anyway, and Google’s natural extension of Broad Match could have snared some of these searches. The frequency could plausibly be as low as 1 in 50, and Google may have turned up the dial on Broad Match to capture a wider range of queries (which, if you’ve read the previous articles, means that the auction will tend to generate more money, because each auction will usually include more bidders and that drives up the paid price).

New Signal?

Cross correlation is a statistical technique that asks “how similar are the variations in these two data sets”. If I have a daily record of impressions, for example, I can break it into two periods, and compare, as in thi case, Feb 2006 to Feb 2007, with the range Feb 2007 to Feb 2008. I get a strong correlation co-efficient, meaning that every year, the pattern is pretty similar. It is even better when I leave out the fixed holiday dates – Christmas and New Year fall on different days of the week.

Looking at my mass market clients, I can see that there may be a signal of broadening Broad Match, aside from the Search History Permutation. I have accounts where I have managed the daily spend to approximately the same budget spend for ages – given that there is a weekly profile of search, when daily variation is expected, so long as the weekly profile is maintained. These have suddenly escalated since March 1st. The bids were the same, the keywords and adverts the same – and thousands of active keywords are involved. Why would the paid price escalate, suddenly?

This may actually be other side of the Search History Permutation, though. Consecutive searches mostly include implausible searches that wouldn’t trigger anything, or plausible searches that weren’t part of the current keywords. So in a small number of occasions, additional advertisers are recruited. This pushes up the bids. Even an appearance as low as 1 in 40 (2.5%) causes a measurable price change in the auction, when you have a hundred thousand auctions a day.

Date Inconsistency

Now, I believe that I first saw “odd adverts”, which I didn’t immediately connect with Search History Permutation, some time in late January. A lot of AdWords advertisers make insufficient use of negative keywords, so I’m used to seeing inappropriate adverts showing up from time to time. However, I’m pretty sure that the effects were visible, at least sometimes, in January 2008.

So why do most of my accounts that show a change, show it from March 1st, rather than January the somethingth?

I have no idea. None whatsoever. I’m worrying at this discrepancy. What if the changes in prices and impressions that I’m seeing are not caused by Search History Permutation, but some other algorithmic tweak?

Well, I do see *some* visible effects – there are some odd searches in the logs – odder than I’d expect to come from Broad Match.

You know the biggest signal that I’ve found something? Visits from the part of the world that mostly houses Google, to the blog article about Search History Permutation. When the first article was published – either the Googleplex decided my recreational pharmaceutical mix was unusually strong and they couldn’t stop laughing, or I struck a nerve, because it is the highest daily readership from Palo Alto that I’ve seen – IP addresses in the Google ranges. (No, I don’t really do drugs stronger than coffee or capsaicin – sheesh, a joke).

There’s probably still a piece of this puzzle that I’m missing. Ah well, back to trying signal analysis techniques and staring at lists of keywords and search queries, bids and average paid prices. There’s some way to make this make more sense – I just to have find it.

Summary

Search History Permutation probably has an effect across AdWords on the order of about 1-2% of searches, though the effect may be more pronounced especially in niche markets.

The effect appears to be an increase in Average CPC of somewhere around 3-5%, but may be higher in niche markets. I haven’t sorted out the maths to justify this, just observations, so far.

For accounts with the right niche keywords and higher bids, the effect may be pronounced – sometimes drastically reducing conversion rates, possibly on the order of -90%, because so many of the wrong searches are brought in. For example, if you specialise in one named category of something, and bid high for it, then I think that you’ll be shown much more frequently for the mass market searches than the niche searches – and that will destroy CTR, conversion rates and profit margins.

2009/02 I can no longer detect Search History Permutation using the diagnostic tests that I previously used – I believe that this is no longer operating, or it has become more subtle in its effects.

Original article

My first article on Search History usage was experiential; you can do the searches yourself and see the strange results. This article offers a different type of explanation with a lot more detailed argument. It raises the question for me – is Google’s use of AdWords Search History to generate adverts for unrequested keywords, fraudulent? The development of the argument travels through marketing, micro-economics and a bit of stats. I’ve tried to set a background that makes this easier to follow – but it is a bit lengthy. Sorry ’bout that.

Signs And Portents

I treat AdWords a bit like cryptography, or black box control engineering. Messages go in (adverts, keywords, bids, geotargets, etc) and translated messages come out (impressions, clicks, conversions, paid prices, etc). Google does something in the middle. It’s part of my job to infer what Google does, and manipulate inputs to optimise the output. There’s a bunch of fancy techniques that could be used (Stochastic Perturbation with Simultaneous Annealing, for example – what a wonderful name, eh?), but much of it comes down to, in my opinion, marketing messages, the buying process and micro-economics. The complex maths get their day only with large data sets and high volumes.

For small advertisers, the volume of data needed to make techniques like SPSA and Taguchi work, are hard to achieve and pay for, and subject to a lot of statistical noise caused by uncontrolled and/or unmeasured factors such as seasonality, market price fluctuations, confidence, organic search results, press releases, Google’s changing policies and procedures, the Editorial Review process, trademarking protection, and so on. Mostly I rely on techniques that appear to work in general cases, with tuning when enough historical records are established.

I determinedly read messages, signs, sigils and portents from Google – blogs, Wall Street Analysts reports, Nielsen, the occasional goat sacrificed at midnight, etc. But the most important pieces of evidence come from the AdWords account, from web analytics or web server log files, web site contents and visitor trajectory, and from looking at competitive advertising.

The consequence of a lot of thinking, experiments, and the experience derived from spending a few million of my clients’ funds in advertising budgets, is that I’ve developed some techniques that give me the most data for interpretation. I’m quite happy to share these techniques – my primary value to my clients is “insight”. Paid search work is just one tool in that shed.

I’ve found that it is most useful to focus on Exact and Phrase matched AdGroups. There’s a specific way to set these up to get the greatest signal – I’ll detail it some other time. But the effect is that even if the bid for exact matched keywords drops, that the phrase match and broad match adverts don’t run – think “negative exact keywords” and “negative phrase keywords”. Also think “This’ll Stop Google From Abusing Expanded Broad Match And Destroying My ROI”.

This technique reveals some intriguing things about users and search. Most importantly, it used to show how and when users repeated searches.

Characteristics Of Search

If you have a rare search query, as an exact matched keyword, it might only attract a few hundred impressions per year. As Google have become more experienced, they’ve started aggregating small volume queries, so this signal is harder to see now, than it was in 2003-2007. However, old records show something interesting. If you have one of these “long tail” queries, then you see that they tend to get used in bursts. You get no activity in months, and then you get a run of 2-15 (usually around three to six) searches with the same query. Then it goes quiet again, perhaps for months.

Why? Because one searcher has decided to find something and is repeating their search. I know that I do this. I’ve seen other people do it. You are trying to research around a problem and you keep coming back with the same query, perhaps to look at more results on the page, or to get a second or third page.

Patterns of search query repetition appear to vary in some ways that are amenable to analysis. AFAICS, it changes with how the search is regarded by the user (what I call the “intent”), and the phase of the buying process. If the searcher is in the early phases, then they do a lot more repeat searches. If they are late phase, for example after making up their minds to buy from a vendor – then the number of repeated searches tends to be small – they know what they want and they just have to find it. A characteristic migration of phrases might be to start with “Cheap Holiday” multiple times, then some specific searches for types or locations of holiday which may have some repetitive components, and ending up with a single search query for “Expedia.com”, “travelocity” or some other site that was identified as having the product to buy.

This evolution of search queries interacts with the new search history permutation mechanism in a regrettable way.

When a user repeats the searches, the results stabilise, and are just like the results you’d normally get from AdWords. If there’s been a break in activity, then a new search after the break duration is treated as if there is no search history. For example, type “Brazilian Tax Credit” tonight as the last search you do, then come back after a nights’ sleep and search for “US Vacation” and you *won’t* see the Brazilian Holiday and US Tax adverts (from the example in the previous article). Well, I don’t, anyway.

Prospective buyers, early in the buying process, appear to repeat searches. If these users walk up to a cold machine, boot and start searching, they see nothing different caused by the search history. However, early phase searchers are not, usually, close to buying – by definition. You can, of course, help accelerate them to purchase with the right advert copy and the right content on your site. But the general behaviour *appears* to be that multiple repeat searches are mostly coming from people who are unlikely to buy now, today.

Early phase searchers may not even be looking at paid search adverts seriously – I believe that I tend to see lower CTR’s for keywords where I regard the intent as being weak; early phase searchers often seem to use organic results rather than paid search, because organic results will often lead to pages with discussion of alternatives, while paid search typically leads to a specific solution. In other words: consistently targeted adverts are shown to the group who probably could benefit from some message variation (use those alternate creatives, marketeers!). These early phase searchers are also, typically, the lowest converting group – because they may get distracted by a different type of solution or decide to not continue with the purchase process, or the latency exceeds the 30 day AdWords Conversion Tracking or the default Google Analytics 6 month tracking or ritual cookie deletion.

In the later phases, I believe that people vary the search more. This is where they are comparing features, benefits, finding discussions and threads to justify their decision. These later phase prospects are the ones that see randomised adverts, because the search history is invoking bizarre extraneous keywords to the party.

The consequence of the evolution of search queries is that the most bizarre conjunctions of irrelevant adverts are shown to the users with the highest interest in buying right now.

This is not desirable for advertisers. This is not helpful for search users. Why do it?

Permutation

The nature of the Search History interaction appears to be a simple combinatorial permutation. That is, given “cheap holiday” and “us vacation”, it probably generates “cheap vacation”, “cheap us”, “us holiday” and “holiday vacation”. While this example generates some plausibly interesting candidate keywords for searchers, it is demonstrably weakening the answers to the most recent request. See the two examples in the previous article for details.

The core question is whether permutations of the current and previous search query generate plausible searches that the user might have made. If this was a valid technique, might we not expect it to have been used for organic search, first?

That is to say, if Google, who are probably about the largest, most rational and experimentationist entrepreneurial organisation on the planet have *NOT* used permutation to improve organic search results, why would anyone imagine that it benefits paid search to do so? I’m not aware of any previous activity in paid search that has resulted in organic search following the behaviour. I can see that Google copies lessons learned from organic search into paid search. You want some examples?

• 404 checking – 404′s are lethal for organic search, so Google verifies that paid search pages are present.
• Landing Page Quality Scores – appear to be based on what makes a good result for organic search (though I have a nasty suspicion that the nature of search evolution and the buying process implies that there *should* be a difference between organic and paid search optimal landing pages).

This paid search usage of the search history appears to be novel, and appears to offer a reduction in the overall effectiveness of searches. Combinatorial explosion also means that with lengthier queries, the number of combinations is increased markedly. Some of these may well overlap with keywords that have a higher bid and/or a better CTR – so these will be favoured over other adverts that are more directly relevant to the searchers intent.

Markets and Market Prices

OK, now we’re in the dismal world of Economics.

If I’m an advert trader, I can set up multiple markets for bidding for placement. I might set up one market for “ford prefect” and another market for “edsel”. This is like Google – there used to be a market for each keyword. Google’s Broad Match extended the search queries that were involved in the market, but usually in identifiably sane ways. So a market for the keyword “cheap holiday” could include search queries for related concepts such as “cheap vacation”, “free holiday”, places known for or associated with inexpensive vacations such as “cancun vacation” or “holiday cyprus”, and even specific company names closely associated with cheap holidays.

The more of these other search queries that I can recruit to my market, the more price competition I can engender. This is why, I think, Google likes paid search users to use Broad Match by default – you participate in a larger market and that makes obtaining a higher value price in the auction. More bidders implies a higher price from the market – I can’t currently think of a counterexample in which more participants recruited to an auction will reduce the price struck. The generalised second auction apparently used by Google, doesn’t seem to offer a way to allow larger volumes of bidders to reduce the price that is struck, compared to an auction with fewer participants.

Indeed, this observation is the entire basis of an industry – keyword finders for SEO and paid search. Find the rare keywords in which the auction includes fewest bidders, and, the argument for this industry goes, you have found the auction in which you can pay the least to obtain traffic (or use SEO to get that “free traffic”). Of course, Broad Match negates this industry for paid search to a great extent. I have (seriously) considered setting up AdWords for well established brands, in which I simply used the brand name, with carefully selected negative keywords, to do all the work. Google will gleefully match well established brands with the primary and even secondary characteristics for their main search queries. Why do work you don’t need to, eh?

So, given Google’s ability to extend an inference for, for example, the keyword “thomas” to imply “cheap holiday”, or the search query “thompson cooke” to bring visitors to a major travel site (check your web server log files and observe the mismatch between keyword and search queries – that’s where the evidence is), why would Google need to use permutation to increase the auction?

As another example, look at what happens in organic search. You miskey “erlers danlos syndrome” and it offers to correct the spelling – because the history of interactions and the nature of searches have taught Google what the normalised spelling should be. Users have become used to the idea that Google will correctly guess the question that was intended. Why would paid search uniquely require a type of search query expansion that is *not* used for organic search conceptual extensions?

I believe that Google uses search history permutation to create an artificial market. As well as related search queries, the search history drags in unrelated queries. So a chain of “Brazilian Tax Credit” and “US Vacation” searches yields combinations including “us credit”, “brazilian vacation” and so on – and you can see adverts in the primary article with these exact adverts shown. *BUT* it is a false market. These advertisers had no strong intention to appear on those searches. If they did, they’d have used those keywords – just as an organic search would have extended the searches to include those results. If Google had seen a relationship, then Broad Match should bring in those queries to the advertiser.

Is this fraud? I’m no legal expert – I have no real idea if this is technically a fraud. But doing things to paid search that you wouldn’t do for organic search at least raises the question of whether this is deceptive advertising for Google – if advertisers were operating in the reasonable expectation that search for keywords worked like search for search queries, then might Google have implicitly broken the contractual expectation for advertisers? If you create a market that recruits bidders who don’t really want to participate, so you can increase the paid price in the auction, is that fraudulent?

Local Optimisation vs Global Optimisation

I suspect that the essence of the problem is an attempt to locally optimise. For example, if you are a programmer, you can expend effort to refactor an algorithm for a local optimum, and later discover that the global performance has been damaged.

I suspect that this is what Google have done. They’ve apparently decided to stop showing adverts on searches where few people clicked on adverts. They’ve compensated for this by using a piece of information that they hold – your last search – and created a higher value market for the adverts. This increases revenue in that transaction. But I think that this creates a problem for global optimisation, and Google’s brand value – it also does very little for advertisers. Users seeing irrelevant adverts will also often blame the advertiser for stupidity, as much as they blame Google for poor quality matching.

Optimising my clients’ accounts won’t help much, though. The problem is that I can’t optimise my clients. Google has created a strange new market, apparently to boost the Average CPC at the point at which people are most likely to have abruptly shifted search focus, and are generating the most random results at the point when someone is about to buy. I can’t control Google’s recruitment of irrelevant advertisers. Even if my adverts are exactly on focus for the intent, when pushed down the list of results, I get fewer impressions and a lower CTR – because people don’t want to go wading through random garbage results. I might even get pushed off the first page. It’s simply frustrating, because it is beyond my control.

The adverts that I most need to place, are the ones that Google has apparently decided are a reasonable place to extract additional revenue from advertisers.

Monetisation

Historically, Google has succeeded by consistently producing good search results. They resisted introducing paid search until they had a model that worked – no inline search results, and an apparently rigid wall between paid and organic search results. I am in awe at the subtlety that AdWords used to have. It was a fantastic tool that I used for market research, as well as lead generation or sales.

The virtuous cycle for Google has been that good search results meant more user recommendation and so a growth of the user base. Advertisers like being able to reach a large audience.

Note how this depends on the quality of the search results page.

Evans and Wurster’s 1999 book, “Blown To Bits” (Harvard Business Press – no commission on this link, folks) was the first book that clearly explained, to me, the theoretical underpinnings of the value of Yahoo, and tied it to marketing principles that made sense for me. This book is also a great guide to monetisation and market share. In other words, it explains the economics and marketing principles behind what seems to have been a core Google strategy for the last decade:

By delivering the best page of search results, Google stays ahead of the competition, and thereby dominates search.

As we’ve seen, permutation damages the likelihood, for later phases of the buying process, that Google will give the *best* page of search results. It may be a better page than competitors, but it is not the absolute best page that Google could deliver. Arguably, when looking for a commercial solution, paid search is a better answer than organic. It depends on the market, and the state of organic results. I know that I have, and have had, over the years, clients with huge CTR’s and high conversion rates, because the organic results are wrong for final phase product oriented searches.

Does this damage the Google brand in the eyes of the search user? I think it does, and I have some clues that point to this.

Paid search vendors are intermittently criticised by industry watchdogs for failing to adequately clarify that paid search results are not chosen on the same basis as organic search results. In conversation with users, they will often hold that the number one advert shown by Google has been selected with the same or similar criterion as the number one organic search results. There is certainly a higher intrinsic CTR for the number one advert position, especially if it appears above the organic search results.

It is intriguing, isn’t it, that industry commentators and so many paid search booklets of “affiliate secrets”, emphasise that appearing as the Number One paid search result may actually dampen overall ROI. I believe that this is because it recruits searchers who are in the wrong phases. Instead of being at the point to consider purchase, these searchers are still discovering what it is they need to know, before they go shopping. They also trust Google to deliver relevant results.

What will happen to users as they approach the final phases of an attempted purchase? They will see a largely static list of advertisers for each repeated query turn into a set of adverts in which perhaps half or more of the adverts have no direct relationship to the search intent. I believe from purely personal observation and extensions of marketing psychology, that users will extend less trust to paid search results, especially when the top poistion from advertising is clearly off-topic – as it is, accidentally, in both of the test cases that I previously published.

Far from demonstrating Google’s long term commitment to improving search results, permutating the recent search history appears to weaken page relevance, and this drives down the long term monetisation of search – because it means that other vendors offer a relatively better search results page. This is really good news for Yahoo! and MSN. Their adverts will actually have a higher trust placed on them, so long as the auction is kept reasonably fair and uses only the contextual extensions that organic search would use to extend the keyword to similar *weighted* searches. If I was a user, looking for things to buy, and in final phase search, I’d change my strategy to use MSN or Yahoo! – because I’d get more consistently focused adverts, that help me achieve my ends.

Advertiser Effects

Because Search History Permutation enlists more advertisers, the effect is quite interesting. Painful. But definitely interesting. It seems to differ according to your normal bidding strategy.

If you typically bid below position 5, then you are likely to be pushed further down the page, or even off the first page. Impression rates will decline, sometime markedly. CTR will be typically somewhat decreased – but no huge decrease in CTR, I think. You should see that your average position is declining, as irrelevant adverts are brought in above you.

If you typically bid above position 5, you may be recruited to show adverts on irrelevant searches – your adverts have a high CPM, making them valuable for Google to show elsewhere. The result is booming impression volumes, as your adverts are shown on searches you don’t care about – and your CTR crashes down. I think that your average position will be largely unaffected.

Any way you slice and dice it, this looks like a poor move for advertisers.

Summary

Searchers evolve their search queries during the buying process. From my research, the greatest variation in search queries appears to happen as searchers near the actual purchase. The final search appears to consist of a single query – for the business that was identified as the right one to buy from. This is statistical inference – so for some users there will be a single search, and for other users there many be multiple repeat searches that lead to a sale; the argument is not invalidated by a single counterexample – you need a chunk of data to reveal that this is wrong.

Google’s current use of search history recruits new keywords to participate in the auction, resulting in the appearance of adverts for which the searcher has demonstrated no interest.

This appears to be something that is not in the best interests of searchers.

This appears to be something that is not in the best interests of advertisers.

This appears to be something that is intended to increase revenues to Google, by manipulating the conditions in which bids are evaluated.

The exposure to additional participants in the auction appears to allow a higher price to be returned from the auction.

This appears to damage the long term brand value of Google, at least for advertisers and probably for searchers.

It appears to offer comfort to Yahoo and MSN search – their page values are now relatively higher than they were before Google made this change.

If this isn’t evil, what is it?

If this isn’t fraudulent activity, what is it?

This appears to be yet another example of covert messages from Google that indicate how they really think. Ignore the babble about how much they love you. Google apparently thinks of advertisers as gullible stooges who are there for the money they yield. Advertisers, for Google, it seems, are a resource that demands no respect. They won’t tell you this, but it is how they treat you – sending you clicks that won’t convert is great for Google and lethal for your business. Removing Advertisers ability to control click quality is great for Google and destructive for the relationship between Google and Advertiser.

Trust is earned, but it is fragile. Google keep destroying the trust that their organic search results have built, when dealing with advertisers.

Update

2008-03-18 Slightly clarified criticism of Google, bringing it back to AdWords, in the summary. I vacillate as to how much damage this does to the relationship between Google and Searcher, and between Advertiser and Searcher. Seeing irrelevant AdWords adverts is often, but not always, blamed on stupid advertisers.

I use Akismet for this blog, and it traps most of the spam, most of the time. Occassionally it traps a real comment but the most frequent problem it presents is a comment that is unique, but uninformative. Was the comment made just to gain rank for a less known blog? One of the clues that I use is the age of the posting. By and large, the older the posting, the more meaningful the comment has to be. So for a brand new post, I might, barely, let a “great article, you are always interesting” through. When the article is six months old, you need a substantive comment with some real content or a proper question.

Another clue that I use, but that is annoyingly hard to extract from FeedBurner and Akismet, is what brought that commenter to the site. I’ve just found a comment that fell into my “it’s an old article” and “uninformative comment” category – so I wanted to see if it was just linkspam. Looking at the webserver logfiles shows that the search this user made was ‘[SEO] “powered by wordpress” “leave a comment” “no comment”‘. So, it’s spam. Looking at the site was another clue.

Why do I care?

Well, neighbourhood, and signal to noise ratios. I really don’t like linking to Get Rich Quick sites – mostly splogs. Most of them are, IMO, a way for the owner to get rich and not for the visitors to get rich. I hope that I’m a responsible marketeer – so I want to lose the scammers and make them work harder for their money.

The other issue is signal to noise ratios. Blogs that repeat the same tired information just annoy me. Don’t they annoy you?

I was looking for a clients’ problem and the search showed the same original article information popping up in blog after blog. Very few of the repeats gave attribution, and none of those that I read expanded on the information in the original blog. This isn’t such a good example, but the original gave away too much information about my clients’ problem… but check out a search for “akismet-admin workaround“. See how the same info is repeated, sometimes without attribution, but exactly the same edit is proposed? Notice how Google suppresses duplicate information – it is indexed and ranked, but not shown unless you click the “Repeat the search with the omitted results included”.

I don’t really know how to solve the problem of other sites duplicating unattributed articles, at *this* end. That is, as a single blog, denying links to sites that exist mostly for self-promotion, any action is pretty ineffective. However, Google could apply harder filters. Even in the weak example search that I cite, blog entries like this image, are plentiful:

This is pretty typical of sites where I wouldn’t want my client adverts shown, and that offer little value to users (because the information is already available, better presented, part of an active community). Note just how irrelevant that advert is to the search and the article content. It’s still an impression, though. If you clicked on something that irrelevant, imagine how interested you are likely to be in the product… I suggest (and conversion rates support this) that you have no immediate interest – nothing that fits in a 30 day cookie window, at least.

The problem for Google is that sites like these generate revenue. While Google has been strict with preventing AdWords advertisers from affecting search results – there really does seem to be a strong wall that prevents AdWords from affecting organic results – Google probably should recognise that part of the reason for duplicating information is to gain rank, so that visitors will be exposed to Google’s own advertising system. AdWords isn’t directly affecting search results – but the publishing system is.

Conclusions

Personally, I’m going to continue rejecting lightweight comments that lead to splogs or splog-like sites. This is more work than I’d like, but until de-spamming tools can recognise and mark splogs in links, I’ll do it by hand. This slightly increases the value of this blog to real readers.

Since the point of splogs is to take popular (or possibly popular) articles and republish them, in the hopes that the search engines will reward them, it looks as though the main cause for their rise is search engine ranking systems. And that’ll be the reason for them to disappear again. So the main activity for splog reduction, which will in turn have an effect in reducing comments on real blogs that are merely part of a SEO strategy for sploggers, will have to come from the SE’s.

I will continue to eject splogs from my clients’ Content Networks, by using site exclusion. Conversion from splogs is the lowest that I have measured. (Yes, I have now seen a conversion from a domain park).

Google could reduce the frequency of duplicate content entries in listing by a filter that requires a proper reference to a site with similar content. This would make the “lowest possible effort” Get Rich Quick site builders to have to do some work. Since the original author should be the earliest article, this should end up that *mostly* the reward for duplicating articles will be removed. Will this be hard to do? Yes. But that’s what happens when you rely on a citation model for pagerank – Google have made the bed…

Google should demote the rank of blogs that promote AdSense content *visually* before the article. I’ve yet to come across a blog that is valuable to users, that devotes the top 600 pixels to advertising. If you can find me some counter-examples, I’d be delighted to see them! :) This will increase CTR in the Content Network. Evidence – none. But I do have experience of working with large clients and building websites; when people linger on a site rather than bouncing in a few seconds, there’s a higher chance that they’ll be engaged and will click on something else important to them.

I’d like to say that one of the clues to a good blog is whether the articles have substantive comments (not just pingbacks) – but if scammers knew this was important, then it’d be enough to generate loads of comments (automata for this would be *easy* to write and hard to detect). And “substantive” is terrifically difficult for software to detect.

Controlling the quality of content match sites will have a beneficial effect for Google, eventually. Right now, they are a junkie. They need a fix of advertising and Wall Street expects continual growth. However, adding new publishers who simply duplicate content, often without attribution, simply makes crawling the web harder and fills organic search results with useless listings. If Google is strict about reducing spammy publishers, then search results in Google will improve, and revenues will at least stay in the same range… and it is likely that overall CTR in the Content Network will rise as users spend more time on useful sites.

Perhaps Akismet could be improved for comment spam decision making if it included the first-seen referrer_info for the commenter. This would make it easier to work out why a commenter is submitting comments – I’ve no problem with marketeers promoting their sites here, if they make a substantive comment, based on solving real problems. I just don’t like degenerate freeloaders. Call me biased.

Update

2008-02-08: An older article has an interesting and recent comment by John Nagle, about a system to reduce exposure to spammy sites. Personally, I don’t think it will work for most advertisers. IME, the sheer quantity of low volume sites means that such a system will expend a lot of effort to deny sites that wouldn’t re-appear anyway. Put it like this… In a few tens of minutes I can sign up to new free domain host, and be publishing a spammy site, with an aged domain. However, getting traffic from each advertiser, sufficient to trigger a rejection, is a slow process – the site will make money. I believe that while making new, ranked, spam sites is faster than detecting and removing them (for all advertisers – not just the one) the problem will continue to drag down all content advertising.