An Analogy

Recently, restaurant revenues have been generally in decline. Is this because people are finding restaurants misdirect them to food?

Or is it because of another, perhaps slightly more obvious, factor?

What about people optimising their budgets?

Is there any reason to think that businesses do not also look to their operations in a recessionary environment? Would the shareholders be out in lynching gangs if a business was to ignore a recession?

Why Is Advertising Really Decreasing?

It’s a recession.

That’s it. When the marketing budget goes down, what do you do? You focus money on the most effective items.

Paid Search typically is icing on the cake. Not always – it depends on the business and I can cite companies where this isn’t true, too – but we’re looking at industry wide factors. IME, Paid Search is most often used to add a chunk of business for an established site. They could drop the advertising, and still see leads and purchases. So when they are keen to save money, it is an easy line item to drop.

In a recession, users spend more time looking for bargains, and click more. Conversion rates typically decrease, and ROI decreases. What is the right response? Decrease spend, and improve the site.

Instead of spending more on advertising, companies will direct effort to Search Engine Optimisation. SEO is an investment. Over months, you keep working on improving relevance, increasing user satisfaction, helping users to link to your business, developing low cost business relationships, and perhaps using grey and blackhat techniques – depending on the type of business you run and the risk you will take.

You also invest in site improvements – for a large company, that is cheaper than paid search. By optimising the site you improve user satisfaction, you can improve search engine rank, and you can decrease the need for paid search.

Strategic Impact

Two deceptively simple-seeming numbers tell a strategic truth.

In January 2008, about 90% of our leads for new business were primarily people wanting AdWords management.

In January 2009, about 80% of our leads for new business are primarily people wanting SEO and Conversion Improvement assistance.

Most of our AdWords clients have reduced spend. A few have increased their spend. But nearly everybody looking for extra business is asking us to find more economical ways to win business. That means looking at other channels, such as FaceBook and Twitter. It means exploring the use of blogs, and discussion forums. It means asking for feedback on the site content. It means looking at what users do and don’t do, and working out how to provide them with the best answer. It means improving site navigation and usability. It means offering up to date content.

Misdirection and Satisficing

The *worst* part of the article was a complete misunderstanding of how search engines, especially Google, work.

Google makes money because it doesn’t focus on making money. It focuses on satisfying users. Because Google has a large number of users, it is more appealing to advertisers. If Google presented search engine results that *didn’t* satisfy users, then users would seek other search engines that offered a better result. Advertisers would seek other opportunities. That has a direct impact on revenue. How?

The more advertisers there are in an auction, the more Google makes in that auction. If just one company bids, and users like the advert in large quantities, the advertiser can pay $0.01 per click. If there are two advertisers and users think both of the adverts are pretty good, then the average cost per click is higher – how much higher depends on the bidding strategy of the two companies.

However, if only one company bids on a keyword, and search users hate the advert – either the CTR is low or the advert is misleading – then Google will disable the advert. With a sufficiently unpopular advert you can bid $100 and your advert won’t be shown, even if no other advertiser is advertising on that keyword.

This is called “satisficing“. Google has created an auction mechanism that allows businesses to compete with each other. It has also created a system that allows users to vote on the acceptability of adverts. Unacceptable adverts – things that users hate – will eventually disappear even if the business is prepared to pay outrageous amounts.

There is no misdirection in that system. None of the parties may be completely happy, but none are unwilling to be a part of the game. That’s satisficing in a nutshell.

Misdirection Can’t Satisfice

So a central plank of the argument, that advertisers misdirect, simply isn’t borne up. If you have any real experience of paid search, you’ll know that you can decrease your Average CPC on Google by delivering improved adverts, better landing pages, easier purchase paths, links to supporting information, better customer service – *NOT* just by bidding more. Bidding more, in general, reduces ROI. Bidding management as a sole strategy is a failed idea. It was true in 2002. It is obsolete as a strategy now, and has been obsoleted for years.

Ignore The Rest

If someone pontificates at such length about something they don’t understand, I can’t read the rest with any sense of belief.

I *do* know something about PPC. I’m an AdWords Help Forum Top Contributor – I’ve read *thousands* of advertisers questions about why their adverts don’t work. I have free tutorials on how to improve adverts – which people say work for them; all the tutorials focus on increasing search user satisfaction. I’ve spent, on behalf of paying clients, budgets of up to $500,000 per calendar month – each advert leading to a specific landing page that answers the search query, with up to 4000 unique adverts flying at the same time, to bring users to the highest satisfaction landing page.

I’ve advertised, on behalf of clients, on competitor names and seen low CTR and low conversion rates and reducing Quality Score. I *know* that Google pays attention in organic and paid search results, to user satisfaction.

I’ve seen too many examples of clients in distress, asking for help, where they haven’t understood that they can’t just pay to appear.

Summary

Paid Search is declining because you get one sale from about 1% of clicks; it costs for each of the hundred clicks (or whatever the rate is – we see between 1:5 and 1:500 for various clients). In a recession, you can save costs by reducing items that are per-sale. SEO is not a per sale factor, it is closer to a capital investment; you can weather an economic crisis by living off the organic search results or investing further during the downturn so that as the economy recovers, you are better positioned for increasing user interest – but you aren’t paying more while users shop around.

Almost every single client that we have is working on improving their site, and improving their SEO results as a consequence. A year ago, they’d have spent money to bring in leads, and spent less time improving the site. That’s why PPC spend is down. Not because users are misdirected, but because advertisers are actually focusing on improving performance in a downturn. It is often cheaper to address user needs than to advertise and an improved site is improved for some time – whereas a click is transient. Pay Per Click appeals less than Conversion Improvement and SEO. That’s it. That’s the story.

Pretty flaming obvious, IMO. Oh, gee, I wrote about 2009 strategic advertising responses months ago. In a prediction. Hoop de doo.

Update

Danny Sullivan versus Eric Clemons in TechCrunch.

I occasionally see PPC experts stating that QS isn’t as important as some other factor that they are promoting. This video makes it clear that QS has to be a central component of managing Return On Investment, and can even significantly affect the costs of an Awareness program – in which ROI is less important. The importance of the account’s accumulated Quality Score and long term response by users – longer term than CTR – can have huge impacts on paid price, which in turn benefits ROI. QS isn’t *only* CTR or *only* Landing Page factors.

Although this doesn’t explicitly describe the Content Network – AdSense placements – I’m pretty sure that there are differences. SmartPricing is not covered. In my experience whatever it is that SmartPricing does, can be counterintuitive to advertisers, sometimes with much more paid per click to high volume non-converting sources than to low volume repeat converting sources – I’d rather see a higher CPC to a site that sends my clients repeat converting traffic. Following Hal Varian’s suggestion at the end of the video, I’ll be asking for a clearer explanation of Content Network Pricing, especially with the new behavioural targeting system that has entered Beta.

Measurement

If you don’t have website statistics now, ask your website management company, or your staff, to install the free “Google Analytics” package. There are many free packages and you may already have one, but a big advantage of Google Analytics is that it integrates well with Google’s advertising systems and reasonably well with many other advertising systems.

Make sure to ask that “error pages” – the “404 page” and any “500 pages” – also have the Analytics code on them. You should investigate and repair your website urgently, or your advertising, if large numbers of visitors reach these dead and useless pages.

A big weakness of Google Analytics is that it doesn’t do quite the right things, until it has been configured. Configuring it can be difficult for beginners – so here’s a good book with detailed configuration guidance, for technically oriented staff – Brian Clifton’s “Advanced Web Metrics with Google Analytics“. If you are non-technical, I suggest that you flip through the book, note the types of information you can get with postits on each interesting page, then ask your technical staff to implement the techniques on the marked pages.

As a minimum, you should ask for Google Analytics “profiles” that give you “Error Pages”, “New Visitors” and “Returning Visitors”, with “filters” that eliminate traffic from your own staff, and ignore traffic on any sites that have stolen your web pages without changing the Analytics code.

If you expect to use AdWords or other paid sources of traffic, you should have each of those paid sources of visitors broken into at least two separate profiles for each – for example “AdWords Visitors – New” and “AdWords Visitors – Returning”.

You can also ask your staff for reports to be automatically emailed to you weekly, so you don’t have to login and work out how to get what you want. There’s also a free iGoogle Analytics “plugin” so you can have stats on your Google Home Page – if you use that.

Understanding Visitors and Your Site

You should be able to answer the following questions. If not, you or your staff probably need to develop an understanding of how your website and your analytics works.

• How many people visit your site? Per day, per week and per month? And when, during the day, do they visit?
• How do new users find your site – are they coming from directories or from search engines or other sites?
• What search queries are visitors asking the search engines?
• Are visitors bouncing – leaving the site as soon as they arrive – in large numbers?
• How long do visitors stay on the site?
• What are the most and least popular pages with users – the ones most and least visited and the ones that users stay on longest and shortest? Is it what you expected?

You should be getting most new visitors from search engines – is that happening?

You probably should be getting return visits from customers – are you getting returning visitors?

If the pages for your best products and services aren’t getting visitor attention – why not?

See if you can improve your performance, by making changes and watching the statistics.

Customer Engagement

Think about the searches that prospective customers should be using to find your business. When you talk to them directly, what do they ask you? Not in your professional jargon, but with their use of language?

Now go to the “AdWords keyword ideas” tool. Put in your user’s important phrases and ask for suggestions. These are usually words and phrases that your site should be using and may even be used as the names or titles of web pages on your site. If you use language differently from your likely clients, you are less likely to attract them to your site and keep them engaged with your offers.

When you have a conversation with a client, lead or prospect, think about whether your website clearly describes the same things in the same way. If your website message is different from what you tell people, try to make the web site reflect what you are doing and saying.

If someone lands on the web page for a product or service, without going through the home page, would they know what you are offering and how to get hold of you? If you have a Unique Selling Point on the website – is it only shown on the home page? People who use search typically jump deep into your website, and will miss messages that appear only on the home page. So if you offer free shipping, or a free first order, or whatever it is – make sure that offer is visible on all likely landing pages.

Make sure that you have a call to action on each solution page, with prominent contact information. We’ve come across a surprising number of sites without phone numbers, inquiry forms or addresses… At the point when visitors should have decided they need to talk with you or place an order – make sure that they can do so. A rough rule is that “users will do what you ask them to” – so ask them to buy.

Before you make changes to the web site, here’s one more review step… does your website tell customers about you, or do you talk about them and their problems? Your website should demonstrate that your first priority is solving your customer’s problems. A swirling-in logo of your company, or the letters of the company name coalescing from a cloud, may look pretty neat, but does it answer a prospects’ first question – “do these guys have anything that solves my problem?” Focus on what your prospects need to know to become clients, and don’t delude yourself that they want to sit through the same 40 second monologue from you at the start of each visit, going on about how great your business is… They won’t like it. Really.

Get to the point quickly, and describe the problems that your customers used to have, and what you do to fix them. If you solve several problems, offer one web page for each, and use the home page to reference the different solutions. Each solution page should link to related pages, so if visitors aren’t in the right place, they can easily find where they should be.

Being Found – Search Engines and Other Visitor Sources

There are two main ways to appear in search engines. For the main search results, you use “Search Engine Optimisation”. Do read free advice from major search engines about selecting a SEO business before you pay for services.

• Google’s SEO advice
• Yahoo SME – SEO advice

Paid search advertising is the alternative. Smaller businesses shouldn’t just pick the market leader, Google AdWords, but consider the smaller players, MSN AdCenter and Yahoo!Search Marketing. There’s usually less competition, and the smaller audience doesn’t make much difference to the volume of visitors when the budget is small. They also attract slightly different audiences – so failure on one of them doesn’t mean they’ll all fail.

The AdWords Help Experts blog has a video showing how to set up advertising on Google AdWords. Do *NOT* use Google’s “Starter Edition” – it is poorly designed for new advertisers. Exercise great care – Google does not help you to control costs and improve ROI – their eye is definitely on their revenue, not yours. Do not expect to carry to AdWords, the same trust that you have in Google’s search engine, or you will lose money.

SEO is slower to return results, but often at a lower cost. If your website is pure Flash, gaining visitors through SEO will be harder and more expensive. Paid search advertising is faster, and it is easier to identify the return on your investment. We recommend that if you can, use Paid Search first, to quickly find which keywords & pages work, and then use SEO to target them.

You can also appear in business listings, such as Google Maps. If you sell products, you can list them, for free, with Google Product Search. You may have images and pictures that interest people, and you can get listed for those. You may make videos and publicise those for Video search. There are planty of ways for a little imagination and creativity to get your your business noticed.

• Qype – expanding worldwide, offers reviews of local resources and interaction with local and interested members of the community. See the Merjis listing showing that we offer internet marketing in Bedford. Accepts adverts, too.
• Gumtree – classified listing directory, accepts adverts
• Craigslist – massively popular classifieds
• eBay Stores – if you have products to sell, eBay has buyers.
• Yahoo Directory – paid listing, human moderated, the original large scale internet directory
• Open Directory Project – free listing, human moderated and a contributor to the Google Directory.
• LinkedIn – link to professional colleagues – see my profile for example
• Twitter – status feeds for customers, personal conversations, all sorts, Twitter is flexible
• FaceBook
• MySpace

Do *NOT* pay for “Search Engine Submission” tools – if your site is properly linked to the rest of the web using the services and sites that I’ve given you, the major search engines will find you anyway. Do *NOT* list your site on free and unmoderated directory.

Internal Response

Leads received from the website are often left in email inboxes for a few days. Web leads are as urgent as a phone call. Make sure you and your staff respond promptly.

Finally, you probably have news about your business sector. Can you write it up, even if in only a few short sentences? A business blog with short, or even long articles like this one, can provide a way to communicate with new leads to show them how you help your clients, and it helps your SEO efforts. There’s lots of free blogging software that you could host on your own site, or you can use one of the public blogging services.

About Us

The author, Jeremy Chatfield, is a Google-recognised Top Contributor in the AdWords Help Forum and manages the AdWords Help Experts blog; he is a Google AdWords Qualified Individual. Merjis, in Bedford i-Lab (+44 1234 834760), helps clients to improve their web site performance.

Material Disclosures

The link to the Google Analytics book above is an “affiliate” link. If you click and buy, we may make a small amount of money. It’s a demonstration of another common technique for attracting visitors (using “affiliate programs”) and may be a way for you to additionally monetise your site.

None of the other links are affiliate links, even when an affiliate program exists for that organisation.

This article is an expanded version of an advertorial in our local business newspapers, “Business to Business” and “Business MK“.

• Google will struggle to retain revenues using a variety of techniques
• Searchers will spend more time browsing and convert after more clicks
• Online revenues will generally increase – but business margins will be squeezed
• Internet Theft Scandals – Click Fraud, Phishing and Account Theft

Details and the consequences? Read on…

Google Will Adjust To Preserve Revenues

The basic idea of paid search is simple. Searchers submit search queries, and advertisers pay to have their adverts shown to more or less interested searchers. This meshes with the purchasing process (buying model) in several ways. At the most basic, when someone searches for your unique trademarked business or product name, it is likely that this searcher is intending to buy something. More likely than if they’d typed a competitors name. When they search for something that describes the product category, they are less likely to buy – they are in an earlier stage of the buying process.

Auctions have some basic characteristics. As a broad generalisation, the more bidders there are in an auction, the higher the revenue for the sellers. So Google’s goals have to include increases in the number of bidders in an auction – Broad Match and adding ever more advertisers all help with this. Convincing advertisers to bid high – traffic scales heavily with position; Google needs to convince advertisers (or, rather, prevent them from becoming aware of the relationship) that position and click quality are not related, but that position and volume are related. Finally, Google needs more outlets, so that advertisers see more reasons (more impressions) to be part of the ecosystem; however, that largely means adding new, smaller volume, publishers – some will have niche specialist interests, but overall this encourages more Made For AdSense sites, which, IME, have an appallingly poor conversion rate and value (reducing click quality by adding more outlets).

You’d expect that in retaliation, advertisers would then want to stick to Exact Match and using only Google’s Search Pages. There’s other factors at play, though.

Users miskey – they can’t remember the name of the company properly. So someone looking for the travel company Thomas Cook might key “thomson cooke”, “tomas cuik’ and all sorts of other variations. This means that Exact Match isn’t enough to identify all people looking for the trademark or brand name. Broad match helps by allowing Google to send these near misses to the right advertiser. Google do a phenomenal job of matching; if an advertiser like Travelocity doesn’t use “cheap holidays” as a search term, then Google will match a high bidding Broad Match keyword to that search.

People will also type more specific searches, longer searches; most searches, by a long margin, have more than two words in the search query. Users will type “logitech mac support” or similar, to more rapidly jump to stuff they know exists somewhere in the vendors site. Some of these longer searches won’t lead to sales – as in this example, some searches are support inquiries. So Exact Match also fails as being too specific, because these search queries wouldn’t be matched. Phrase Match is useful to find unexpected variations and more specific searches – leading to the idea that you can help users to jump deeper into the site with the right combination of technology and advert. Again, typos and miscomprehension by users will mean that Broad Match captures additional searchers who intended to find the business, but would have failed with solely Exact or simple Phrase Matches in the campaign.

Google’s problem is that when someone has decided on “Honda”, or “iPhone”, the searcher is unlikely to be deflected by alternatives. The consequence is that there will be few competitors for a brand name. Brand names are usually the best return on investment – so the average costs per click will tend to decrease for brand names. This is part of what you’d expect from a basic analysis of the buying process – you generally type a specific company name after you have researched and typically just before buying. So there is an inbuilt pressure to decrease advertising on exact matched competitors name – resulting in lower revenues from trademark terms. That’s a problem for Google.

The earlier phase searches, which tend to be naturally higher in volume as people look for alternatives from which to choose, naturally imply a worse ROI – you need to pay for more clicks to support the research, and the conversion rate is lower because searchers are researching. This too tends to drive Average Cost Per Click down, in order to retain a positive return on investment.

Google needs to keep advertisers focused on Broad Match. This lets Google place unsold inventory, offer competitor adverts on trademark searches, and so on. The goal for Google is to increase the number of participants in the auction (helping increase average cost per click and hence inreased revenues), to find ways to increase advertising on unsold inventory and to affect the minimum bid.

Some techniques that are likely to be used by Google?

Expect to hear more about the difficulty of reaching interested searchers with organic search, now that personal search is becoming more deeply embedded. If advertisers think that they are missing audience because of personal search, that they can only reach with paid search, then there will be more advertisers and increased competition. This will be partially driven because of the effects of recession… in some market segments there will be reduced interest, and this will naturally translate as reduced search volumes, resulting in fears that personal search is sapping share of voice.

Expect to hear more about Quality Score changes, probably surrounding the calculation of the minimum cost to appear on a page. This is not the same as, but is confusingly similar in name to, the estimated cost to appear on the first page. The intention will probably be that if the CTR is lower than some target value derived from the rest of the network, that the minimum price is adjusted to meet some revenue goal – it won’t be written like that – I can’t guess how Google will describe the technique, other than that is likely to involve some description about “improving user search experience”. There will still have to be a way to allow $0.01/click – a feature that attracts many advertisers, few of whom have any real chance of achieving this… but it remains an important differentiator over Yahoo and the other competitors in this space. I think it is possible to offer both, because you only get $0.01 on high volume, high CTR keywords – IOW, not things that affect most advertisers.

Expect more dilution of “real search”. Google still wants additional outlets and will continue to re-present domain parks, “fake search”, selected content match outlets and other “opportunities” as if they were what naive advertisers expect from keyword search (that is, that the advert is shown directly in response to a real user search that leads to a directly relevant site – much as happens with organic search results). Google will want the increased impression volume and the *overlap* of keywords with different intent that allows a single outlet to have larger counts of competing advertisers.

Expect more messaging about the synergy of paid and organic search – that even when you have achieved page dominance for the targeted keyword, you should still be advertising as well topping organic rankings. This increases competition, ensures that there are at least ten plausible advertisers, etc. I expect the messaging around this to increase as advertising volume and value shrink.

Expect more overseas advertisers. Google doesn’t offer USD bidding to smaller overseas bidders. The exchange rate isn’t notified to overseas bidders. So, especially when currency exchange rates are in flux, Google can make margin on exchange rates. I expect that Google can easily and all-but-undetectably collect additional revenue on overseas bidders, by tweaking exchange rates – AFAICS, the auction is held in USD, so exchange rate changes are needed for anyone not using USD. This would be non-US income – no-one in the USA will blink at using exchange rates to improve revenues from non-US companies.

Radically – if Google were to remove Exact Match, they could make dramatic transformations of revenue expectation. Would they do this? I think they would, if they could claim that it improved the search users experience. If advertisers fail to appear on a substantial fraction of searches, especially when those clicks turn up in organic search clicks, it would allow Google to say that exact match was preventing searchers from seeing the results that they want to see. This would be a huge step for Google, and they’d probably need a lot of research. It’ll be piloted by some large accounts and evidence will probably be drawn from wide-category vendors, like eBay. It’ll turn out to be complete rubbish for narrowly focused vendors, but Google’s objectives are maintain share of search, and only secondarily to satisfy advertisers – the auction will take care of some that advertiser anxiety.

Increased Search Time & Reduced Conversion Rates

Most people are planning on reducing spend. They’ll do so by spending more carefully. They’ll look around more. They’ll be taking more personal recommendations. Trust in the stability of large organisations and well known brands will continue to be eroded – which means that the right businesses, with the right trust validations, can emerge from nowhere; just because your business has been running for 5, 10, 50 or 150 years is no guarantee that you haven’t recently based your business on false expectations of investments and earnings. A new business with the right accreditations can play on the same field as a 200 year old business, and can raise fear and doubt about the financial stability of well established players.

So, shoppers will be more wary. You’ll need to improve your sites to have online messages address current concerns. The fears, uncertainties and doubts that were addressed in marketing communications last year, won’t work so effectively this year; the concerns are different. Your customers will be concerned that even buying from a major brand will damage them, or is at least risky. Reassurance and validation will be important – especially with the media focusing on bad news. Bad news sells and media outlets will want more viewers. The at-risk media outlets, old broadcast media, seeing declining share of advertising budgets will want to stimulate sales through increasing interest in controversy, further weakening consumer confidence. Failure to address offline media reports online will impact conversion rates – though you will want to avoid directly feeding the controversy.

Minor brands should be able to gain, if they are cash flow positive or can find investors with imagination, foresight and cash. Guarantees and warranties, future-safe products and ways to reduce service costs for consumers; ways to manage domestic finances effectively; ways to cut back while still having luxuries.

The results of these will be to put some strange pressures on paid search. Companies that have previously survived on word of mouth and low or zero advertising, will need new customers. They’ll advertise. As new advertisers, they’ll make a lot of mistakes and won’t be willing to pay for expert assistance. That will increase claims of click fraud, and apply upward pressures on bids, sometimes from advertisers that shouldn’t be in the auction.

Existing advertisers will want to improve performance and will decrease budgets and target the spend on the most effective keywords and adverts. This will increase pressures on the paid search companies to dilute the inventory – the result is likely to be that overall clicks/conversion will continue to worsen, otherwise the paid search companies will report losses, and decreases in search volumes. It’s always bigger news if a big company loses, than a bunch of smaller businesses – so expect that larger businesses will use their asymmetric control of information to manage smaller customer businesses expectations and margins, through manipulating click quality. Expect the search engines to protect themselves at the expense of smaller advertisers.

ROI will generally worsen, but there will be islands where specific companies have struck the right messages that resonate with users. The result will be that paid search competition will continue to heat up. I expect that the average CPC will decline, the total value of paid search will decline, but some advertisers in various niches will have justifiably higher Average Cost Per Click and increased spending.

Affiliates

The affiliate industry will also become even more heated. Out of a job? Looking to make money with a low capital investment? Then you could become an affiliate…

However, who needs more novice affiliates? What does an influx of untrained, cash starved and at least initially ineffective affiliates do, when the real super affiliates (not the one man and a dog operations blogging from “super affiliates are us”, but the real, quiet, and highly effective super affiliates) already handle about 80% of affiliate traffic? The answer is that these new affiliates will provide free advertising for marginally effective businesses, by spending their own money essentially as an investment. Expect more companies to switch to affiliate advertising models to control their in-house marketing costs and to reduce spend on advertising agencies. Expect a lot more annoyed novice affiliates.

The affiliate industry will boom; but leave a lot of disillusioned “internet advertisers” in their wake. There will be some emergent new stars – not everyone who comes in will fail, and some existing leaders will cash out and move on. Expect some churn in the top affiliate products – but that’s pretty standard in the industry anyway.

I expect that the real super affiliates will be the same next year as this year – they have developed their techniques, and they are effective. They might retrench a little and change their focus on the businesses they are interested in, but they’ll survive and probably continue modest growth; their main obstacle to growth has historically been cash; 60 and 90 day payment cycles on sales conversions mean that working cash is tied up, effectively limiting them to “four to six inventory turns” a year. That reduces their rates of growth, and with banks not in lending mode, these guys are bottled up, at least in paid search.

Online Revenues Will Increase

Well, they will unless the internet theft scandals are outrageously large. In a search for the best value, comparison shopping online and internet supported purchases over the phone will become more important. Where businesses aren’t already selling online, there will be pressures now to do so, to reduce the costs of sales. The final result will be that more is sold online – even though total sales from all sources will decline. There may be some countries where there is a decline in online sales, but it will be less than the decline in total sales – the internet will be seen as one of the brighter spots. Just.

There’s only one reason for internet sales to increase. If executed effectively, you can reduce the costs of making the sale. That tends to be less and less true the lower the volume of sales; so expect continued outsourcing to places like Yahoo!Merchant Stores, where the infrastructure and development costs have already been put in place, and allows dropping the cost of sales. The interim step between often dreadful sites with no compelling call to action and a full online sales site, is a phone number; but placed on an inactive site that doesn’t reflect your best price and the unique value in buying from you rather than a competitor, this will have no effect. So expect the more savvy small business to look for CMS based websites where they can update their own content. I’m expecting that smarter SME’s will be investing more in templated CMS backed websites than the current static crop.

And metrics. Web analytics – free tools like Google Analytics – properly used, can reveal a lot. The problem for SMEs is setting up and understanding the data. There should be an increased demand for skills in setting up analytics, and interpreting what is happening.

Internet Theft Scandals

With money tight and some smart people out of a job… expect internet fraud to increase. With businesses looking for excuses to write off problem debts, expect more disclosure of problems, probably triggered by an inadvertently exposed online scam. This is obviously a tentative projection, as it depends on unforeseeable circumstances. However, this year is the first year since 1994 (when I started internet sales seriously), when I can see reasons to add excuses to the balance sheet, some understanding of the risks to businesses and that even corporations are affected by fraudsters, and a large value of online business that probably will increase (at least in relative terms), and an increase in technological capability by scammers and fraudsters. The combined pressures might make it attractive for the first time to attribute losses to technologically sophisticated thieves.

Online security for the average consumer has not improved in any seriously identifiable way since 1994, other than the progressive plugging of vulnerabilities in web browsers and servers. The introduction of Secure HTTP (https, or “secure server” technology) back then, provided users with a somewhat artificial degree of confidence. Many web sites offer access to financially significant resources with only an account name and password. Account names and passwords are insufficient for best security practices. Banks now often offer multiple levels of password and CAPTCHA, and may require authentication through encrypted PIN checkers – beginning to approach the holy trinity of security (“something that you are, something that you have and something that you know” – at least two of those are addressed by the better banks). However, there are still far too many ways to spend money online that are protected only by guessable account names and passwords, and by essentially unprotected mechanisms to send money to scammers.

There will be scandals about this. Probably soon. If there’s a sniff of a problem, then the offline media will pounce. These media need to pounce and will argue forcefully; their businesses are in decline, and they’ll need to savage the failures of online businesses to help protect their own. While this would be a significantly negative sum game for all online sales, the benefit of defection by an attacked business will be high; “We really made money, and we only show a loss because of internet theft we couldn’t control” will be an attractive excuse to some business, at some point. And then the skies will open and the extent of internet crime will be an issue.

That will damage sales. Hugely. Even if the effects are already part of current accounting systems, reported as part of business accounts and effectively managed by all online businesses, the *perception* of reductions in safety will be very damaging. I don’t see any sign of widespread adoption of even “best practice” account management – internet wide consistent login methods and messages, proper understanding of what a “secure server” means, and so on. While users are prepared to pay money to sites they can’t properly identify in return for promises to deliver products they don’t receive, across national borders (losing nationally accountable and interested police enforcement), this problem will continue to grow. The individual losses are small; the investigations complex. At some point, this will become a more serious problem to deal with than Enron and Madhoff. I think it’ll be this year.

Consequences

New advertisers will start by thinking about paid search – increasing pressures there. This will result in disappointment and claims of click fraud, especially if new advertisers discover that a click is not a click… different clicks have different values, even if Google conflates clicks from multiple sources. Expect a lot more noise and heat from new advertisers who feel they have been mislead.

Existing advertisers will mostly reduce and focus spend. AvCPC will have a downward pressure, counteracted by the SE’s including low value inventory – weakening conversion rates and hiding it in a generally slower market with inbuilt tendencies for longer latency and reduced likelihood of sales. ROI will get worse; the question is whether the business can take a few quarters of bad ROI in order to survive to an upturn. There should be increased attention to improving the web site, however, this will be diluted by the feelings of senior management that enough work has been done on the web site – “it’s a finished product” – and that there have been sales, so all that is needed is more visitors. IME, that’s usually wrong – there’s usually a lot of ways to make a site more effective in selling, even for long latency sales.

There will be a switch to increase SEO efforts. If purchasers will spend more time in research anyway, increasing the clicks and site visits per sale, then organic search results become more interesting. This will increase interest in spammy linking as people learn their way into search engine marketing. There will also be increased interest in affiliate advertising – if you aren’t skilled in internet advertising, then getting affiliates to advertise on your behalf will be interesting. However, this again should result in increased interest in web site design improvements; it won’t, because once a business have “put your brochure online” the naive perception is that there’ll be nothing else to do. There’s still far too many naive advertisers with ineffective web sites and a management team that focuses on volume of traffic, not the quality of that traffic and the quality of the response from the website.

There will be a decrease in search volumes for products. Despite the increased research for purchases, search volumes for commercial products will be negatively affected. That’s partially because the economy is slowing, but mostly because people are learning the online environment; they know where to go to get various things at decent prices, so they don’t have to do so much searching and researching – I haven’t heard anyone complain for the last 18 months that they are “useless at searching”. This year may be first year where the combined effects of knowing your internet neighbourhood and slowing commercial activity, actually dent the growth of both paid and organic search.

A significant internet theft scandal will cause a dramatic decrease in online consumer spending, if it bursts. Expect new web browser technology and new web site authentication services. At the moment, I’m pretty sure that the only way to tackle the identify theft problem is solutions that, as an industry, we’re nowhere near considering. The short term consequences will be to push the internet clock back towards use supporting research for purchases, not for actual online purchases.

CMS’s will become more important for SME’s. They’ll finally be able to build customised landing pages, important for improving the effectiveness of paid search.

Web analytics, neglected or misinterpreted, will increase in significance – and Google Analytics low price point makes it a very attractive platform to use, despite the poor general knowledge about how to best use it. The downside of “free” is that expectations of the value are low, and users are unwilling to pay a lot to understand the benefits; smarter businesses will invest in learning how to use GA effectively (IOW, not as it comes out of the box).

Weaknesses in the paper

There are a few embedded misperceptions in the “AdFraud Anecdotes” PDF.

I’ll probably add to this list as I think more about it.

• Advertisers do not react rationally. That’s partially a consequence of information holding asymmetry.
• Google controls Broad Match and Auction Quorum Sizes to improve Google’s returns – and hides the crap search queries in reports as “18,0000 other unique searches”
• Google is in complete control of the quality of matching, by default. Google controls click quality, advertisers choose bids and can forgo volume by selecting phrase match and exact match, or accept Google’s decisions about matching. Most of the variation in CPA & ROI that I find can be directly attributed to Google changing the nature of search queries that are being matched.
• Advertisers are remarkably reluctant to measure – because for a small business the costs of developing the understanding of measurement is expensive; so is hiring in the talent to understand the data. I infer that small businesses have a higher percentage of undetected and undetectable click fraud and greater difficulty in establishing a “useful for management purposes” CPA (internet noise makes data collection long period – I may write up the details of this).
• The paper addresses large advertisers quite well. Typical of economists and governments – but single large entities are *not* representative of the larger numbers of smaller businesses – scale changes impact.

Strengths of the paper

This shows that some people within Google are clearly understanding the role that Google has.

It is interesting that the thinking and insights exposed in this paper *haven’t* percolated to the AdWords Sales Teams, at least in the UK. I’ve recently had a rather unsatisfactory meeting with a major account team, who said that Google has no insight into conversion… Despite our mutual client having AdWords Conversion Tracking enabled for several years. If ROI was important to Google, you’d have thought they’d have noticed, and been rather more interested in how recent change have affected ROI. Wouldn’t you? Instead, I got the usual lecture about increasing spend… instead of them giving me techniques to manage AdWords to improve ROI in the face of Google’s efforts to undermine that. I may write about that more, when I can work out how to disentangle any explanation from any specific clients’ data.

The PDF provides some rather Delphic oracular premonitions of the recent user interface changes, for example, separating metrics between Google Search and the Search Partners and the Content Network; allowing (some) control over domain parks, 404 page advertising, etc. Careful reading is required and may be rewarded.

I may make some predictions about likely further AdWords UI changes, when I’ve had a bit more of a think… Frankly, stumbling on this has been a bit of a shock…

Well, when I say “wrong”, I suppose I really mean “less than complete”. I think the reason is quite subtle – and not explicitly recognised in the white paper. That doesn’t mean that they don’t know, and it doesn’t mean that they haven’t worked it out… it just got me thinking about some work we’ve done. That, in turn, means a brief look at recent history.

Google’s dominance of paid search, especially in the UK, means that many users won’t be aware of the original innovator in the arena. “First mover advantage” (yeah, that’s irony for you) went to GoTo.com, who eventually became “Overture” and then “Yahoo!Search Marketing” (Y!SM).

Bidding Models

GoTo’s original model for an advert auction (pre-Panama) was that bids would be compared and the longest standing, highest bid, would appear first. A simple auction, easily understood. Also, an auction in which automated bidding would be hugely important. Why so? Because the majority of clicks go the highest ranked listing, but the value of the listing would depend on conversion (or affiliate reward). Getting your advert to the right position would be a matter of fast paced juggling.

Automation can help reduce costs and mistakes in an auction like that, for all advertisers. Tactics, such as choosing when to bid, and how frequently, become important. And little issues like bidding gaps become hugely significant – because you pay 0.01 more than the advert below you. So if you can find a spot in which the two adverts above you are paying just 0.01 more, but you have a 0.05 spread to the advert below you, you pay much less than your higher competitors, which may justify the lower click volume on a lower position. Lots of calculations to be performed quickly. Automation, absolutely useful.

GoTo’s model for search revenue was also quite different from that for Google. Initially, GoTo had to partner with higher volume third party sites. This lead to the acquisition by Yahoo. Google’s first advertising was on their own site, and they could then sell the higher value adverts to search partners who were already prepared, by GoTo, to handle externally supplied advert inventory. This results in an apparent difference in results between Google and Yahoo. Where Google impressions may soar markedly with position, GoTo’s already broad spread across third party properties meant that the impression volume was not so accelerated by position. That was also subtly influenced by Google’s auction mechanism – covered elsewhere on this blog – which offers absolutely the highest returned value for every advert space used.

As an early advertising system, GoTo offered a single advert. It costs the publisher cold hard cash to vet adverts. So running a single advert for each advertiser for each keyword, reduces costs. Important for a startup. However, ask anyone in direct marketing what their most powerful technique is; it is the parallel A/B test. The ability to test copy for effectiveness can transform a failing campaign into a successful campaign, sometimes with the most subtle of changes. Google’s AdWords allows alternate adverts – dozens of them for massively parallel A/B/C/…. testing. This changes the way that the bidding system works – because you need to separate the behaviour for each creative that is used.

Budgets and Bids

The way in which budgets are set and satisfied also results in some intriguing differences between the bidding systems. In GoTo’s model, it appeared that adverts participated in all auctions – even if they did not have the budget to do so. Google’s model appears to run the auction only between participants with enough budget to be able to offer an advert (there’s evidence for this that I may explain in another article – it’s moderately complex reasoning and somewhat off-topic for this thread).

Something really important to carry away is that CPM and CPC (cost per thousand impressions and cost per click) models are interchangeable – if you can make some assumptions, or some measurements. For example, if you can assume a 2.5% CTR. If you do, then for every thousand impressions you’ll see 25 clicks. If you know the CPM, you can then infer the CPC. Google makes assumptions like this, all over the place – you can even infer what the assumed values are with some experiments – I may write those up. It was pretty cute working out what Google’s hidden numbers are :)

Why CPC?

Psychologically, advertisers are happier with CPC presentations of cost. For example, small advertisers may not want to buy 1,000 impressions. When presenting to board level decision makers, you can tell them that you are buying clicks, not impressions which may or may not turn into business. Clicks might turn into business and definitely turn up on the web site – and if they don’t, you can challenge the publishing system for fraud. But impressions – a lot softer… harder to explain why you don’t get clicks or conversions. So Google’s system has gone for CPC presentation – but I’m pretty certain that the auction itself runs as a CPM model.

Broad Match – Broader Than You Might Expect

Finally, Google emphasises broad match (it is the default, it is touted in training, and gives Google control over advertisement appearances for arbitrary searches – which Google can use to its’ advantage, smoothes the landscape for bidding, etc). The broad match auction allows Google to spread the value over whatever it deems relevant. Bid on your company name in broad match, and if you’ve bid enough, your advert will appear on searches for competitor names. No one who has done any courses in Marketing Communications should be happy at this point – MarCom tells you that you need to focus an advert on a competitors specific weakness, relative to your business, if you want to capture someone searching for a competitor.

These differences mean that the nature of the auction on Google is less about fast paced changes – for most advertisers, who do not have stock that ages, and in turn leads to significant reasons for most advertisers to avoid automation or to reduce emphasis on manipulating bids as a technique. The focus for success on Google shifts from bidding strategy and into ad copy, keywords, match types and landing pages. Arguably, this is the right emphasis anyway. The nature of GoTo’s auction system drove attention to bidding, and their inability to offer parallel A/B testing of adverts, demoted interest in improving CTR.

Aggregated reports also make a huge difference, too. GoTo’s model was pretty simple. You could infer, for a given time period, what the position was, and the paid price, and the impressions and the clicks. For Google, we have a variable delay time (between around 15 minutes and 3 hours, with a minimum of 45 minutes for reliable reporting), and aggregation – you can’t get to results for the last time period, for periods of less than a day. This means that to infer what has happened late in the day, you need to know what happened earlier in the day – because you’ll subtract all of those from the aggregated result. However, this leads to a blurring of values late in the day. By 10pm, you’ve had 95% or more of the clicks that you’ll get, and so changes made to bidding have a relatively tiny effect on the numbers that are reported – things like the averaged position over the day are more heavily influenced by the earlier days’ higher activity. This means that the effective measurement times to get insightful data are reduced – you can’t make sensible decisions about changes when 90% of your results have been collected.

Seasonality

What really cripples automation for many small advertisers, and can cause some interesting problems for large advertisers, is seasonality. To effectively select the right price every Christmas, you need to have a model of the change in impressions for the previous Christmas. Building up that model is expensive. Retailers carry this in their heads, but it needs to be expressed to the model before it reaches the date… otherwise you find that the model is no longer predictive, but reactive, and always behind the curve.

Take, for example, the classic New Year resolution to stop smoking. Keywords relating to stopping smoking have a low volume of search over the year. Starting just before the New Year, it ramps, and declines by the end of January, reaching a peak that is vastly higher than the normal monthly run – just for this period. If you use a predictive model, it’ll keep projecting that tomorrow will be like last month… it’s had 11 months of being right. It will typically project that tomorrow, the volume will be down. And of course, it rises again and again through January. So the algorithm spends all its’ time chasing the rising curve. How it reacts to the declining interest is also problematic – with no history and a short time planning horizon. Most algorithms will be dominated by “tomorrow will be like today, with minor changes” – so a declining curve has them retaining bids for too long as the market declines.

Because the auction is really for CPM, and broad match means that bids are compared between wildly different searches, the whole effect is to smooth the bidding landscape and slow the variation of bidding changes. Bidding on Google is pretty stable, after the first frantic period of working out th messages and building a CTR history.

Noise

Complicating all of this, is that the Internet is a “noisy” signalling environment. Bursts of searches may reflect real changes of user interest, or just statistical variation that means nothing. Designing algorithms that take care of high noise environments. with unknown seasonality, is pretty tricky. Getting retailers to describe their expected annual search impressions is also sometimes and exercise in futility – because over- and under-statements cause problems that can be worse than having a near-Bayesian model.

None of that is the defect

The missing point is that each keyword has several built-in characteristics. Different searches have a different implication, that predisposes the search user to specific types of advert. For example, a search for a brand name, especially a business’ own name, usually indicates a high intent to hear from that business, not a competitor. The CTR for a competitor will be tiny, and the CTR for the business will be high. This leads to an inverted paid price. That is, the bidding landscape will tend to have the lowest average Cost Per Click at Position 1. At other positions, the AvCPC tends to be higher, as the advert is under competitive pressure.

Even worse, if the product being sold is a niche, the Average CPC becomes somewhat less significant – because it is important that the higher adverts weed out the segments that you can’t afford to handle. The optimum position is then dictated by competition messages – inferred by increasing conversion rates in lower positions.

Getting the bid right can’t rely on a simple Newton-Raphson optimisation, but has to look for non-local minima. Programming systems to find non-local minima is tougher than most businesses can handle. It is possible that Sayu.co.uk has handled this – but a number of the largest automated bidding systems, especially those popular with large agencies, seem to lack any serious bidding model.

When the non-local optimimum happens to coincide with the traffic maximum, and automated bidding systems fail to identify that, it is a pretty poor failing.

Anyway – thanks to Gerda Arts and Sayu.co.uk – it’s good to have someone explaining what they do in more detail than the handwaving at bigger competitors.

Ranking By CPM, Not By Bid

If Advertiser-1 has a CTR of 10% and pays (that is “pays”, not “bids”) $0.10 per click, then they have an effective CPM of 1000 impressions times 10% CTR times $0.10 Average Cost Per Click = $10.00.

If Advertiser-2 has a CTR of 1% and pays $0.99 per click, they have an effective CPM of 1000 times 1% times $0.99 = $9.90 effective CPM.

If you were running a search engine advertising company and wanted to make the most money, would you make Advertiser-1 your highest position advert, or Advertiser-2?

You’d pick, I hope, Advertiser-1.

This simple calculation demonstrates why a simple auction based on ranking bids alone fails to deliver the best revenue for the search engine. It also deeply satisfies a need for search engines… clicks. A “good page” for a search engine, in 1997, would have been one that got the most clicks – the most results were interesting to users. These days the value is a little more complex to calculate – a search engine should be looking for satisfaction with a search results page, not just clicks; if the number one result satisfies most users, and the rest of the search results satisfy all other users with a single click on just one of the other results, it has done a good job… but working out the balance between lots of clicks and inferring satisfaction is much harder than looking at percentages of clicks. Hence the AdWords Quality Score… more on that later.

Example Strategies

I’m going to describe three advertising systems, and I’ll call them Asearch, Bsearch and Csearch. I genuinely don’t have any intention of mapping these to existing paid search businesses – I’m using them to explore how apparently small changes in the mechanics of running a nominally similar auction can result in revenue changes – or, from the advertisers perspective, ROI changes.

All the businesses use the same basic model:

• Adverts are submitted with keywords and bids
• For each impression, each keyword can show up in no more than ten advert slots per page
• Adverts are ranked by revenue – that is, they are ranked by their effective CPM, not their bid

Budgets

I’m going to throw in another real world limitation – budgets. Advertisers like budgets. Budgets allow planned spending and avoid unpleasant month end surprises, or running out of cash on day 15 of the advertising month.

Hurrah for budgets.

However, budgets are interesting for paid search companies – they affect the auction in some odd ways.

Let me also assume a few advertisers. 1advertiser has a high budget. If there was a way to get someone extra to their site through search, they’d take that chance and so they bid 120% of the daily actual spend. 2advertiser has a low budget – they can only afford to spend 10% of the budget that the search engines recommend to them, and their strategy is to appear as Number 2 when they do appear. 3advertiser is B2B and sees a wide variation in search volume and conversions – weekends are pretty much dead for sales, there’s a lot of research activity on a Monday, and Thursday tends to have peak sales volumes. They bid to Position 3 with a 100% budget. These are simplified bidding strategies, to illustrate what happens. I’m not claiming that these are optimal bidding strategies or even realistic companies.

Asearch, Bsearch and Csearch look at budgets and all of them apportion 2advertiser (the 10% of estimated budget guy) a rate of 10% for showing their advert. That is, the advert only appears on one search in 10, on each of the three search engines. However, the SEs have a different way of running the auction:

• Asearch includes all advertisers in all auctions, whether they have the budget to actually appear, or not. The positional calculation is done at the beginning of the day.
• Bsearch includes all advertisers in all auctions regardless of the budget. The positional calculation is done for each impression.
• Csearch only includes advertisers with the budget to show an advert. The positional calculation is done for each impression.

Assuming that all systems have the same number of advertisers (same competition levels) and the same type of audience, and uses the same system to match search query and keyword, which of these small changes to the auction will have an effect? I believe that Bsearch will make the most money from advertisers on a pure economics basis. Why?

Who makes the most money?

Csearch apparently leaves money on the table. Each non-participating advertiser means that the values at the top of the auction are lower. If you only have 10 advertisers, and one of them drops out, then the remaining 9 advertisers will see reduced competition and on average will pay less per click than if all ten advertisers are jockeying for position. Although the auction mechanism is the same, especially with large numbers of advertisers using budgets of less than 100%, each auction is a new activity and as a result the cost per click varies greatly, depending on the exact participants in each auction. It’s actually slightly more complex than that. I might cover the effect of varying volumes on search on paid price granularity in yet another article.

Which of Asearch and Bsearch makes the most? In pure economics terms, I believe that it is Bsearch – because the revenue is optimised over shorter time periods. That is, if there are transitory reasons to prefer one advert over another, then Bsearch will respond faster to changing market place needs. However, Asearch still does better than Csearch – because the larger number of advertisers in each auction should maintain the price higher. For Asearch and Bsearch, just because 2advertiser doesn’t have the funds to pay for the advert, doesn’t mean that they aren’t in virtual position 2 and that 1advertiser should be paying a penny more than 2advertiser. Effectively the auction is run for more than 10 places, and then any advertiser that doesn’t have budget to participate, simply doesn’t have their advert shown.

Easiest to build?

In implementation terms, Csearch is the simplest. At the beginning of the day, Csearch calculates the ranking for all advertisers, and only recalculates (just once) if a participant in the auction for that keyword has changed their bid. This means that advertisers can be given a reliable price and a reliable position – so long as no other advertiser changes their response. It makes for a very predictable system. However, advertisers pay a penny more than the costs of non-participating advertisers. If this system is used, a competitive strategy would be to bid on competitor keywords, but with a miniscule budget and a high bid; this would help drive up the auction value at low expense to the advertiser, costing competitors quite a chunk of change. 1advertiser would see a consistent paid price, no matter whether 2advertiser’s adverts show or not.

Maximal Revenue

Bsearch conducts an auction for each impression, but also assumes that all advertisers are present. This allows Bsearch to optimise revenue in each cycle, which should lead to maximimum revenue. Again, the “absent” advertisers drive up the Average Cost Per Click for those with the budget to actually appear. 1advertiser would see a consistent paid price, whether 2advertiser appears or not. The most obvious way to diagnose this rather than the Csearch strategy is what happens when the system moves from using an assumed CTR to using a measured CTR – Csearch will revise the position day by day, but Bsearch will change position during the day, especially when the advert is fairly new. Um – that really needs another lengthy article. Just take it from me for now that there are some diagnostics for inspecting which models may be being used… I might give away some inference tricks in a future article.

Fairest Auction

Asearch conducts a “fair” auction. That is, the cost of a click is only calculated by reference to participating advertisers. If an advertiser doesn’t have the budget, then that advertisers bid is not part of the auction. Asearch will get the lowest revenue, but offers an auction that most advertisers will consider fair. When 2advertiser is not showing, 1advertiser will see lower competitive prices – so the average cost per click will decrease.

We can make some predictions about the behaviour of the advertising system, as seen by advertisers. However, like anything involving reality, it gets more complex as you look deeper…

CTR Assumptions And Budgets

I rather glossed over the effect of budgets with glib statements about “2advertiser only has 10% of the budget” and went on to say that that meant they’d only see 1 in 10 impressions. It is, of course, more complex.

The Click Through Rate estimate affects the spend, especially before the advertising has developed an account history. Create an advert with a CTR that beats 2.5%, and you can go over-budget. This happens because in order to estimate the 10% threshold, the SE must assume a CTR. If you beat the CTR, then the system keeps delivering impressions at the 10% rate – but with a higher CTR, you get more clicks per impression. The result is that your spend is higher than the predicted value. You go over budget.

When you go over budget like this, it isn’t your fault – the system delivered impressions based on a poor estimate of the click through rate. Reputable search engines will reimburse you for their failures to correctly estimate your spend. For example, Google will refund overspend built up this way. That’s good news for the advertiser, as they get some free clicks and perhaps even some conversion on the first day or so of overspend.

It is even possible (using tricks I won’t go into here) to manipulate Google into fairly consistently underestimating, so you get lots of days of partially free clicks. Google, of course, doesn’t like this. You’ll be dicing with a permanent ban if you use techniques that might result in heavy overspending, consistently.

Assumptions

It is possible to examine what assumptions the SE’s make. Experiments can be done to probe what an SE considers to be a normal CTR, when it makes assumptions. This appears to be increasingly harder to do… perhaps because the SE’s are now getting smarter. A few years ago, the assumed CTR was the same for all keywords, but it shouldn’t be. Different keywords in different market segment, reaching people at different stages in the buying process, will have different basal CTR’s. And that means that the CTR assumption should be different for different keywords. I see some signs that that the SE’s may be starting to do this per-keyword assignment of assumed CTR…

Going Over Budget Costs the SE

If the assumption is of an average CTR, then, on average, you haven’t lost money… Um. Wrong. When the SE overestimates the CTR, then the advertiser sees their position gets worse when the CTR history kicks in. If the CTR is better than the estimate, you get more clicks on that day, exceeding your budget, meaning that you may get free clicks and the search engine has lost revenue by feeding you inventory that it could have used for revenue generation on someone who had remaining budget. So it is vital that they decide how to adjust the advert flow rate to get the best revenue – that is, so that advertisers budgets are exhausted, when and only when the impression inventory is exhausted.

So how do search engines defend themselves from giving away clicks? Especially, how to calculate the rate of impression serving for the whole day, at the beginning of the day, and not adjusting the rate, throughout the day? That is, calculating for each advertiser, dynamically, whether to show the next impression, or not, is expensive – more expensive than deciding once, at the start of the day, what the flow rate should be.

Note that this is crucially dependent on the costs of that calculation. Deploying more hardware, especially when you consider Moore’s law, allows the decision to be made dynamic, rather than once per day.

One of the first exercises is to excessively throttle those with low budgets. If you can use up all the inventory with advertisers who offer a higher budget than needed, then you avoid giving away free clicks to advertisers who have a limited spend.

For example, if 2advertiser (the 10% of estimated spend guys) are advertising, then a search engine should allow for some variability in search user response. If the SE allows, for example, a variation of 25%, then they should only allow the advertiser 80% of the impressions that would normally be expected to exhaust the budget. Otherwise there is a risk of selling the inventory of impressions for free. If the budget is, OTOH, much larger than the expected spend, then offer every single possible impression. See Google’s policies on spending over the budget.

If advertisers can detect that the flow is throttled by the search engine, then they should change the tactical response to setting the budget… So if you given an estimate of spend of $100 per day, and you consistently spend at the $100/day mark, and you suspect that the SE might throttle based on budgets. offer $120, and see if you get more impressions per day, than when you offer $100 as a budget.

Summary

I hope that I’ve convinced you that the search engines can make different choices for how to operate an auction. Even if the auction appears to use the same rules, the details of those rules can affect the revenue returned to the search engine, and consequently the ROI perceived by the advertiser.

It should also be clear that larger auctions generally act to raise the price. This has an implication on what seems like a purely technical issue – how search queries are matched. As this discussion *should* have made clear, the apparently technical choice of which search queries map to which keywords, affects the revenue from the auction. Search Query Matching is as much an exercise in economics satisficing, as it is in technical excellence.

Budgets are not simple – they affect ad flow rate and may, depending on hidden decisions by the search engines, reduce the rate of adverts even for an advertiser intending to achieve full flow advertising. This is because search engines want to strike a balance between offering inventory for free and offering advertisements throughout the day to exhaust an advertisers’ budget. Leaving unspent advertiser budgets is as bad as delivering free inventory.

Updates

2008-09-24 Typo corrections.
2008-10-05 Typo corrections; changes for clarity.
2008-10-23 More typo corrections. Added link to Google policy on budget overruns.