AdWords Quality Score Changes To Correct A Design Defect?

Published on September 16th, 2008 by Jeremy Chatfield

Earlier this year I observed several accounts where the impression volume had drastically decreased, badly damaging the clients’ business. Google insisted that they had changed nothing. To those damaged, it looked like an organised change of some sort. There have been other instances, such as around July 11th, when another step change was visible to some clients. Still Google denies any changes to the system.

How can you have a system that doesn’t change, but still have various customers complaining that on specific coordinated dates, their accounts have been badly damaged? I think there’s an underlying reason, and it is a subtle design defect. If the description of the latest changes to AdWords Quality Score are accurate, the cumulative effect may be that this design defect is corrected.

Artificial Intelligence and Learning Sets

I have long suspected that Google uses, in addition the PageRank derived algorithms, some kind of AI that collects information about clusters of words and their proximity. I think this is derived from organic search, by looking at large quantities of pages and correlating the presence of various words. Clusters of words in the same contextual area are a good thing for a page. My further guess is that clusters of words from disparate areas weaken the likelihood of relevance - another component of PageRank. I haven’t gone mining patents to see if this is mentioned, but I’d be fairly surprised if it isn’t patented, by someone at Google - that’s a prediction from this hypothesis :)

This same knowledge base of contextual similarity could be used for paid search. Evidence is in the way that AdWords assigns the very first Minimum Cost Per Click for a keyword. As examples of evidence for this, there was an interesting common problem for many new programmers of the AdWords API (application programming interface). If an AdGroup is created and populated with keywords before any adverts are offered, then the Initial Minimum Cost Per Click is the same as if the keywords completely failed to match any words in the advert - the “$10.00 MinCPC” problem.

Yet, if you use “home loan” as a keyword when the advert says “mortgage” you don’t get charged $10.00. You probably get charged around $0.11 to $0.22. It’s not as low as the MinCPC when you use the same words in keyword and advert, but Google has learned that “mortgage” and “home loan” are synonymous. So you get a minor penalty. Why be penalised at all? Because Google knows that echoing what users type generally increases the CTR. An advert with a copy of the keyword therefore is more deserving of a lower MinCPC than an advert with a synonym. The weaker the synonym, the higher the very first (Initial) MinCPC you are offered.

I suspect that humans are involved, and an assignment to a human selected “level of match”, because these “weaker match” pricings have distinctive values. When I see “$0.22″ in a new account, I immediately assume that it is the Initial MinCPC, just after the AdGroup has been made, and that the keyword is a close synonym of advert copy. If I see a $0.50, I assume a weaker synonym. At $2.75, I’m thinking about a word in the same area - say “flights” matching with a specific airline name; not exactly a synonym, but if you were thinking about flights you are probably also close to thinking about an airline and vice versa. So in the current system, matches are assigned to bands, and crossing a band will result in a specific value of MinCPC to be *initially* assigned. It can be changed later by the AdsBot, human reviewer and CTR history - so looking at an account that has been running for some time will not usually show these signature values.

This is similar to the way in which Google has to make an up-front decision about which keyword of a set of synonyms will attract impressions. The non-preferred synonym attracts no impressions and the preferred form gets all impressions - even if the other form would be a better exact match. The non-preferred form is Inactive For Search.

Changing Without Change

But, what happens when you reset the system? When the “learning set”, used to train the AI is changed? It will recalculate networks of relatedness. It does not need to remember how it used to assign relatedness before - with the AI’s that I’ve used, you *couldn’t* retain the memory, but had to completely rebuild the network. It has been many years since I was involved with AI work, so it is possible that newer techniques could mix old networks and new training sets, but I’d be willing to predict that if it is done, it is a rare technique. As a result of changing the learning set, specifically the new order in which the AI-in-training encounters data, and a new set of data with slight variations in search content and results, can cause different choices of keywords to be selected as representative. The result is that some keywords previously chosen as active, become inactive, and vice versa.

Consider, for example, a brand with a unique product name, such as the iRobot Roomba. Someone searching could look for an “iRobot Roomba” or a “Roomba”. They mean the same thing. “Roomba” and “irobot roomba” are synonyms. But Google tends to allocate all the impressions to one or the other, in Broad match. If you have both keywords in the same account, you tend to find that one or the other attracts all the impressions. (I should point out that iRobot is not a client, and this is a made-up example for which I have absolutely no evidence; I didn’t want to use a clients real data to illustrate - it uses the same generic principle that we have observed in multiple clients, though).

The assignment of Active vs Inactive does change. We have examples where, in January, all impressions go to one form. By March, traceable to a specific date on which the change happened, all impressions for those products have switched to the other form (identified in several unique brand/model names in our clients’ accounts). No one at Google has made the decision, but that’s what the AI has learned is the best way to match… Google reps can say that nothing has changed.

Google Slap

When this happens with rarer searches, and a business depends on that stream, it can look as though Google has made a decision to pull the plug on the business. Impressions on a carefully chosen set of keywords can die to nothing, overnight. Phone calls to Google will result in denials of any changes to the system. But still the business is in the pits.

However, most businesses have a mass of inactive keywords. So long as the account is generally doing OK, few people spend the time to investigate whether the pattern of impressions has changed. As a result, large numbers of businesses will concurrently report no change in behaviour. As the story of individually damaged business is put together, it looks increasingly as though Google has selectively targeted specific businesses. But the system hasn’t actually changed. It has changed the data set that it uses, but the system is still the same. And as a consequence of making once-and-for-all decisions about steering search queries to different specific keywords, some accounts will be unexpectedly hammered.

Learning Sets and Wall Street Analysts

Looking at the accounts that I have, I think I can spot some specific dates when learning sets were updated. The frequency appears to be increasing, and appears pretty closely linked to the points when Google has talked to Wall Street Analysts about improvements to Quality Scoring. Retraining with larger data sets should allow the system to more accurately choose the keywords used for Broad Matching, and thereby increase quality of user experience. So I can imagine preparing a larger training set, talking to the analysts, and releasing the new training set in the expectation of better matches. And then being surprised when a number of businesses claim a foul play.

Two birds, one stone.

If I were a conscientious designer, that observation of damaged businesses, might lead me to reappraising a design decision to allocate search queries to specific keywords, and to make the system more dynamic.

That’s what this announcement says that Google are doing. Making the system less pre-determined, and better able to match searches and keywords. It has the additional effect that this should also increase the number of bidders in an auction. If you’ve read the article about Paid Search Auction Strategies, you’ll understand that this design change affects the value of the auction - Google makes more money off auctions with more bidders in them.

New System

Under the new system, Google say that no keyword is ever truly inactive. This could be a consequence of allowing all synonyms to share in impressions. If this is so, then the era of the “Google Slap” may be over. Instead of these unpredictable dates when your small business with a finely tuned keyword set, suddenly loses all market traction, the new training sets will simply result in improved quality matches for users, and advertisers may instead see a rising performance as the AI better matches searches and broad matched keywords.

So, here’s my prediction for the new system. More keywords will get impressions. There will be fewer “Google Slaps”, and they will coincide with announced policy changes. Average Cost Per Click will increase, as more bidders are engaged in each auction.

However, the system will also continue to Broad Match search queries for competitors. This is because competitors are synonyms for organic search purposes. The low CTR when adverts flown on one brand name are shown against a different brand name don’t seem to have any effect in the old system. In the new system? We’ll see, but I don’t expect the system to learn from paid search… the training set will continue to be drawn from organic search.

I have become so attuned to this behaviour in Broad Match, that my automatic assumption is that I should add negative keywords for competitors. I’ll have to run some experiments and see whether this part of matching has been improved.

Summary

Watch your account carefully. There are still plenty of ways for a change like this to behave unexpectedly.

I’ll be monitoring my client accounts closely. I’m pretty sure that we’ll see some hairy stuff happening, but I do expect it to even out over the next few weeks.

Updates

2008-09-17 Edits for clarity. Additional observation about the likely effect on Average Cost Per Click. Pre-emptively declared that iRobot/Roomba is not and has not been a client, before anyone asks and they raise questions. More sub-headings to break up the dense text wall a little. :)