Automating Content Network Management - Part 1

About three years ago (2005), we started efforts to automatically improve performance of the AdWords Content Network, for advertisers. We were hoping to develop a product, but we were also doing some research to see how things worked and what lessons we could learn. This is intended to be part 1 of a multipart article, focusing on the background for the most obvious techniques to improve performance. In part 2, I expect to cover more of the practical issues and some more of the observations we made.

The Content Network is Google’s name, in AdWords, for most of the AdSense network. There’s an additional component in AdWords that accounts for the rest - “Placement Targeting”. You need a different campaign type for Placement Targeting, and you may find that sites that are presented using the Content Network are not available using Placement Targeting, even if they deliver a good ROI for you. I do not know, and have not researched, why some Content Network sites are unavailable for Placement Targeting.

Managing keyword search and content match in a single campaign is fraught with problems - detailed in other articles, here and elsewhere. As long as three years ago, we’d recognised the issues and were already optimising separated Content Network campaigns. The problem that we were trying to resolve was that, even with separated content match campaigns, we had great difficulty in achieving a good volume of conversions at a competitive ROI. Every time we drove up the volume, the ROI got worse. Every time we drove to a great ROI, we lost volume.

After some investigation of the actual sites that drove business or drove unproductive costs, we became unhappy with the state of many of the sites on which adverts were published. Generally either because the content match yielded low relevance sites, or because the sites were constructed around steering traffic to adverts - and sites like either of these categories generally showed the lowest conversion rates.

The graphic below shows how a UK English user, with no search history in Chinese, a browser set to English, in receipt of Chinese language spam in Gmail, is shown Chinese adverts. This sort of “just the content” matching is partially why the Content Network has such a low CTR. There’s other reasons to do with the Buying Process and Intent, but contextually irrelevant matches through word matching is probably one of the main causes:

The Content Network & Gilligan’s Island

We decided that the only ways to escape this trap (the inverse correlation of ROI and volume) was to exclude sites that absorbed money, but delivered no business. The measurement system was to be Google’s own AdWords Conversion Tracking (GACT) and the metric would be unique measured conversions.

GACT does not offer impression tracking, so conversions would only be trackable if the visitor clicked on an advert. In addition, Google appears to prefer the first click in a 30 day window - so if the visitor clicked first on a paid search advert and later returned via content match adverts, then the conversion would be attributed to a paid search. It is *NOT* clear to me what Google does when the first click exceeds the 30 day cookie window - for example, perhaps the second click could now be counted as a conversion in the 30 day cookie window, providing new attribution of a previously ascribed sale, to a later click. Using measurements of this stuff is pretty complex, especially when you use someone elses’ stats collection and reporting system and their documentation is kept simple. However, if the products that are sold are unlikely to be repeat sales over a 30 day window, GACT stats will be much more useful than not having them.

There are other techniques to optimise the Content Network. For example, the choice of geotarget, keywords, the budget and bid all affect where the impressions are delivered. Google also probably has controls, hidden from the advertiser, that affect the choice of sites for publishing. We infer the likely presence of these Google-managed controls, from periodic changes in behaviour that would otherwise require seismic shifts in the way that the internet works. Groups of sites and the flexibility of content matching varying in huge ways over a period of days; I contend that the simplest explanation is that Google is testing ways to optimise their revenue, rather than that substantial sections of the internet change allegiances to publisher networks in such short time periods - especially when some of the sites coming and going clearly follow “Made For AdSense” templates.

To do the site optimisation research, we assumed that geotarget, advert copy, keywords and budgets had been optimised by another system. The product research area was to investigate whether it was possible to optimise site exclusion with automated techniques. Complicating it further was that at that time, Google restricted accounts to 500 site exclusions - this may sound like a lot, but it can be exhausted by a moderate sized advertiser in a month or so. This limit has been relaxed in late 2006, IIRC.

If the project worked, then a major human exercise in identifying sites that were likely to perform poorly, could be automated. This would save costs for advertisers by denying unproductive spend with Google, and reduce management fees for users, allowing a larger user base for my business. The causes of the unproductive spend could be click fraud - the Content Network seems a particularly ripe target for fraud - or could be users who just weren’t anywhere near ready to purchase, outside the 30 day window allowed by Google’s cookie and that of most web analytics packages, or poor relevance. For the purposes of improving ROI and volume, the exercise is not specifically to tackle any single cause, but to maximise the profit whatever the problems are. However, when we get to the next article, we’ll see that the causes of low performance are intricately bound up with performance improvement.

We decided that there were two main techniques for optimising site exclusions, which need not be exclusive. The rational basis for decision making is to use measured ROI. The other technique was to train an Artificial Intelligence (we picked a neural network) to choose sites as likely to be effective, or ineffective, after a single click from that site. Indeed, for the purposes of training a neural network to recognise low performing sites, we could use the ROI technique to identify how sites performed, and then use those sites as the learning sets for the AI. If the predictive powers of the AI are good, after training, then sites can be rapidly identified as being more or less likely to generate sales, saving serious costs.

So here, then, was our escape from the trap. Use economically justifiable techniques to identify sites. Build up a database of sites that work and sites that fail. Infer the characteristics, using an AI, that will allow us to select likely poor performing sites and high performing sites after just a single click. Would this be enough to get us off Gilligan’s annoying Island, or see us still trapped and ready for the next episode?

Identifying Poorly Performing Sites Through ROI Targets

The first technique is a basic “proof of incompetence” test. Assuming that the bid, budget and advert has been optimised, then ROI calculations will offer a count of clicks. Example:

Average Paid Price is $0.10

ROI Target is $10.00

Average number of clicks to achieve ROI = $10.00 / $0.10 = 100 clicks

There’s a particularly complicated calculation to work backwards from this, to the number of clicks you must see before you can assume with a specific confidence level that you are not going to achieve the target ROI. I’ll run through a simplified version of the thinking that leads to the calculation.

Imagine that the first 100 clicks do not lead to a sale… can you turn off the advertising to that site? No. Because click 101 could yield a sale. So long as we then get a second sale before click 300, we’re still achieving an average of 100 clicks per sale - to a definable confidence level. The more clicks and sales we see, the higher the confidence level. So if we got one sale in 1,000,000 clicks, we’d be very confident that we were not going to average 100 clicks per conversion. OTOH, if we saw 10,000 orders in 1,000,000 clicks, we’d be very confident that even if we saw 500 clicks and no sales, that it was not likely to be a sustained problem.

For simplicity, lets double the target ROI click volume… In this example, we must see 200 clicks and no sales at all in order to decide that this site is unsuitable *for this offer*. It may be suitable for a different offer, of course… Because Content Network matching is literal, irrelevant advertising is a frequent hazard (look at the screenshot of a poor match to a page resulting from a search for “akismet-admin”, offering Windows XP Registry tweaking - completely irrelevant and probably matched on “admin”).

Using the numbers above, the proof that the site under consideration will not convert, is a spend of $20.00 (twice the ROI target). The confidence level for this is pretty low; it’s better than random, but still quite low. However, stick with it and let’s see where it gets us.

If you have only a few sites appearing in content network AdGroups, then this additional payment over the ROI, to prove that sites *can’t* make the ROI, is a burden initially. Say that you have ten sites under consideration, and they have equal volumes of traffic. Overall you may be achieving a $20.00 ROI, and so you need to double the performance (remember that all other factors have been optimised - we’re now only looking at which sites to exclude). So we need to at least halve the number of sites involved. That means we need to waste 5 times $20.00 (the proof of incompetence level) to identify sites that definitely won’t work in the target ROI range - a “wasted” $100.00. However, that means that the remaining sites must achieve an ROI of $5.00 or the clients’ target has been failed…

In practice, you need to allow a significant overspend on useless sites, in order to ensure that you end up with a collection of sites that achieve $10.00 average ROI. The overspend is function of the numbers of sites to which you are exposed. The larger the publishing network, the less likely you are to see the same site repeatedly appear and the more sites that appear with low spend levels below the point at which you can reject them.

Now, that’s assuming an unrealistically simple model. Let’s make that model more complex and closer to reality.

The usual behaviour in the content network is a profile similar to a power law (Zipf’s law). The distribution of clicks and impressions will tend to follow a curve with a few sites that attract a lot of impressions and clicks, and a lot of sites that attract a few impressions and clicks. For a large client (think $10,000 per month, testing on the Content Network) this might translate to around 2,000 sites, of which less than ten will achieve or exceed the $20.00 target spend, and the vast majority of which have a handful of clicks.

This means that the client faces a first month cost of $10,000, but has only positively identified a handful of sites as meeting or failing to meet the success criterion. We can exclude these sites (make them a placement targeting target if they worked, otherwise we exclude them). We now have another month of spend… and a similar sort of ratio. A handful of sites will rise above the detection threshold, and in addition to the previous thousand sites, we’ll see four or five hundred new sites, and of the sites that we previously saw, we won’t see half of them again, this month. So the spend gradually increases, but the rate of accreting validated sites is just a few identified sites every month, with an ever increasing count of sites that we’ve not previously seen (though this rate of adding new sites declines - but is subject to factors outside the control of the advertiser).

Particularly note that the ROI is being measured with respect to each site. This means the overall campaign ROI will be much worse than the target. In the first month, there’ll be quite a bit of explaining to worried clients that this is just what was expected and that it will eventually get better. IMO, that’s not a message that most new clients will be happy to hear… so this technique isn’t very “client friendly”…. You’ve just charged them a bunch to set up this complex stuff and the first thing that happens is an overspend against the ROI target. OTOH, they could have stayed with the old system and have achieved pretty much the same result. This isn’t a good harbinger. This makes for a specific type of sales problem, that you probably recognise from products that you’ve used.

It takes many months to identify a large enough group of repeating conversions to present a useful collection of validated sites… but each month we see a $10k spend. It shouldn’t take any more explanation to reveal that payback times using this technique are *very* long. Practically, the reliability is low because it depends on the sites with success continuing to succeed….

I’ve developed a sincere scepticism of assumptions of sustained performance of a site in the content network, over the years I’ve been using it. I’ve found sites that provided repeat conversions and then after a while, the site changes tactics, or Google tweaks their hidden levers, and the performance falls through the floor. I’ve even had repeat converting sites simply drop out of the Content Network - for me… continuing to display adverts from competitors and completely irrelevant advertisers and having those sites unavailable in site targeting. It is an exercise in frustration.

Take Google’s own property, YouTube, for example. Last summer, I had a client with some conversions using a specific target on YouTube. Google changed the targeting and despite overspending the ROI target, I couldn’t achieve another sale - the location and the matching of video content were simply not working, for this client, any more. Achieving a single sale, or even repeat sales, does not guarantee that the site or the performance will be the same, next month.

Second Technique - the AI

OK, so we’ve seen that only using target ROI (plus some fudge to allow for the internet being noisy) can be a pretty fast way to lose large quantities of money and a slow way to identify good sites. If we can use the patterns that we find, perhaps we can use a pattern-recogniser to more quickly identify sites that don’t work? Then we can choose to dump the ineffective sites and focus spending on sites where we can’t determine an answer or know the returns to be fine.

AI techniques offer some help. A good tool for recognising unknown patterns is the Neural Network. We built a three layer neural network. You need to train a neural network. We believed that around a thousand sites would be needed to train and test the Neural Network… a thousand good sites, and a thousand bad sites, and an additional set of sites that had known value, but that were not part of either of the training sets - 3,000 sites in total.

Complicating this further, the results only hold true for a specific product. If a different product is offered, it is possible that a site that was previously failing, may now work - and vice versa. So the sites that were marked previously as “Good”, “Bad” and “Unknown”, could be marked after a second trial as “Always Good”, “Always Bad”, “Sometimes Good”, “Sometimes Bad” and “Still Undecided”. That is, there is probably a group of sites that, whatever the advert, will tend to have a lower than usual conversion rate. Conversely it is unlikely that there will be many sites that, whatever the advert offers, have a high conversion rate.

Another way of categorising would be “Sites that seem unlikely to convert, whatever the offer”, “Sites that might convert for a relevant offer” and “Sites where we need more evidence before deciding about this offer”.

Categorising the sites is the most difficult part - because content matching is so dependent on the use of words, not of context. The result is that a site that may not convert for one offer is not always poor at converting for other offers - maintaining a universal list of sites that don’t convert is made harder by having to expose those sites to multiple offers, until proving that *nothing* sells on them.

AI’s need good categories - the clearer the signal in training the AI, the more likely you are to get a decent result when applying the AI to real world data. So, for example, if the criterion is “artistic” - then you’ll really have to train the AI to detect what “art” means. If, on the other hand, the decision is “half the adverts appear between top of page and the first content”… well, it is *likely* to be easier to train - if you can write the software to perform all the CSS and JS jiggery that is possible.

Part of the exercise, then, is to define the types of entity that are present in the input, and from which the AI will draw its’ inferences.

Rounding Up Part 1

ROI based techniques aren’t worse than standard management practice - but are more expensive than techniques that rely on identifying whether a site is likely to be effective, based on immediate inspection after the first click.

Automation could be used to identify sites on first click, and add them automatically to site exclusion, pending investigation (human or automaton).

There appears to be a potential to identify, especially across a wide range of advertisers, sites that generate clicks that don’t convert for any offer. Optimising for a single advertiser looks like a long, slow process, from the stats given above.

Coming up, in Part 2

Collecting the data, hazards of data collection, risks of site exclusion and the likely response of the Get Rich Quick guys who build websites that earn them money, but don’t earn anything for you. This has significant impact on whether a system is workable, and the investment involved to build and deploy it. It also ties into bot nets and human networks of fraudulent clickers, and Google’s undisclosed techniques for identifying clicks.