SEO, Click Fraud and Mis-Attribution

Published on November 23rd, 2009 by Jeremy Chatfield

I’ve been involved in some paid search click fraud measurement for about five years. It’s pretty interesting work, trying to understand whether the clicks you’ve bought are related to the traffic on the site, and any qualifiers that you’ve added, such as geotargets and the keywords. Oddly, it has provided a sideways illumination on a topic in Search Engine Optimisation - web analytics attribution.

I don’t remember seeing this addressed in any of the web analytics blogs and books that I’ve read for the last few years. It’s possible I missed something, and I’ll gladly add references to those articles, if anyone contributes them in comments here, or in emails to me.

Observation

Like much of good science, this starts with an observation:

When you look at the sources for which you have control of the requested URL, a substantial fraction have no referring info.

To understand what that observation means, we need to know a bit more about how visitors arrive, and what sources of data we have about the visitor.

Page Requests

Paid search, banner advertising, email and so on, allow you control the page request offered by users. You can add tracking parameters, and that lets your web measurement system collect that the request came from a particular source. That’s pretty important for click fraud measurement. If you aren’t adding tracking tags, then the visitor could be turning up from anywhere for any reason.

For example, lets’ assume that you are advertising for an Organic Cosmetics business. You place an advert on a keyword offering “organic acne treatment”. You bring people to a page on the site, for that treatment - http://www.organic acne treatment.com /our-products . You *MUST* add a tracking tag, or use some kind of autotagging (offered by AdWords, Yahoo!Search Marketing, etc) so that clicks from advertising can be measured by your web analytics service, and attributed to your paid search campaign.

What the tag looks like, will depend on your Web Analytics package. If you use Google Analytics, you might have tags like “?utm_media=ppc&utm_source=yahoo&utm_creative=organic+cosmetics”. So the requested page will be something like:

http://www.organic acne treatment.com /our-products?utm_media=ppc&utm_source=yahoo&utm_creative=organic+cosmetics

Without that, all you know is that someone requested that page… or is it?

And, of course, knowing that the click arrived as a result of an activity where *you* control the page request (typical of paid advertising opportunities), means that you now have a handle on part of the click fraud questions.

Referer (sic)

Something else that a web browser can optionally tell the web server, is known as the “referer”, in RFC2616 about the HTTP protocol (See Part 14, on Headers). This “referer” header sent from your web browser to the web server, says what the page request was, for the page *before* the request to the server. So we might see that the request for our page in the example, was referred to by “http://www google com/search?q=organic+acne+treatment” (and there’s usually some other stuff in there, describing the language and the browser and so on). So we know that a search engine was the source, and we know the keyword.

However, we don’t *know* that this click was from paid search. A common frustration amongst new advertisers is failing to identify clicks from AdWords, because advert tracking tags have not been added. Without the tracking parameters, all you have to work from is the “referer” header volunteered by the browser - and that only tells you that the user was looking at a page, not whether they clicked a free or a paid link.

So, one common misattribution is to *over-allocate* clicks to organic search and *under-allocate* to paid search, because neither tracking tags are available, nor is there any common way to tell from the “referer” field, that the source was paid search. I’m going to ignore the further complexities of referrer headers when using contextual advertising - it is much more complex :)

Tying the Pieces Together

Assume that we now have tagged clicks, so we can tell the paid sources that are sending traffic. We have the paid source telling us what they claim. We can match the two pieces of data. If they don’t match, then we have some further questions about the sources of mismatch. I’m not going into that, in this article. I’m going to take a sideways look at that other data-stream, the “referer”.

When you have a good source of paid clicks, one that you can trust as delivering a high fraction of the clicks they claim, and where we get good conversion rates and matching keywords and search queries, we can infer that the people being sent are are also “good” - no or few spoofing robots, no or few paid clickers or fraudsters, etc.

If we now look at the “referer”, we should see that all the visitors come from a page with paid search on it, shouldn’t we?

We don’t. Anywhere from 10% to more than 40% of the clicks have no referer_info at all, or have some clearly dummy data inserted instead of a real referrer.

Attribution Of Origin

So if we have no tracking parameters, we can only know what the origin is, if we have the “referer” field correctly filled in. And in about a fifth of the paid search cases, we don’t. So what does that tell us about “Direct” traffic?

It should tell us that Direct traffic is partially composed of search engine driven traffic, that is missing a referrer header. Some of it will be paid search, some organic search and some from other resources that are potentially trackable.

In other words, there may be more than 50% more clicks that should be attributable to organic search, than are showing in normal web measurements. And Direct is over-represented as a consequence of the way that data is collected.

If you have paid search clicks being tagged and tracked, this means that there is a systematic data error, in which clicks are allocated to Direct, when they come from unknown searches on unidentified organic search engine results.

Summary

Watch out for over-allocation to “Direct” as a consequence of missing or misleading information fed by Web Browsers to web servers.

You should be able to use paid search data to estimate the likely misallocation of clicks to Direct when they should be organic search, and even to estimate the likely frequency for higher volume keywords.

AFAICS, most web measurement and analysis services do not compensate for missing “referer” fields - they don’t even directly report on the number of missing referrer fields in attributed clicks, making the estimation of misattributed clicks hard.