Macros, Analytics, Paid Search Performance Improvement

Published on April 3rd, 2007 by Jeremy Chatfield

How do you tell whether you have actually received visitors from AdWords? You need to use web analytics. There’s two problems with that solution - most analytics are completely useless for explaining what happened with your paid search, and secondly, the search engines only give you part of the information they collect about the adverts you pay for… making it hard to refine your paid search, without being an expert in paid search as well as your professional or vocational expertise.

This article very briefly covers what you can do now with Google’s own Google Analytics and AdWords, and more extensively describes what you can do with third party web analytics and web server log file analysis. I also note what can not be done, unless Google modify AdWords to provide for more useful information for paid search management. I have not covered “missing macros” in much depth nor have I done an in-depth look at how the presence and absence of various macros affects the costs of using the Google AdWords API. However, I’m quite certain that a better set of tag substitutions could both reduce the cost of using the API and improve the performance of advertisers.

AdWords Clicks and Natural Search Clicks

Free and paid clicks look the same, unless you do something about it. That is, if I’m looking at a web server log file, I usually can’t tell whether a visitor has come from paid search or from organic search. Most of the clicks look just the same - the big exception being content match, where, if you do get a “referer_info” field, it may show the click as coming from Google’s PageAd Syndication server.

This missing information about the origin of the clicks is a frequent cause of confusion for new advertisers, who often complain that they’ve enabled AdWords but their analytics package continues to show visitors from Google, AOL, etc. They are perfectly correct - that’s just where the paid search visitors will come from. Google offers adverts on those sorts of sites, so you will see clicks from those sites, but they’ll usually look just like a normal search.

You need a way to make the paid adverts stand out as different from the organic search clicks. The only way that I know of, to make the click source obvious, is to change the destination URL so that the Analytics package can see that this is a special click. There’s three main ways to do this:

• Use a “redirection service” which counts clicks and won’t see visitors from organic search
• Use “tags” on the destination URL, so that the analytics package can see that this isn’t from organic search
• Use a special landing page URL - so anyone landing on it must be coming from paid search

Annoyingly or perhaps fortunately, there is no common standard amongst web analytics vendors for the tags to add. Here’s a few examples of the types of tags that you might add. Note that you’ll need to consult your documentation for your package to see what tags you have to add:

• &cm_mmc=foo-_-bar-_-stuff-_-here
• &utm_content={creative}&utm_media={ifsearch:ppc}{ifcontent:content}…
• &ns_campaign=foo&ns_url=http://merjis.com/

It’s probably fortunate that the web analytics companies haven’t agreed on a common standard, because almost without exception the standard tags they offer for paid search use are insufficient to perform an adequate analysis for a large account and might even run into trouble for a single product offered in a geographically small area.

Close Linked Google Analytics

If you use Google Analytics, then you can enable Google AdWords’ Auto-Tagging and you can get some information about the most popular keywords that lead to the site. Rather more usefully, you can also get some information about the search queries. You want to know about the search queries, because you usually want a close match of keyword and query, and that will also help you to tune the advert and landing page, increasing conversion rates.

Auto-Tagging appears to allow Google to identify much more information than you can get from web server log files. Potentially, you should be able to identify which keywords and match types are most likely to lead to a sale, though identifying which referral sites are least likely to offer a converting user is a slighly harder exercise.

What Auto-Tagging Does

Looking in web server log files, we see that Auto Tagging has enabled a new field, gclid, added magically by AdWords, to the URL. This can cause some web servers to mis-serve pages. Don’t just enable Auto-Tagging and hope for the best.

Warning! Google has, so far, implemented this so that it only adds the tag to real adverts. If you click on the advert from within the AdWords User Interface, the auto-tag isn’t added and the test may prove successful, when real use will show a problem. Even worse, this implementation choice means that Google’s automation will look at the URL without the tag, identifying the site as working properly, but when users click on a real advert they might get a 404 or 500 error from the web server. You really, really don’t want to be paying for clicks that give users a “No Such Page” or “Internal Server Error” message - that’s a poor brand experience and a waste of money.

There’s a deeper issue with the auto-tag, though. It only offers information to Google Analytics. The data can’t be interpreted by third party analytics products. This means that you have to either find a way to make Google Analytics deliver the right information, or to add more information in the normal URL so that a web analytics package can offer useful insights. We have some clients who refuse to use Google Analytics and Google Conversion Tracking, because they already have so many web analytical packages installed (we often see large accounts with at least two page tracking systems and at least a half dozen order tracking systems), or because they don’t want Google to see information about their visitor volumes and sales.

Weaknesses of Google Analytics For Paid Search

GA has some significant problems, especially for those new to paid search and analytics. For example, it is quite common to pay for 250 clicks from adverts, but for GA to offer a Search Marketing analysis showing that only 80 visitors arrived from paid search adverts. That’s with auto-tagging enabled. Explaining the disparity can involve some serious handwaving, but usually only becomes evident when you look at the web server log files.

Perhaps worse, the analysis can’t be exported via an API, and the irregular lag on data makes calculating new bids or dynamically managing fast paced advertising, an impossibility.

What Can You Find Out Now?

There are always going to be things that you don’t understand, but a lot of what happens can be understood if you know the nature of the search, and the context of the search. That means that not only do you want to carry information that this click came from paid search, you also want to bring any other information about the search that you might think is relevant to improving success.

Google AdWords offers some macros - pieces of text that are replaced with real values when the advert is delivered. The known list is pretty small. It is possible that Google has made more of these available, but hasn’t documented them in any useful form. The bulk of these macros were found in a misnamed PDF from a 2003 Google presentation about optimising AdWords.

{keyword}

As well as being useful in an advert, when this is known as Dynamic Keyword Insertion, you can use {keyword} in a tag. Used like “&utm_keyword={keyword}” it will substitute the keyword that triggered the match, without the match type.

This is helpful, but not complete. In order to check on Match Fraud - a sub-genus of Distribution Fraud - you also want to know what the search query was. By cross checking the search query and the keyword, you can see when the search engine is becoming excessively eager to deliver clicks and offer your advert to inappropriate queries.

Additionally, there are three main ways to build an AdGroup (there’s lots of special ways for specific purposes, but, IMO, they can be generated from these basic types):

• Use Broad Match and improve CTR with negative keywords
• Use Phrase and Exact Match, possibly with negative keywords to improve phrase match CTR’s
• Use all three match types, and bid them to make sure that the right match goes to the right keyword and match type

So you REALLY want to know what the match type is.

There is a way to find out, but it requires intensive personal commitment (read that statement as “you do a lot of typing”) or you need some automation - automation that the AdWords API will make you pay for twice (once to develop the software and once as a tax by Google on each use). You need to change the Destination URL for each keyword. That way, you can have keywords with a unique destination URL including a tag like “&keyword={keyword}/broad” or “&keyword={keyword}/phrase”.

Having a Keyword Level Destination URL also presents another problem for the advanced marketeer… A/B testing of Landing Pages. If you want to test two pages to see what the conversion is like, you could use two adverts, with identical copy but different destination URLs. You can then look at the conversion performance to see what it is for each advert and from that, infer the better landing page. You can’t do that when each keyword specifies the Landing Page. If the keyword requires a destination URL because you want to know the match type, then the advert Destination UR is ignored… and you don’t get A/B testing, you only end up on the Landing Page embedded in the keyword destination URL - removing the A/B test.

It’d all be a lot easier if Google added {matchtype} as a macro. Then you could have “&keyword={keyword}/{matchtype}” in the advert, no need for a keyword level destination URL and you can then use identical adverts to send traffic to two or more different landing pages, with full URL tracking of the behaviour. In the interim, if you want to use A/B testing then you are looking at a variety of different mechanisms, each one of which will drive up the costs of using the API and complicates the statistical analysis. The various techniques depend on what you are trying to achieve and the damage to search performance that you are willing to allow, so I’ll not drill into the details here - perhaps another posting, sometime.

{placement}

So far as I can discover, this tag only works on site targeted advertising. It is replaced by the URL of the site on which the advert is shown. Use it like “&site={placement}” to find out where that site targeted click came from.

In one audit, we found a client with 20,000 clicks from the content network of which 30% had no referrer_info data. It was impossible to use these 6,500 clicks to refine the site exclusion list. How much more useful if Google were kind enough to let the advertiser know the publisher ID and the site name, such that it could be used in a site exclusion statement directly? An apparently small change to the advert interface, and we’d all get a lot more information, that would let us better control costs. So, for content match, it’d be good to have “&site={placement}” work, and offer the name of the site that Google recognises as the publisher.

{ifsearch} and {ifcontent}

These macros are used for conditional text replacement if the statement is true. In other words, using these you can pass back to the analytics package whether the click was from keyword search or content match. Quite useful. Example - “&utm_medium={ifcontent:ppc}{ifcontent:content}” becomes “&utm_medium=ppc” or “&utm_medium=content”, depending on where the advert was served.

Slightly less usefully, Google makes no commitment to allowing nesting macros. So, for example, you couldn’t say (or not with any commitment from Google that if this works, it will continue to work) -

&{ifcontent:site}{ifsearch:keyword}={ifcontent:{placement}}{ifsearch:{keyword}}

Since the {placement} macro doesn’t work in any advert that may be shown with keywords, at present, this is a somewhat spurious example, but that’s never prevented me from considering what you could do with a system, if it was slightly enhanced.

{creative}

Brilliantly useful, especially when trying to do A/B testing, {creative} is replaced by the advert ID. You can then use this to test, for example, which adverts have the better conversion rates. Use in tags like “&utm_content={creative}” and your web server log files or analytcs packages can learn which advert causes user activity.

It’s a little awkward to associate the Advert with the magic number that the {creative} macro returns. There are some reports (look at an Advert Report in the AdWords interface) that will associate the text of an advert with the AdID. When you’ve got that report, then the analytics makes more sense. I don’t know of any current web analytic system that shows you the multiple adverts for an A/B next to the conversion data.

Of course, this ties back to A/B testing. And again, the best use of {creative} will happen when the Match Type is also a macro in the Adverts’ Destination URL.

Summary

Current web analytics packages are largely inadequate for measuring response from paid search. Tags added to the destination URL, or redirection servers, are essential to identify users coming from paid search - otherwise they become lost in the organic search traffic.

Imaginative use of the Destination URL allows advertisers to better identify which clicks come from paid search and which are from other sources. Google’s macros allow keywords, distribution system (search or content matching), creative and the site placement to be identified. The current system doesn’t make measurement as easy as it could be, and tweaks to gain more information can result in significant increases of AdWords API costs.