GAP – Is It Worth It?

Published on March 16th, 2008 by Jeremy Chatfield

About six months ago, I re-sat my Google Accredited Professional exam. I took the first one just a few months after the program was started, in 2005. Having taken the exam twice, I’m unhappy. I wouldn’t trust a Search Marketer with only a GAP, to do the right things. This is partly because the exam requires answers that benefit Google, but it also fails to address the current state of AdWords, and misses out on techniques and tools that I use, if not daily, then at least monthly. Surely you’d want someone that knew the latest version of AdWords and the most up to date tools and techniques, not the details of some previous version that used to exist and whose features are no longer relevant? Wouldn’t you want someone skilled in diagnostics, who can spot the errors at Google’s end that cripple your current efforts?

As well as “guiding” GAPs to provide Google recommended solutions, the exam questions also show a lot of subtle defects in understanding. Some of these would actually damage AdWords accounts if you believed and used them.

Passing the GAP therefore means that you have to answer not as you should do, but as you think Google would like you to answer. I’m deeply unhappy about this sort of test, and I have some reasons, and some career qualifications, to comment. For several years I worked as an academic, part of whose duties included setting questions, marking the answers and sitting on Exam boards for University degree examinations at the premier distance teaching organisation, the Open University. I still remember quite a bit about the difficulties of writing a good Computer Marked Assignment (a CMA, back in the 80’s). For example, what problems would cause a question to be zero weighted – and the acute embarrassment of offering questions that were weak and needed to be zero-weighted, if they managed to sneak through the review process.

As an ex-educator, I’m deeply unhappy about company sponsored tests. In general, it is a bit uncomfortable to take company specific tests – because there is no third party normalisation, known in UK academic circles as “moderation”, to make sure that the test is fair, rather than an inculcation to the inner brotherhood or sisterhood. I’d be happier if the test questions were adjudicated by a panel of *non-Google staff* search marketers. Google can propose at least some of the questions and answers, but the panel would control whether the question was permitted, and the weight of the answer. In particular, if I was an advertiser, I’d like to know that someone working on my behalf was not a covert extension of Google – delivering only Google approved answers, that might not be in my best interests.

Anyway, after I completed the most recent exam, I wrote a twenty-something page report to Google, with screen shots of dubious questions and a commentary on each of the them. I’ve given Google almost six months to reply to the detailed objections that I raised. They have not given any specific responses to my carefully documented examples – one email that said “thanks, I’ve passed it on”. In line with the spirit of Information Security analysts, it is time to publish the errors. Google has had time to fix the problems. If they are sufficiently discourteous to fail to respond properly, that’s their problem :)

If Google consequently want to withdraw the GAP that I have, to punish me for exposing bad questions and incorrect answers, I’m happy. At the moment, I think the GAP is not a good indication of expertise. It is an indication of brainwashing and the ability to answer the way that Google wants you to answer. It is not a qualification that endorses someone to manage a client account to maximum advantage *for the client*, neither is it proof that you know how the current version of AdWords works.

I’ll not reveal all the problems, all at once – it’d make this article at least twenty dense pages long :)

I’m also suppressing the entire question and answer, where I can, in order to avoid hand holding people through the current exam. The graphic typically illustrates the question, but may instead illustrate a set of bad answers – like the first example, below.

Since it is about six months since I took the exam, Google may have improved it. But I do recall some of the questions from the first time that I took the exam, and some of those re-appear here. Google seems to revise the exam much too slowly to be a useful indicator of competence – even for your own confidence of your skills.

The count of questions with dubious answers is important. The sheer volume of awkward questions means that in order to pass the Google-set threshold, you probably have answer at least some of these the way that Google wants – or you have to score perfectly on all the other questions. I’m pretty sure then, that most GAPs have either compromised what they know, or don’t know enough to be let loose :)

A Technical Error

The following image illustrates a radio box, which had four items. The candidate is supposed to select one answer out of the four offered. None of the selections had a label, as you can see from the two answers snipped below.

The fourth option asks you to infer which is answer b and which is answer c. There is no option b or c labelled in the answers. This is just a petty trivial administrative issue, but if I was sitting on the Exam Board back at the University, we’d be considering zero-weighting the question… Why? It was an early question in the exam, increasing stress on the candidate, forcing them to make an inference that is not tangibly tied to performance as a search marketer. It is sloppy work by the examining body. Someone failing to work out which was “b” and “c’ might be a perfectly fine candidate, who was flustered by the pressure. I’d encountered a few problems before this question, but the sheer sloppiness of this answer made me realise that I wanted to record the experience.

Question With Multiple Answers

The following question is a problem, because the expected answer would have been correct a few years ago. However, the system has evolved, and now several answers are correct. Worse, there are new additional answers, provoked by these changes, that aren’t represented in the set of allowed responses.

The answers included two correct responses. However, when the question was originally written, only one of the answers was right. Since then, the system has become more complex, and I can think of at least three correct responses – one of which wasn’t shown at all. The answer that is true in 2007, but was not true in 2005 is:

Increasing your Maximum CPC may cause you to reach your daily budget more quickly causing your ads to stop appearing for the day.

This used not to be true – although many people falsely believed it was true. So it used to be a great distracting answer to separate out the clueless. However, Google introduced a new option in the “Edit Setting” page for campaigns, that offers accelerated spending – burn the budget as soon as possible. So, after that option was added, two answers became possibly correct. But the question was left with only permitting a single answer, when two can be correct.

Additionally, the Display URL also interacts with the Maximum CPC. For example, when affiliates are handling paid search for a business, the highest value advert will be used (subject to the account history). So the way to appear on a keyword, when a competing affiliate is present, is to bid (and pay) more. This means that there is a common case for a correct answer (lots of affiliates use AdWords), where a known “correct” answer, is not included in the list. This is damaging to the credibility of the test.

Finally, one of the distracting answers had a probable typo – “Your ad will not appear if our” should probably be “Your ad will not appear if your”. It makes more sense as a distracting answer that way.

Answer lists that don’t cover all the cases, and allow only a single answer when there may be multiple correct responses, are not a good question for determining the effectiveness of a paid search specialist. This question and the answers should be retired and replaced by one that works.

Any GAP selecting a single answer for this question, with its simple list of answers, is doing so to gain the points to pass, not demonstrating that they deeply understand AdWords. That’s just ethically and morally wrong.

Question With An Assumed Answer

The following question assumes that there is a single best strategy.

However, the maximum profit is achieved differently by different sizes of operation and the types of product being sold. Fast paced inventory, with frequent price changes, and search volumes measured in hundred thousand clicks per day, may need very different strategies from someone satisfied with a few tens of clicks per day, and a relatively stable business.

Also, none of the solutions offered the use of “categories” and “comparison” oriented advertising, which I might want to use. This would be influenced by the business’ web site content and ability to support those sorts of advert – and the conversion rate, because these visitors may be fairly early in the buying process.

It is just ignorance of the uses of AdWords that would lead to suggesting that a single answer satisfies all needs. I often use option E, or F – in other words, an option that wasn’t offered. :)

The language of the question is also a problem. Asking for “effective” throws an invisible ball into the air, for the candidate to catch. Why? Because the answer depends on spend and profit levels. If you were doing this for a large organisation, you might be able to justify the savings to pay for the effort, and you might find it effective to hire a new staff member to do complex things. But if a very small business, complex arrangements should be avoided to get the most efficient spending, despite some suboptimal compromises, to save management costs. It is not “efficient” to spend more than the campaign monthly budget to set up a complex offering – something simpler, easier to set up and manage may be more “effective”, for the right business.

This question could be re-used, with some re-wording. For example “Which of the following strategies would be best for…” and then explain the scenario. “What is the best strategy…” may result in the candidate identifying a solution that beats the ones offered, and so failing to choose any of the offered options, or choosing one that best matches their solution but is not preferred by Google.

In general, questions should be written so that the judgement exercised is on the task you are trying to measure, not the ability to infer what the question might mean. Unless you are trying to test comprehension skills, in which case there are better questions than this.

Summary… So far

Google needs to carefully review their questions and answers. Some of the questions are left over from previous versions of AdWords, and what were once correct and unambiguous answers are no longer correct, or no longer uniquely correct.

Some answers show a limited understanding of how to use AdWords, particularly how tactical usage of AdWords varies between different types of advertiser and the size of the business. Selecting the “right” answer may mean inferring or guessing the assumptions that Google used when they wrote the question.

For advertisers looking to hire a GAP: A GAP qualification is better than nothing – but not enough to assure a client that they are getting best practice from someone who is working in the clients’ best interests.

For AdWords users attempting to demonstrate their competence: I’d rather see you with a CIM (Chartered Institute of Marketing) or IDM (Institute of Direct Marketing) qualification. Knowledge of marketing communications (poorly tested in this exam) and the phases of the buying process (completely untested in the exam) are, IMO, very important. So is a good numerical and statistical background – only tangentially tested. I can teach the essential tools and techniques for a marketing-qualified individual in a day, though getting the subtlety out might require some mentoring or an advanced lesson of a few hours, later.

Only another couple of dozen defects in the exam, to come…

Updates

Second part of the GAP Exam Questionable Questions has been published.