However, the rate of goal completion is only one of the factors involved in optimising a web site. If a source has a 25% goal completion rate, but only brings in four visitors in the measurement period – there may be reasons to suspect that the single conversion might be less important than the half dozen conversions from a lower converting source that sends a lot more visitors. The likely maximum number of visitors from a source is important – so understanding all sources of conversion is essential.

Google Analytics has recently added a “weighting” switch to the Goal Measurement page. I was hoping that this might then weight according to how many conversions were involved. Instead, it does something quite mystifying. The source with the largest count of conversions is listed *last* in the sort order, below dozens of sites with no conversions at all. I can’t imagine why a positive count of conversions sorts after sites with zero conversions.

Here’s an example of the same data, without and with the weighting.

Why the largest volume source of conversions is weighted so that it doesn’t show up at all… I have no idea:

Notice that with the weighting enabled, the 5,000 visitor source just vanishes from the listing altogether. Useful though the idea of weighting the converting sources is, if you have a lot of sources, watch out – you might inadvertently lose track of some valuable sources if you simply enable this switch without paying attention.

I don’t remember seeing this addressed in any of the web analytics blogs and books that I’ve read for the last few years. It’s possible I missed something, and I’ll gladly add references to those articles, if anyone contributes them in comments here, or in emails to me.

Observation

Like much of good science, this starts with an observation:

When you look at the sources for which you have control of the requested URL, a substantial fraction have no referring info.

To understand what that observation means, we need to know a bit more about how visitors arrive, and what sources of data we have about the visitor.

Page Requests

Paid search, banner advertising, email and so on, allow you control the page request offered by users. You can add tracking parameters, and that lets your web measurement system collect that the request came from a particular source. That’s pretty important for click fraud measurement. If you aren’t adding tracking tags, then the visitor could be turning up from anywhere for any reason.

For example, lets’ assume that you are advertising for an Organic Cosmetics business. You place an advert on a keyword offering “organic acne treatment”. You bring people to a page on the site, for that treatment – http://www.organic acne treatment.com /our-products . You *MUST* add a tracking tag, or use some kind of autotagging (offered by AdWords, Yahoo!Search Marketing, etc) so that clicks from advertising can be measured by your web analytics service, and attributed to your paid search campaign.

What the tag looks like, will depend on your Web Analytics package. If you use Google Analytics, you might have tags like “?utm_media=ppc&utm_source=yahoo&utm_creative=organic+cosmetics”. So the requested page will be something like:

http://www.organic acne treatment.com /our-products?utm_media=ppc&utm_source=yahoo&utm_creative=organic+cosmetics

Without that, all you know is that someone requested that page… or is it?

And, of course, knowing that the click arrived as a result of an activity where *you* control the page request (typical of paid advertising opportunities), means that you now have a handle on part of the click fraud questions.

Referer (sic)

Something else that a web browser can optionally tell the web server, is known as the “referer”, in RFC2616 about the HTTP protocol (See Part 14, on Headers). This “referer” header sent from your web browser to the web server, says what the page request was, for the page *before* the request to the server. So we might see that the request for our page in the example, was referred to by “http://www google com/search?q=organic+acne+treatment” (and there’s usually some other stuff in there, describing the language and the browser and so on). So we know that a search engine was the source, and we know the keyword.

However, we don’t *know* that this click was from paid search. A common frustration amongst new advertisers is failing to identify clicks from AdWords, because advert tracking tags have not been added. Without the tracking parameters, all you have to work from is the “referer” header volunteered by the browser – and that only tells you that the user was looking at a page, not whether they clicked a free or a paid link.

So, one common misattribution is to *over-allocate* clicks to organic search and *under-allocate* to paid search, because neither tracking tags are available, nor is there any common way to tell from the “referer” field, that the source was paid search. I’m going to ignore the further complexities of referrer headers when using contextual advertising – it is much more complex :)

Tying the Pieces Together

Assume that we now have tagged clicks, so we can tell the paid sources that are sending traffic. We have the paid source telling us what they claim. We can match the two pieces of data. If they don’t match, then we have some further questions about the sources of mismatch. I’m not going into that, in this article. I’m going to take a sideways look at that other data-stream, the “referer”.

When you have a good source of paid clicks, one that you can trust as delivering a high fraction of the clicks they claim, and where we get good conversion rates and matching keywords and search queries, we can infer that the people being sent are are also “good” – no or few spoofing robots, no or few paid clickers or fraudsters, etc.

If we now look at the “referer”, we should see that all the visitors come from a page with paid search on it, shouldn’t we?

We don’t. Anywhere from 10% to more than 40% of the clicks have no referer_info at all, or have some clearly dummy data inserted instead of a real referrer.

Attribution Of Origin

So if we have no tracking parameters, we can only know what the origin is, if we have the “referer” field correctly filled in. And in about a fifth of the paid search cases, we don’t. So what does that tell us about “Direct” traffic?

It should tell us that Direct traffic is partially composed of search engine driven traffic, that is missing a referrer header. Some of it will be paid search, some organic search and some from other resources that are potentially trackable.

In other words, there may be more than 50% more clicks that should be attributable to organic search, than are showing in normal web measurements. And Direct is over-represented as a consequence of the way that data is collected.

If you have paid search clicks being tagged and tracked, this means that there is a systematic data error, in which clicks are allocated to Direct, when they come from unknown searches on unidentified organic search engine results.

Summary

Watch out for over-allocation to “Direct” as a consequence of missing or misleading information fed by Web Browsers to web servers.

You should be able to use paid search data to estimate the likely misallocation of clicks to Direct when they should be organic search, and even to estimate the likely frequency for higher volume keywords.

AFAICS, most web measurement and analysis services do not compensate for missing “referer” fields – they don’t even directly report on the number of missing referrer fields in attributed clicks, making the estimation of misattributed clicks hard.

Rght now, it is 01:27 AM BST (that’s 00:27 Universal Time) or 8:27 pm Eastern or 5:27 pm Pacific. It’s also July 14 for me, and July 13 for the US. So take a look at this screen shot:

If I select “Today” then the User Interface shows the date as the 14th July and the 13th July. The stats reported are updating for the 13th July, but the graphing tool is showing “Today” as the 14th. There are no clicks in reports or in the UI, for July 14th, only for July 13th. It’ll become even more complicated in a few hours, when the East Coast crosses midnight. Use the wrong mix of Ad Scheduling times and you may never work out exactly when the adverts are actually running, as Google becomes confused about which of the three possible time zones is being referenced, and which tool is being set to use which TZ.

Reporting Delays

Complicating this even further is Google’s little note in faded grey at the bottom of the stats table:

Yup – “clicks and impressions in the last three hours” and “24 hour delay in conversion reporting”.

The good news is that the delays aren’t usually that much. That chart above shows some conversions. It’s probably close enough to right. AFAICS, the stats *USUALLY* take about 45 minutes to stabilise. There’s some delays that can cause the stats to lag for longer, and, more importantly for many users, the numbers may decrease over three hours and even for a matter of months later. That’s because the invalid impression and invalid click detection software (click fraud detection) takes a while to work out that a user has a pattern suggestive of suspicious behaviour. The UI and reporting tools do not report clicks, but *billable* clicks. That means you can see a click, which is later withdrawn. In extreme cases you may see a conversion with neither impression or click – someone, identified by Google as a likely fraudulent user, has purchased a product but their impressions and clicks have been ignored.

So, if you believe anything, believe that the latest updating stats are substantially correct for billing purposes as of about 45 minutes ago. Definitely don’t go adjusting your bids as if the stats were being reported in real time… And it becomes more complex yet when you consider the delays in reporting stats in Google Analytics. A topic for some other time, perhaps. :)

Summary

If you think that you’re confused about when something is happening with when things appear to occur in AdWords, it sometimes appears to confuse and puzzle Google, too. Be aware that you may know more about what time events happened locally, than Google does.

Measurement

If you don’t have website statistics now, ask your website management company, or your staff, to install the free “Google Analytics” package. There are many free packages and you may already have one, but a big advantage of Google Analytics is that it integrates well with Google’s advertising systems and reasonably well with many other advertising systems.

Make sure to ask that “error pages” – the “404 page” and any “500 pages” – also have the Analytics code on them. You should investigate and repair your website urgently, or your advertising, if large numbers of visitors reach these dead and useless pages.

A big weakness of Google Analytics is that it doesn’t do quite the right things, until it has been configured. Configuring it can be difficult for beginners – so here’s a good book with detailed configuration guidance, for technically oriented staff – Brian Clifton’s “Advanced Web Metrics with Google Analytics“. If you are non-technical, I suggest that you flip through the book, note the types of information you can get with postits on each interesting page, then ask your technical staff to implement the techniques on the marked pages.

As a minimum, you should ask for Google Analytics “profiles” that give you “Error Pages”, “New Visitors” and “Returning Visitors”, with “filters” that eliminate traffic from your own staff, and ignore traffic on any sites that have stolen your web pages without changing the Analytics code.

If you expect to use AdWords or other paid sources of traffic, you should have each of those paid sources of visitors broken into at least two separate profiles for each – for example “AdWords Visitors – New” and “AdWords Visitors – Returning”.

You can also ask your staff for reports to be automatically emailed to you weekly, so you don’t have to login and work out how to get what you want. There’s also a free iGoogle Analytics “plugin” so you can have stats on your Google Home Page – if you use that.

Understanding Visitors and Your Site

You should be able to answer the following questions. If not, you or your staff probably need to develop an understanding of how your website and your analytics works.

• How many people visit your site? Per day, per week and per month? And when, during the day, do they visit?
• How do new users find your site – are they coming from directories or from search engines or other sites?
• What search queries are visitors asking the search engines?
• Are visitors bouncing – leaving the site as soon as they arrive – in large numbers?
• How long do visitors stay on the site?
• What are the most and least popular pages with users – the ones most and least visited and the ones that users stay on longest and shortest? Is it what you expected?

You should be getting most new visitors from search engines – is that happening?

You probably should be getting return visits from customers – are you getting returning visitors?

If the pages for your best products and services aren’t getting visitor attention – why not?

See if you can improve your performance, by making changes and watching the statistics.

Customer Engagement

Think about the searches that prospective customers should be using to find your business. When you talk to them directly, what do they ask you? Not in your professional jargon, but with their use of language?

Now go to the “AdWords keyword ideas” tool. Put in your user’s important phrases and ask for suggestions. These are usually words and phrases that your site should be using and may even be used as the names or titles of web pages on your site. If you use language differently from your likely clients, you are less likely to attract them to your site and keep them engaged with your offers.

When you have a conversation with a client, lead or prospect, think about whether your website clearly describes the same things in the same way. If your website message is different from what you tell people, try to make the web site reflect what you are doing and saying.

If someone lands on the web page for a product or service, without going through the home page, would they know what you are offering and how to get hold of you? If you have a Unique Selling Point on the website – is it only shown on the home page? People who use search typically jump deep into your website, and will miss messages that appear only on the home page. So if you offer free shipping, or a free first order, or whatever it is – make sure that offer is visible on all likely landing pages.

Make sure that you have a call to action on each solution page, with prominent contact information. We’ve come across a surprising number of sites without phone numbers, inquiry forms or addresses… At the point when visitors should have decided they need to talk with you or place an order – make sure that they can do so. A rough rule is that “users will do what you ask them to” – so ask them to buy.

Before you make changes to the web site, here’s one more review step… does your website tell customers about you, or do you talk about them and their problems? Your website should demonstrate that your first priority is solving your customer’s problems. A swirling-in logo of your company, or the letters of the company name coalescing from a cloud, may look pretty neat, but does it answer a prospects’ first question – “do these guys have anything that solves my problem?” Focus on what your prospects need to know to become clients, and don’t delude yourself that they want to sit through the same 40 second monologue from you at the start of each visit, going on about how great your business is… They won’t like it. Really.

Get to the point quickly, and describe the problems that your customers used to have, and what you do to fix them. If you solve several problems, offer one web page for each, and use the home page to reference the different solutions. Each solution page should link to related pages, so if visitors aren’t in the right place, they can easily find where they should be.

Being Found – Search Engines and Other Visitor Sources

There are two main ways to appear in search engines. For the main search results, you use “Search Engine Optimisation”. Do read free advice from major search engines about selecting a SEO business before you pay for services.

• Google’s SEO advice
• Yahoo SME – SEO advice

Paid search advertising is the alternative. Smaller businesses shouldn’t just pick the market leader, Google AdWords, but consider the smaller players, MSN AdCenter and Yahoo!Search Marketing. There’s usually less competition, and the smaller audience doesn’t make much difference to the volume of visitors when the budget is small. They also attract slightly different audiences – so failure on one of them doesn’t mean they’ll all fail.

The AdWords Help Experts blog has a video showing how to set up advertising on Google AdWords. Do *NOT* use Google’s “Starter Edition” – it is poorly designed for new advertisers. Exercise great care – Google does not help you to control costs and improve ROI – their eye is definitely on their revenue, not yours. Do not expect to carry to AdWords, the same trust that you have in Google’s search engine, or you will lose money.

SEO is slower to return results, but often at a lower cost. If your website is pure Flash, gaining visitors through SEO will be harder and more expensive. Paid search advertising is faster, and it is easier to identify the return on your investment. We recommend that if you can, use Paid Search first, to quickly find which keywords & pages work, and then use SEO to target them.

You can also appear in business listings, such as Google Maps. If you sell products, you can list them, for free, with Google Product Search. You may have images and pictures that interest people, and you can get listed for those. You may make videos and publicise those for Video search. There are planty of ways for a little imagination and creativity to get your your business noticed.

• Qype – expanding worldwide, offers reviews of local resources and interaction with local and interested members of the community. See the Merjis listing showing that we offer internet marketing in Bedford. Accepts adverts, too.
• Gumtree – classified listing directory, accepts adverts
• Craigslist – massively popular classifieds
• eBay Stores – if you have products to sell, eBay has buyers.
• Yahoo Directory – paid listing, human moderated, the original large scale internet directory
• Open Directory Project – free listing, human moderated and a contributor to the Google Directory.
• LinkedIn – link to professional colleagues – see my profile for example
• Twitter – status feeds for customers, personal conversations, all sorts, Twitter is flexible
• FaceBook
• MySpace

Do *NOT* pay for “Search Engine Submission” tools – if your site is properly linked to the rest of the web using the services and sites that I’ve given you, the major search engines will find you anyway. Do *NOT* list your site on free and unmoderated directory.

Internal Response

Leads received from the website are often left in email inboxes for a few days. Web leads are as urgent as a phone call. Make sure you and your staff respond promptly.

Finally, you probably have news about your business sector. Can you write it up, even if in only a few short sentences? A business blog with short, or even long articles like this one, can provide a way to communicate with new leads to show them how you help your clients, and it helps your SEO efforts. There’s lots of free blogging software that you could host on your own site, or you can use one of the public blogging services.

About Us

The author, Jeremy Chatfield, is a Google-recognised Top Contributor in the AdWords Help Forum and manages the AdWords Help Experts blog; he is a Google AdWords Qualified Individual. Merjis, in Bedford i-Lab (+44 1234 834760), helps clients to improve their web site performance.

Material Disclosures

The link to the Google Analytics book above is an “affiliate” link. If you click and buy, we may make a small amount of money. It’s a demonstration of another common technique for attracting visitors (using “affiliate programs”) and may be a way for you to additionally monetise your site.

None of the other links are affiliate links, even when an affiliate program exists for that organisation.

This article is an expanded version of an advertorial in our local business newspapers, “Business to Business” and “Business MK“.

Thanks to “bizwriter” on Twitter for drawing this article to my attention.

The question of excluding your own IP address came up on the new Google AdWords Help forum today, and I realised that I didn’t know where I could find a series of screen clips showing the bits to click on. Here they are. :)

If you access Google Analytics from inside AdWords, the coloured bar in the clips will be AdWords Green, not Analytics Orange.

Log in to Google Analytics, or select the Analytics tab in your linked AdWords account. If you see a graph of results, click the “Analytics Settings” link in the top right, to get to the “Analytics Settings” page, with a list of your profiles.

On the far right of the Analytics Settings page, click on the “edit” link for the profile you want to change.

On the Settings page, look for the “Filters Applied to Profile” section.

Click on “Add Filter” to the far right of that bar.

Select the “Exclude All Traffic From an IP Address” drop down. I’ve censored my IP address, but you can still see the “\.” between numbers, *NOT* just a dot.

Insert your own IP address with a backslash in front of every “dot” in the address

Click “save changes”.

That’s it.

When Will You See Changes?

Note that the exclusion *ONLY* happens with effect from “now”. That is, Google Analytics won’t go back in history and remove previous activity from your site. Only stuff you at look at from today onwards will exclude your own traffic.

Viewing ONLY your own traffic

Of course, you could create another profile for your site, and use it to monitor *only* access from your IP address – so you can see what your own company users are interested in, on your site. Just use a custom filter to only include visits from your own IP address.

I think I’ve worked out one of the reasons why you might see this misleading message.

I’m pretty sure that it is a bug.

Ad Scheduling Symptoms

I have a few clients who have Ad Scheduling enabled. Ad Scheduling, or Day Parting, allows tailoring delivery of adverts to the time of day when customers are most in the mood to buy. It’s another targeting mechanism in AdWords.

You can choose, once in the lifetime of an AdWords Account, the Time Zone. By default, Google picks their home town Time Zone – Pacific, some 8 hours behind me in the UK.

Take a look at this Google Docs Spreadsheet, which shows the effect of some of the Ad Scheduling for an account. Think “hosting service maintenance” or “shift change in the call centre” or something like that – late in the evening, Mountain time. Mid evening, Pacific Time, or in the wee small, and unimportant, hours in the UK.

The first evening scheduled period starts at 08:30 Pacific time until 09:30 Pacific. Advertising runs just fine. Then, even though advertising *should* run from 10:30 Pacific to 3:00 am Pacific, advertising doesn’t actually resume until we start a new *Google* Budget day – at Midnight, Google Time. In the interim, although there is unspent budget, cash in the account and a desire to spend, Google will insist that the account has reached the spending limit.

Google staffers believe that the delay in serving is caused by some kind of editorial review. I’m moderately expert in editorial reviews, about as much as someone who has never worked at Google, and who has been careful to avoid questioning Google about Editorial Review when under a Non-Disclosure Agreement, can be. There is probably more information available under NDA, but then I wouldn’t know about that :)

This is nothing like any other review process. I’ve never heard of or come across a *daily* regularly scheduled review. Doesn’t mean that there isn’t one, but it seems like a very strange way to run a business. Editorial review, especially human mediated, is expensive. If you were automating one, then there’d be signs – like visits form the AdsBot. I have measurement tools in place, so I can look for visits by the AdsBot (I’m looking for user behaviour, to optimise adverts and landing pages – but I can also see the AdsBot, too). There are no coordinated visits from the AdsBot at the point at which advertising fails to run when it should.

So that means if it is a review process, it is human. I don’t believe that Google have a staff member assigned to suspending and reviewing an unchanged account at the same point every day, and enabling it at precisely midnight Pacific time. That would be stupid. Google isn’t stupid.

But not being stupid and not making mistakes are two different things. Smart people can make mistakes. You can learn a lot from a well judged mistake.

This looks entirely like a bug. A bug in which a late re-enabling of Ad Scheduling, has Google looking at the budget and going… “No – this budget, in 24 hours, set to accelerated spend? It’s gone already… Nothing to see, move along”. At midnight, Pacific, Google goes “so, what is on accelerated spend, and scheduled to run? Ah ha!” and you start getting Ad Flow again.

What Can You Do To Fix It?

Directly? Probably nothing. We’re in workround territory, not fix-it land.

Depends on the down time and the Schedule I think. From other clients, if you restart earlier in the day, then you can get flow. It doesn’t seem to be affected by budget. It may be affected by “accelerated spending” – I’m seeing signs of some relief from the problem if I pick standard serving.

If your down time is short, then just keep rolling – the loss of spend to bring people in when you can’t service them, is probably less than the loss of earnings for the longer period when you aren’t bringing in any prospects all.

You’ll really need to work out what costs you more – losing visitors for the late turn on, or spending to bring in prospects when you can’t service them properly.

Summary

I’m fairly certain that this is a bug. There may be ways to exploit this is specific markets. If you have an industry wide “down period” – then your competitors have probably altered their advertising schedules to fit. In which case, there should be a ripe spot where advertising costs are suddenly lower. Might be worth looking at whether you can tweak your down-time schedule and offer capacity when the advertising *should* be better…

If you are a big company… then lean on your advertising agency to explain why your spend misses some hours when you should have flow. And lean on Google to fix it.

There may be other reasons. I’m not claiming to have nailed them all. If you have the “Account Spending Limit Reached” message, with unspent budget… and you don’t have Ad Scheduling, drop me a note and let me have a look at your account. I might spot another pattern. :)

Pretend that you advertise for a TV network. The TV company offers live scores on their web site. So you pay for the keywords “football live scores” and “live football scores”, “live scores”, etc. You’ll bid less for “live scores” than the keywords that include “football” – people also look for “nba live scores”, “cricket live scores”, etc and their CTR and conversion rates are lower on your football-only pages.

For this account, you disable Content Match – this is initially focused on keyword search; eventually you’ll operate another campaign with differently constructed AdGroups and adverts for the AdSense network. So, no content match for now.

You use Broad, Phrase and Exact Match. Broad match will catch “cricinfo.com”, “crickinfo.com” and other stuff that is related to sports scores – so you need to start building up a good long list of negative keywords. But whatever else, at least you know that you have excluded the Content Network.

Despite that… your advert could appear here:

Can you tell the difference between this pages presentation of a keyword search targeted advert and AdSense? No? I can’t. If that’s not content network, then what is it? It isn’t keyword search in any way that I understand the term. Of course, Google don’t actually define “search pages” and “content pages” in their advertising contracts… So if Google defines this as a “Search Page”, no advertiser has a leg to stand on in complaining to them.

Here’s the three live campaigns for this client, this month, with identifying details and exact keyword data blocked out. This is just to show that this client has not been paying for contextual adverts:

How can I tell that the account has been displaying adverts here? Referral Information. Web server log files and web analytics reveal the source:

See that third line? “RightHealth.com”? That’s the one showing “Liver Scores” adverts. You can’t tell from this snippet *when* that data was collected. I can’t see a good way to show you that this was collected recently, but I observed this in the clients web analytics between the dates 23rd October and 26th October.

Note that referral information can be missing – some browsers don’t send it, bots don’t usually set it, and it can be withheld intentionally under various conditions. Referrer information can also be wrong – intentionally or unintentionally. The Referrer information comes from the web browser of the user, not Google. So if this referral source has been faked by a user, they’ve gone to some considerable effort to make the click look like a plausible one – they’ll have had to use the keyword to find a page with a spelling variation that could have triggered a content match advert, and then stuffed the referrer with that. The question is… why would they bother? Until I can understand an economic motive to go this level of work, I’m going to assume that, for this instance, there is a low incentive for the user to lie about the referral source.

Had this genuinely been a content network click, there *is* an economic incentive to fake the browsers information – it deflects attention from a real source of low quality clicks. Only the “gclid” can reveal where the impression was actually served, and Google appear to be the only organisation that can decode the meaning of the gclid value.

There is one additional clue as to what is happening, and the dates. I was initially running this campaign only on Google Search. I enabled the Partner Search Network on Thursday 23rd this week. Screenshot from the Account History with identifying data removed:

These types of click only happen when the Search Network is enabled. They disappear when using only Google Search.

Additionally note that I have excluded 404 pages, war pages, domain parks, etc from these campaigns. So there’s no reason for anything remotely resembling a content network advert to appear. I’ve done my best to remove any reason to show these adverts for any reason other than keyword search.

A completely inappropriate context (liver scores), on a targeting method I’ve denied.

Does it get worse than that?

Could Google have already excluded this click from my clients’ bill? Quite possibly. But who can tell – Google won’t offer a click by click itemisation of which clicks are billed and which are excluded. So you have to vet everything and appeal everything. Extra costs for the client – adding to advertising costs.

Ignoring the arbitrage

I’m not going to get started on the arbitrage – see that Ask advert? Ask resells Google Keyword Search adverts. So the low cost Content Match advert brings people to Ask for Keyword Search, amplifying Ask revenues.

The value to *advertisers* of this type of arbitrage is completely a different question.

Questions

Wall Street Analysts – want to know why Google’s result are better? Does this example suggest a mechanism?

Advertisers? Want to know why your ROI goes down as well as up? Do you check your referral sources exist and are relevant to the type of advertising that you paid for?

Search users? Ever wondered why some adverts seem so poorly aimed? It may not be the advertiser choosing inappropriate keywords and sites, it might be Google’s greed. OTOH, I see so many badly designed AdWords accounts that I wouldn’t be surprised if you did see a lot of irrelevant adverts.

Why is Google so nasty about landing page quality scores and so ineffectual at matching search queries and adverts? Is it that they have a large stock of unsold inventory and they throw any advert at that to see what sticks? Should advertisers treat Google’s apparent interest in Quality Scores as a way to raise expectations that Google is as diligent on search queries – a marketing technique to raise advertiser’s bids in the face of a variable quality searches?

Conclusions

I’ve fired off a pretty stroppy email to my account rep. I’ll let you know how it goes.

Blog spam is interesting for several other reasons, of course – not least being that there is a widely used technique to de-rate comments – the NOFOLLOW attribute for links. As with a bunch of other bloggers, I tend to prefer using spam rejection tools rather than nofollow. IMO, it’s better for users to see real comments than to (ineffectively) defuse spamming efforts with NOFOLLOW. CAPTCHA is moderately effective, but still places a burden on real users – and also still allows outsourcing spam generation to low cost economies.

As should be quite obvious (try to submit a comment) we don’t use CAPTCHA, so we get human and machine submitted spam. A lot of machine submitted spam. I normally only see it when I read the spam queue. Yes. I do that. OK, I’m wierd. I like to know what the buzz is. If big companies are sponsoring spam, knowingly or unwittingly. I discover all sorts of stuff – some of it seriously unpleasant.

I wondered whether there was a way to use machine submitted spam to measure blog rank. I couldn’t find anyone writing about measuring spam to individual blogs and the implications for the blog. The articles probably don’t have enough search rank to appear… I did find a recent article about spam and blogs and particularly about the value of closing comments to old articles. I’m not entirely convinced by the article – although there’s clearly a lot of research gone into it. I think there’s a key point of disagreement – Lorelle VanFossen asserts that blogs are found by robot crawlers, not by people using search engines. She says that blog spam doesn’t start until there is an inbound link. However, AFAICS, it’s the inbound link that causes search engines to rank blogs and allow the results to appear. It seems to me that the evidence for the link being involved is high, but I infer that the link gives pagerank… Something else I need to add to the research queue, I suppose.

Here’s an extract from an email that we’ve recently received, offering me low-cost-economy “link building”. Note the emphasis on relevance. IOW, this is built using search engine results, not webcrawling. This is typical of the unsolicited proposals that we get. These approaches clearly work or they wouldn’t be using this kind of text. They aren’t offering traffic directly, but rank, and they are clearly using search engine results to identify the targets.

My concern isn’t so much how spam starts, as the changes in volume. And that, I think, casts an incidental light on origins. If spam is a consequence of rich inbound links, then spam volume will tend to increase as a blog gains links. A model based on inbound links as the source would not tend to show decreases in volume. Yet… that can happen. Blog spam can decrease as well as increase – just like your other investments.

Why does spam volume change?

The point of spam is usually to improve the page rank of the targeted content, and partially so that readers will click on the links (direct traffic generation rather than indirect through rank increases). Spammers look for high page rank blogs that accept comments, especially when the area of interest is close to the spammy link to be planted.

Monitoring search engine queries that lead to this blog is often informative. Several times per day, I’ll see a search query like “internet marketing blog comments”, “shoe blog comment”, or “adwords blog add comment”. These guys are not looking for a substantive blog in order to join the community – they are looking for a blog that has reasonable page rank and offers the possibility that their spam will be shown. Because I’ve used shoes in examples of advertising in various articles over the years, this blog used to be quite highly ranked amongst shoe blogs. Amusing.

You can see this effect on other blogs that offer no moderation or no-despamming of submissions. Here’s a screen shot of part of a page on a fairly new blog that currently has low rank, and no significant protection:

So why does that blog page get no comments for weeks, and then suddenly have hundreds of spam comments? Why can some blogs survive for years with a handful of spammy comments and no identifiable protection, while other blogs get thousands of spam comments submitted, even though there are none or few visible on the blog?

Spam is non-linearly proportional to rank.

Once the observation has been made, I think there’s a pretty obvious interpretation:

The higher the page rank of a blog, the more that it will be spammed.

There is other evidence. An odd quirk of cut and paste in Google’s results used to lead to an embedded space in the search query. We’ve seen this signature every so often, in our web server logfile analysis. In addition to the obvious spammers looking for a blog to dribble on, there’s an even smarter crew doing product specific searches and then cutting and pasting to get to the site.

From observations of this blog and those of client accounts, the particular spam rejection technique seems irrelevant. Whether a “silent killer” like Akismet is used, or one of the CAPTCHA implementations, or the religious use of NOFOLLOW, spammers will try to submit spam. This implies that it is cheaper to submit spam to a lot of blogs, some of which are protected against it, than to develop the software that verifies whether the spam is accepted and published.

I had a look at web server logfiles, to see if there was any pattern between IP addresses that might indicate a separate spider followed later by a spammer. I can’t see one. I didn’t really expect to do so. Botnets, dynamic IP addresses, anonymising networks – the only reason to be closely correlated these days would be if you *wanted* be found. The nature of automation is such that there should be no real reason why a technique like measuring packet time of flight should work – because there’s no significant time dependencies involved. So I gave up on the idea of genuinely identifying and correlating the sources of spam…

The essence appears to be that spam targeted blogs are found, not by crawling the web, but by using search engines. This is fairly smart – because blogs that are higher in search engine results are obviously being crawled by the search engine spiders, and will contribute more to weighting, than a blog that isn’t even indexed by the search engines. Different spammers will have different search queries for finding blogs to target, but the overall effect will be that higher rank blogs will tend to attract more spammers than lower ranked blogs – because higher ranked blogs will appear in a wide range of results for different but related search queries – kind of like a Levenshtein distance of blogs. Higher ranked blogs will tend to appear on a wider range of searches, as well as being higher in each set of results – so there is an intrinsic non-linearity, where higher ranks attract significantly more spam than would be expected. There is a selection based on position – the higher the position the greater the likelihood of being a target – and on breadth of matching (a higher ranking blog will tend to be more authoritative about a wider range of stuff, with the way that Google ranking works). Hence a massively increased interest as blogs get higher ranked. Or a significant decrease in attention for minor slippages of rank.

Changes in Blog Rank

Changes in search engine rank appear to drive changes in volume of spam. The question is, what provokes changes in rank? This is a standard exercise in SEO, at least as I practice it.

I’ve used this blog to experiment on the factors that make a blog more, and less, interesting to users. I’ve also got experience of our clients blogs. This experimentation accounts, in part, for the frequency of postings here, and the variation in types and quality of article. A few articles (which I am in the process of marking up) are of dubious quality – the question in my mind was “will readers call me on it?”. Some articles are clearly antiquated – events have superceded them – what happens to those? Does the style of an article dictate the number and quality of inbound links? Which is the most popular page, and the page with the most and highest quality links?

Experimental Results

Really interesting results.

Linkbait articles, with or without logical flaws or internal inconsistency, are the most highly linked-to, but receive almost no traffic today, only a few months later.

A long, detailed, one year old, technical article about gclid, and another about adwords conversion tracking from two years ago, attract the most consistent day-in, day-out traffic. The third highly read, long duration article is about AdWords Editorial Review, but nearly all the people who read that are *not* coming from search engines, but from articles posted in the AdWords Help Forum. That’s probably because Editorial Review is still a pretty specialist term, much less likely to get attention than the problems it causes :)

Commenters pay no attention to the date of publication. An article that was correct at the time of publishing, may be superceded by events, and will then receive comments as if the article was current.

Embedded links in either the top or tail of an article, referring to updated information, are almost completely unused. The average time spent reading this blog is typically more than two minutes – some days it hits more than 6 minutes as the average time, and only drops significantly below two minutes when we’ve been Stumbled. In other words, the articles are read, in depth – but less than 5% of readers even follow a primary attribution. If I’ve written in response to another article, the source is rarely clicked on.

One of the least clicked links, that I notice, is the one in the gclid article, near the end, referencing Cut-Me-Own-Throat-Dibbler. So far as I can recall, it has been clicked three times in about 18 months, despite being in the most popular article. A few dozen people spend about 4 to 8 minutes each reading that article, each day, or about 10,000 readers spending a cumulative 1,000 hours of reading (an entire solid six weeks of reading). Three in 10,000 = 0.03%. Not a lot of onward clicking, is it? OTOH, the idea of writing an article that has cumulatively taken six weeks of reading is pretty awesome and makes me feel guilty about article content… Important to avoid wasting that much time!

Rank is not significantly affected by even quite extensive article rewriting. Some articles have been revised over the years to reflect changing information. They get remarkably consistent volumes and subjects of spamming, before and after rewrites. Comments don’t appear have much effect on spam volume either. Highly commented articles are no more likely, AFAICS, to attract spam, than uncommented articles. An article is more likely to get comments, if someone comments shortly after publication. That is partially because most articles have a short lifetime. Once the lifetime is over, they get a few visitors, who rarely comment.

The comment policy page is one of the least visited pages on the site. I’ve written all sorts of stuff on there, over the years, mostly for my own amusement. It has, at times, been a draconic rant against spammers with severe words about leaving meaningful comments.

Changes in the comment policy have no effect whatsoever on spam volume.

Changes in the wording around the comment box affect the volume and type of real comments – but do nothing to dissuade spammers.

Properly constructed outbound links within articles with a FOLLOW attribute (implied by default), to credible sources, won’t damage rank, and won’t cause many readers to leave the site. If the article is interesting, they’ll stay with you. If it isn’t, they’ll leave rather than click on a link in a useless article.

Summary

Monitoring spam levels is an unusual and intriguing way to detect changes in search engine behaviour and the overall rank of the blog.

Linkbait, unsurprisingly, still works to get inbound links. That’s still a great way to get rank. Be contentious. Be provocative. Standard journalistic conventions for dispute and disagreement… Doesn’t engender wisdom, peace and happiness, but does get you noticed. If the article that follows is sufficiently provocative, it’ll get lots of links within minutes.

Article topic and treatment has a huge impact on longevity of traffic. A comprehensive article that solves user problems, lasts for ages and may become permanently popular. A newsy article has a lifetime of hours to days and then gets random low traffic.

New articles do more for rank than refreshing old articles. Something in the range of one decent (as in “attracts inbound links”) article per week, will be just about enough to maintain rank. Two would be better. I infer and have observed that search engines will rapidly give rank to a new article, but do not do the same for revised content for an old article.

IME, popular articles covering technical issues should be corrected in place – even if it means significant rewriting. Strictly, I think that re-writing should force a re-dating of the article. Possibly it means that the article should be migrated to “static” content on the main website rather than being a blog article. I’m still muttering to myself about this conclusion, so the advice may change. The reason is that users don’t follow even very obvious and repeated links to later and better information.

Readers rarely call out the author, even on internal inconsistency. That’s actually worrying. If there’s an intent to convey factual accuracy, then the responsibility falls hard on the author. Blogs aren’t peer review…

Use plenty of links to authoritative references and similar articles. Isolating your blog doesn’t make it more authoritative. Referring to other blogs binds you in to the community and generates more inbound links, which brings you more readers than you lose to the links.

Most comments that pass spam filters will also pass any reasonable moderation – but occasionally some comments are clearly intended for the author of the article, or are amazingly off-topic (but not spam – requests for help with specific technical issues only peripherally related, or to do homework exercises, etc).

Very few readers and commenters look at the Comment Policy. Have one on your blog, but make it liberal. Most people are either spammers or really interested, interesting and helpful. I learn a lot by looking at the comments. I’m grateful for the insights you share with me, and exposure of the problems of interpretation that I inflict in the articles.

Use a welcoming message near your comment box, rather than some aggressive statement about spam. Assertive anti-spam statements will dissuade some real commenters and will do little or nothing to change submitted spam volume.

Monitor the searches that lead to your blog – changes in the nature of the searches may reflect changes in how the search engines perceive your content. I’ve found Lijit to be a helpful tool – more helpful than FeedBurner for understanding searches. If you visit this blog frequently – once every few weeks, or so – you can watch how the search cloud changes.

Consider joining the No NOFOLLOW crew. Nofollow hasn’t stopped blog spam, but it has provided a mechanism to penalise sites outside the trusted set. It reinforces isolation of high volume and high ranking sites from the rest and does almost nothing to stem the tide of blog spam. I wish Google had put their efforts into publicly accessible spam filters for blogs and web form submissions, not an attribute that creates extra work for little perceptible positive benefit. Actually, that’s not entirely true. I’m pretty sure the tag helps Google. They’ve recruited webmasters and blog writers to do their job for them… identifying the garbage.

Updates

2008-10-21 – edits for clarity and typos. Added new illustration of typical link spamming UCE.