When the networks merge, the combined total should probably be about 20-25% of the AdWords total. Now that’s of a size to start getting properly interested in optimisation, and one can justify some management time to organising and sorting it out.

Because we’re anticipating increasing client spend on the enlarged network, we’re looking at what techniques we can use to optimise. That’s what we do… optimise stuff, after digging into it. So I wanted to comment on something that I thought would be useful to do, in the larger network. I went to the Microsoft adCenter user discussion forum to contribute. And I got this, when I tried to submit a comment:

Microsoft adCenter discussion forum requires MSIE, through non-display of CAPTCHA on non-MSIE browsers.

Microsoft is making use of their advertising platform pretty difficult. Firstly, I don’t like this whole “you must use MSIE to get the experience”. That’s so 1997. These days, there are browsers that offer very usable interfaces. Oh look, I’m using three of them now – Chrome, FireFox and Safari. It’s not as if Microsoft even said “MSIE9 is so far ahead that we’re making that the standard” (because that’d be a pretty stupid thing to say, anyway), because MSIE8 *can* be used – and that’s certainly no more advanced than any of the other major browsers.

Nope, Microsoft have made de-facto that to effectively use their web interface for their advertising system, you have to run their operating system and their browser. It’s so… retro non-chic. It’s a marketing approach out of some 1970′s vendor lockin agenda.

Anyway, the point of frustration was passed today, when I realised that not only do I have to use a Windows system to interact with the advertising platform, I also have to use a Windows system to interact with the community forum. It didn’t use to be that way. A year or so ago, I could post message. Now, though, I have to complete a CAPTCHA – which, regrettably, is not visible on Chrome, Safari or Firefox, and has no recommendation as to the plugin I need to adopt. Nope, if I use a non-Windows system with a non-MS browser, they just don’t care to hear from me.

With no CAPTCHA form visible, how do I submit the form with a CAPTCHA? That’s just a design and test fail.

And there’s another thing…

If I’ve just typed a comment to your blogging platform, couldn’t you had the courtesy to let me know that I needed to be running MSIE *before* I start writing? I mean you do *know* that your “community” now requires MSIE to be used before a comment can be added? In my case, that means usurping a Windows machine used for compatibility testing and adCenter operations, so I can contribute to a discussion. I don’t feel that generous to Microsoft. I have better things to do with my time than switch machines, in order to post a comment in a discussion thread.

But that’s not even the serious thing…

If I’ve just spent five minutes typing in a reply on the forum, Microsoft knows quite a bit about me. Potential details include stuff like the pacing of characters, the previous activity of the Microsoft authentication service by that ID, the IP address, possibly backtracking the IP address, and details of the packets to indicate the type of operating system I’m using, all those browser header details, and various other clues that I’m probably a human. So if you can’t tell from my logging in and all that typing, that I’m human – how the hell are you going to tell from a single click that my advert clicking is or is not fraudulent?

How about it Microsoft? Can you actually detect bad clicks at all, or do you just guess that some percentage are bad? The amount of spend we’ve done on behalf of clients, and the ROI, has never meant that it is worth checking the adCenter fraud detection system. I think I’m getting interested now.

And why can’t Microsoft make its discussion and feedback systems accessible to other browsers, like any other competent technical organisation? What makes Microsoft so technologically incapable that a *discussion* forum requires a technology limitation on the platform used for access? This is just so primitive. I am *so* glad that I decided my organisation would not be using Windows as a standard operating platform. This level of lockin is just tedious obstructivism. At least have the courtesy to tell me when I log in that “some features of this discussion, such as replying to comments, are not possible unless you use Microsoft Internet Explorer 8 or 9″? Then I won’t waste my time trying to engage.

Microsoft has not convinced me of the Windows Advantage. It has convinced me that they are technologically primitive, and that using Windows systems would be a technological backwater. That’s the complete reverse of what they probably thought they were doing when they made this design decision. The signal I get is “if you use Microsoft products, you’re locked in, unable to choose what is best for your business, and we don’t care at all”. That’s not a good message to people who aren’t using Windows now. It is not going to convince us that using WIndows is a good idea. Forcing users to do something is weaker for branding, than exciting them.

Let’s have a look at what Google Product Search is, and does, first… so we’ve got some context.

Google Product Search

It’s free. Anyone can sign up, and feed products. Well, almost anyone. Google has a restriction that there must be one supplier for a given product URL. So you can’t have lots of people all listing the same DVD on Amazon – Google doesn’t like spam, and multiple appearances of the same product from the same vendor at the same price, count as spam.

Within limits, then, Google Product Search is free and a wide range of businesses can use it. You just need to sign up for a Google Merchant Center account, and publish a feed. A feed can be as simple as a properly formatted CSV file (exportable from a spreadsheet, once per month) or as complex as a machine-readable and real-time updated XML file synchronised with your product inventory and pricing.

Product Search results – well, they don’t appear in every search, and they don’t alway appear as number one in the search listings. That last item is especially important and tells us, the rest of the world, something about how Google is treating Google Product Search.

Why is it important that Google Product Search is not always number one for product related searches? Anyone involved with search engines knows that number of clicks to a listing is *very strongly* correlated with the position on the page. A listing in position 1 can easily get ten times, or more, as many clicks as exactly the same listing at the bottom of the page. So for Google to list Google Product Search at positions *other than* position 1, tells us that Google is giving up a lot of influence. And remember, Google isn’t making money (none directly, and only indirectly some revenue) from Product Search listings.

The implication of the varying position of Product Search results is that Google is not giving undue weight to their own product. The reverse implication – that Google is artificially reducing the ranking of competing product listing services – is not addressed by that observation. Just seeing that Google doesn’t inflate it’s own rank in all cases, is not enough to say that a different service has not been penalised.

Are there any other vertical search engines in Google results?

If Foundem has a case, it is that they are being excluded, artificially from results. You don’t just get listed highly in search results because you exist. Well, not unless the search is pretty rare. If you’re talking about the highly competitive spaces in which searches relate to things that people might buy, the competition is intense. To show there, you need a lot of “link love”. You need a lot of people to link to your site, with the right words, and you need the site to be well designed, and you need to have *credible* links. If you don’t have all of that, you don’t stand much chance.

So, let’s assume that Foundem, being fairly new, and not having made much of a media impact before today, doesn’t have a lot of link love. Can we support their argument by showing that Google doesn’t highly rank vertical search engines?

Annoyingly for Foundem, there are vertical search products that show. Businesses like Trovit, with their used car facet-based searches, letting you slip a slider along the price range that interests you, does show up. And they show up for place reviews. And other stuff… It is arguably a better user experience than Google Product Search for used cars. Trovit have page one appearances for some products. I can see their impact on clients’ sites.

Google *does* allow vertical search specialists to appear in search results. That, for me, puts a dent in the argument that Google excludes vertical search specialists. If some do appear, and they collect clicks, in multiple categories of product, then Google is not completely removing them from listings. A dent, but not conclusive evidence.

Do Ciao and Foundem have a case?

I don’t think an appeal based on “we must be shown because we’re a competitor” has any real merit. I’m not a lawyer – so that’s not a commentary on whether the courts would find it to be true. But in terms of search engine results, if you aren’t sufficiently loved by users, to displace the other ten businesses that are shown, that’s not, IMO, a reason for a court to decide that you should be shown. Win the users’ love… and you should be listed.

But, since the algorithms are trade secrets, and the data collected is known only to Google, for an outsider to claim that they have been penalised by Google, is pretty difficult to discern, or prove.

Unless, of course, you have the backing of another search engine that produces broadly similar results, in which your site appears more highly ranked and in which you have no evidence to believe that the results are influenced. In other words, if you were financed by Bing (the only other serious volume search engine left in the game), then you might have strong evidence that you don’t have spammy links, spammy comments and other factors that would cause de-weighting.

Ciao is owned by Bing. Might there be some evidence? Or is this just Bing (Microsoft) playing games with Google. FUD (Fear, Uncertainty and Doubt) have always been strong components of Microsoft marketing techniques. I certainly don’t have any confidence that Microsoft is doing this because they love Egalité, Fraternité, Liberté and the European Way Of Life.

Foundem – it’s a small, relatively new player, in a crowded market. I suspect their only problem is the classic one – not enough users love them to help their site to rank highly. But I haven’t investigated their site or backlinks. That’s just a first pass impression with no evidence. I’ll gleefully admit I could be wrong – though I’d be very surprised to be badly wrong.

But the possibility of Google reducing the rank of a disfavoured competitor, or slightly inflating the rank of a preferred information supplier, is something that gets me twitching. There’s no oversight for Google. We have only their word that they are using what they publicly claim to be fair. If, in the depth of a recession, California based Google were to be weighting US businesses slightly higher in the UK than they should be, it would direct between tens of millions and billions of spend to US companies and away from British companies. Same for France, and Germany. Is Google making sure that the US economy brings in a little more business than it *should* do? I have no idea. I can’t measure it. I can’t tell. It does worry me. I worked in the USA for about a decade, and I know how insular US businesses can be. The rest of the world has fewer investors, fewer buyers and works in funny languages and currencies; it is easy to fall into the model of thinking about what works best in the US economy and rolling that out, worldwide.

So there *may* be some basis for European businesses to wonder if Google is deweighting their business. It may be deweighted because it doesn’t fit a market model that Americans understand. It may be deweighted because an American business is, even unconsciously, working for the end of a recession in their domestic market. It may even be a conscious effort. That really, really worries me – because I don’t think even a national government outside the USA, could get the data to investigate whether Google is deliberately, or unintentionally, diverting economically significant activity, artificially, to the USA.

Do Ciao and Foundem have a case? I don’t know. I’m looking forward to finding out, if it is released, just what evidence might make them think they do. I hope it’s a lot more compelling than “we want our share of the Google billions”, “Microsoft is using me as a sock puppet to attack Google” or “Google’s Buzz is creating a weakness in perception we can exploit”. I fear there may be some interesting biases in Google’s system that do favour US businesses – simply because I can’t see any way to prevent there from being such a bias, or to detect one if it were present. And *that* has implications for national security – the protection of the realm. Interesting times.

Updates

2010-02-27 – SearchEngineLand article about Microsoft’s involvement.

Best comment so far from “G-Force”, to embed some ASP code in each page that checks the file name, and 301′s to the “right” case variant. Heavy work – individual page edit for each unique page. Unless, hmm, one uses one of the Unix tool kits. I could see using Perl to auto edit each file in a directory hierarchy.

What is the authoritative name of the web page?

Assume that your web site uses IIS and has a home page available as “/index.htm” You can then use the following as silent synonyms of the home page:

• /index.htm
• /Index.htm
• /INDEX.HTM
• /InDeX.hTm

Why is finding the same file under multiple names a problem?

Whether you use a web browser, or a search engine bot crawls your web site, each one of those URLs appears to be a different file, albeit with identical content. If you have optimised the content, then each is a candidate to be the answer for search users. So the spiders do need to track each file name variant, and check them, to see what has changed.

In each case you get an immediate web server response of “200″ – file found. Link love can be spent on a wide variety of paths that lead to the same place – but the spiders aren’t told that. There is another way to do this, which does not work, out of the box, on Linux and Apache, but is fairly easy to set up.

Web Brand Is All About Experience

If the file system does not fold case – that is, it treats upper and lower case letters as two distinct things – then a request for a file with mismatched case delivers a 404 – File Not Found. Now that’s a bad user interface experience. Brand is all about experience, so why punish your users with a 404 because they can’t remember what the capitalisation of your ProDuct (sic) is?

You need to find a way to deliver both the web page that the user wanted to get to, and also let the spiders know that there is one authoritative page – there may be a lot of different links to get there, but just one resource.

On Merjis.com we use a technique that helps the spider understand that we have one page, and that case changes are a link problem, not a server duplication. If you try to reach:

• http://merjis.com/contact
• http://merjis.com/Contact
• http://merjis.com/CoNtAcT
• http://merjis.com/CONTACT

then you should get to the same page, the Merjis contact information page. In terms of the user experience, this is just the same user experience as on IIS. But we issue a redirect on all the non-standard forms of the page name and IIS doesn’t. Spiders can see that only one page for “contact” exists on the Merjis site, even if it is accessible through many different URLs. This cuts down redundant crawls, focuses link love on a single page, doesn’t lose references from typographically challenged links, gets users to the page they want whatever the case of the URL they type, and is generally A Good Thing.

How To See A Redirect

If you don’t use a tool like “wget” or one of the Firefox HTTP inspection tools, your only real clue to our redirection is that whatever you did type in the URL bar, is replaced by our chosen URL for the resource. Between your input and our response, we added a redirect. Spiders will see the redirection, and only index one page and can pour all the link love on that page.

That’s completely unlike IIS standard behaviour. The default behaviour is to fold uppercase and lowercase. That means you see the URL that you typed. There’s no information that the file is a single file known by many names.

Spiders can’t guess that a single file is a single file – they only know what they are told. They get told what links exist in sitemaps and by other link references across the web. If a spidered site has references to mixed case versions of names, then the spiders will tromp madly off to each alternative case version of exactly the same file.

SEO Means Never Saying Sorry To Stupid Spiders

I’m of the opinion that helping the spiders to find the right information, helps SEO. Sending spiders to a dozen spelling variations of a path, doesn’t boost rank, unless the spiders are clever. Even without that, sending spiders to crawl redundant pages, when they could more frequently crawl real content, is a waste of the attention from search engines. If spiders were clever, they wouldn’t crawl redundant paths to the same content, repeatedly, across the whole server… So give them a hand to get to the right single file that should be taking all the page rank.

This default case folding behaviour means that IIS again contributes to spamming search engine indexes. Not so bad, except that it causes yet another problem.

My Web Analytics Don’t Fold Case

When you are trying to analyse what is happening to users, just one miskeyed filename can result in the analytics giving you multiple paths to, for example, conversion. Typically the JavaScript page bug reports the filename that was accessed – using the same capitalisation that the web server delivered. Why? Because on a large fraction of the other web servers, case does matter and “/index” is a different file from “/INDEX”.

So on an IIS delivered file, the same page can be known by a wide range of names that mean the same thing. But analytics packages don’t (usually) fold case – so each reference to a different capitalisation adds another meaningless node to journey analysis.

[ N.B. See Chris's comment below. I should quantify the assertion that Analytics don't fold case - of course, if any of them *do*, that's another problem... ]

The following deleted section is a bit rubbish. I failed to properly read and understand the robots.txt spec. I interpreted a line that meant all records in the file, to just mean the User-Agent line. robots.txt allows case folding – /MEMBERS and /members are identical according to the later spec; the earlier spec only clearly states that the User-Agent field ignores case – leaving the possibility of ignoring case or respecting it in a Disallow line.

However – I started this article because I found a clients private area on IIS had been crawled and indexed, despite being listed in robots.txt. I still need to describe that investigation – but this article is long enough.

And that’s not all. Oh no…

Google Spiders Content Disallowed In Robots.txt

If you have parts of the site that you don’t want to be in the index, you can use robots.txt to exclude those directories or applications.

Except… you can’t. Not sensibly, not with IIS out of the box.

Say that you have a directory called “/members” and you want only signed in members to see the content. You exclude the spiders with:

Disallow: /members

However… case folding… This directory is also accessible as “/MEMBERS” and that isn’t excluded in this customers’ Robots.txt. So your hidden content is now visible if just one link, somewhere on the internet, or even in your own site, uses a different capitalisation from that which has been put in the Robots.txt file.

Is this Google and Yahoo!’s problem to resolve? IMO, not really. If you choose to use a server that makes the same content available under a range of paths, it is up to you to protect those paths, not for spider developers to guess that you may have shot yourself in the foot.

OTOH, the SEs do themselves no favours for reducing spam in the indexes, and decreasing crawl volumes and bandwidth usage, by failing to recognise that IIS can serve the same page under a multiplicity of case variations. That’s a different problem – but solving one would solve the other. If the Server can be detected as using case folding then using a case independent match of Robots.txt paths would be a useful extension.

I can even imagine adding a new directive to Robots.txt to express that pathnames do not respect case.

Other Case Folding Systems

Well, Apple OS X. It may be my favoured desktop OS, but it has a default FS that is caseless:


$ echo boo > goose
$ cat GOOSE
boo
$


I haven’t tested – I have no SEO clients with Mac servers – but I suspect that Mac servers with a default FS run the same problem of a futile and useless Robots.txt protection.

IIRC, OS/2 aka “Warp” was used for some years as a web server and it used a case folding FS – so if you run one of those ancestral systems, watch out.

Webmasters

Your defence? Well, make IIS respect case in queries and do a proper redirect to the actual file.

That way, spiders could properly use Robots.txt and your hidden content wouldn’t be accessible. Try asking Microsoft about that configuration. Heh. Here’s Microsoft’s page about creating a Robots.txt – note the discussion about case folding? Oh, there *is* no mention of case folding? Hmm. Well, I think that’s a lesson in its own right.

Failing any rational advice from Microsoft, you could do a combinatorial madness on Robots.txt:


Disallow: /members
Disallow: /Members
Disallow: /mEmbers
...
Disallow: /MEmbers
...
Disallow: /MEMBERS

and so on. It’s easy. Yeah. Right.

Have I mentioned that I think IIS adds problems, rather than removing them?

It is all so much more complex if you have cookieless mode enabled for your ASP applications. Because the path given to the robots doesn’t match the path that is denied. Deny “/secret” and you get a path that starts “/(” and goes on to “))/secret”. Combine that with case folding and there is no end to index spamming.

And, of course, Robots are the main thing that need to read and respect Robots.txt.

Given a modicum of sense, this whole area could be made a lot simpler for system admins and webmasters. If an IP address and user agent doesn’t accept cookies, and asks for “robots.txt”, it is probably a robot. Stop sending cookieless tracking paths to that IP and UA.

Sticking In a Reverse Caching Proxy

If you are a technological sophisticate, then you could insert Apache with a rewriter and mod_speling, to get the benefits of case matching and redirection to the single real file instance. You’ll possibly see a slight average speed up for users, as unchanged content is delivereddirect from the Apache cache.

How to set up and configure one of these cute web servers is beyond the scope of this article, though.

Scale Of Problems

It’s quite nasty for people using hosted IIS, who have no significant control. I have no doubt that the SEs do duplicate detection. Matt Cutts has written that in-site duplication isn’t too awful a penalty – probably because they keep spidering IIS sites. OTOH, places like WebMasterWorld have a fair number of webmaster stories about having two or more case variations of the same page in the search engine indexes, with radically different page rank – and that depending upon what has been spidered recently, the position of the site will change radically.

There is a difference between ignoring duplicates and sending full credit for an inbound link to the “master” version of the duplicates. Some folk seem to think that there’s no great loss. I’m pretty conservative about this – why risk losing the benefits of inbound links, just because of case folding?

Case folding for private content is a problem. I’ve seen many complaints about Google revealing information concealed by Robots.txt, and I have strong suspicions that in most of these cases, the complainants were running IIS and had an undetectably case-mismatched link reference somewhere on the site, or had enabled Cookieless mode for ASP.

I can’t find any authoritative discussion about this issue, especially in Microsofts’ Knowledge Base, hence the posting here. Of course it may be that I detest MS so much that I haven’t spent enough time poking around their resources. That’d be a quite valid criticism of me and this article. :)

Other Solutions For Privacy

Given that Robots.txt is effectively useless faced with both Cookieless mode and case folding, you really, really need to get to grips with your friendly page-specific metatags “NOINDEX”, “NOFOLLOW” and NOARCHIVE”.

Any private content should at least have “NOINDEX”.

Arguably, terminal pages (action pages, etc) and private content, get “NOFOLLOW” directives.

Summary

IIS, by default, apparently opens up web sites to some Google Bowling activities. You can even bowl yourself out, if you use case variations in your own URLs.

Carefully review what you do have; probably the simplest suggestion is that you stick to all lower case in every link. Watch out for inbound links that use mixed case – those can scupper your rank.

Use the meta tag for robots to set NOINDEX and possibly NOFOLLOW on your private content on case-folding IIS systems to prevent your data leaking into the search engines.

Best bet – dump IIS. OK. That’s a biased view, and a very selective view. But if you want to rank highly in search engines, setting up and running a LAMP (Linux Apache MySQL PHP) or similar system is easy and cheap. While it does have comparable problems (unbranded 404′s etc) the solutions are also cheap and easy to set up. If I’m really blunt, I wouldn’t actually use PHP, either – the script language design makes it too easy to embed SQL, IME. I’d steer for Python, Ruby On Rails or OCaml… probably ;)

It’s better than finding your private data leaked, or that you’ve been scuppered by a near unfindable typo.

Clients that use IIS do seem to suffer from the strangest problems. I kept finding indexed pages, some ranking absurdly highly, with an path infix like (X(kjhgkjfkuyfku)). In other words, if I had a page like “http://merjis.com/login”, I’d find exactly the same content as “http://merjis.com/(X(stuff))/login, making it look as if I had lots of duplicate pages. Appended parameters as a session ID, I can understand. There’s even ways to cope with those.

The format of the infix was quite rigid – 23 characters starting and ending with a parenthesis, and embedding at least one more parenthetical group. I think the regex “\/\([A-Z]\([A-Za-z0-9]*\)\)\/” will match every example that I’ve seen. Oddly, you could navigate to them using a web browser and they worked, but even spidering the site and grepping the resulting mirror failed to show these strange paths in the HTML… so how did they get there?

Cookieless Session Tracking

This looks like an effort by Microsoft to allow tracking of people that don’t want to be tracked, or who might have, for example, an office based transparent proxy that blocks cookies. AFAICS, when IIS detects that a user doesn’t permit cookies, it starts sending unique paths. The use of the “referer_info” (sic) field allows tracking a single user across the site, looking at where they were and where they’ve gone to. The positive benefit is that users can be given access to stateful services (like logging in) without needing a cookie. It is, I think, effectively a session cookie, rather than a permanent cookie, as it can’t recognise you from previous sessions – your starting point will be the same as any other visitor.

This session tracking seems pretty barking to me. If a user prevents you from serving and saving a session cookie, and you then create a trackable session by using session keys in the URL, you have just violated the privacy that the user requested. If users try to use a use a resource that needs stateful dialogue – something that remembers whether you were logged in, for example – then I think that if the user disables cookies it is perfectly reasonable to tell them that this resource won’t work for any areas requiring an authenticated session, though the public areas may be freely roamed. If they want privacy in the session, then they can’t have access to anything that requires remembering from page view to page view, who they are.

Cookies and Search Engine Spiders

Robots don’t hold cookies. If faced with an IIS server configured to handle cookieless users, spiders end up being delivered with a false directory structure. Every new visit generates a new directory structure. You get excessive crawling and pages appear in the index as unique, when they are really duplicates caused by server behaviour.

Now, this too seems like a petty piece of madness. Given that some users want to maintain privacy to the extent that they do not even want session cookies, and this number is small on web servers that offer services involving a log in or other identification service, why would you make it more difficult for search engines to spider the site? I believe that you can get more users to a properly indexed site, than the number you’d lose from failing to handle uncookied users (unless you offer a specific service for the uncookied, of course).

Robots.txt to the rescue

Fortunately the workround for this problem seems small. In Robots.txt, add a line:

Disallow: *(


For all the spiders that I care about, that seems to prevent crawling the special tracking URL. The consequences of that… well, I’m not convinced it is entirely good. But it does stop silly URLs from being indexed after a single crawl.

Technically, this line says “disallow spidering for a URL starting with anything that has a ‘(‘ in it somewhere”. Although this client seems mostly to suffer from the infix immediately after the domain name, reading around the web suggests that the infix could be put at any directory level slash. Otherwise “Disallow: /(” would work and avoids the failure possibilities of the wildcard.

Spidering Improvements?

I can’t see why spiders should behave like this.

Having grabbed a server identity from the headers and behaviour, it should be possible to then strip out the session tracking from the path. I can’t currently think of a reason to *not* do this – unless you were really trying to sneakily discourage IIS administrators from using this tracking method.

I can think of anti-competitive ways to use the cookieless IIS behaviour. For example, find the user sitemap, point to it and let spiders follow the unique links to every path – then you can explode the crawl on each site by doubling it. And it accumulates – because every session appears to be valid for a long time, allowing repeat crawls and generating new unique paths with every reference.

You’d have thought that this explosion was worth trapping and stopping in the “fast crawl” discovery bots, so that the slower inspection spiders wouldn’t add these redundant pages to the indexes. The presence of these idiot links in link reports, and the multiple crawling of them by various bots, suggests that spiders are still stupid.

Webmasters?

Well, if you can’t avoid running IIS, take a good hard look at your web servers log files. Do you really have a useful volume of search from real users who are cookieless or have you just ramped up bandwidth so that bots can crawl redundant pages without adding any revenue? Is the majority usage of this feature an escalating collection of spiders? Might you get more users if link love wasn’t being directed to duplicate pages on redundant paths?

Put in the crawl reduction “Disallow: *(” and possibly a “Disallow: /(” lines to your Robots.txt and verify them with the Search Engine webmaster tools Robots.txt checkers. At least you’ll be focusing crawl on pages that should be indexed.

Ideally, turn off the cookieless mode. AFAICS, it is a breach of the privacy rights that users were trying to assert. If you offer a service that needs state, then when you detect that you can’t cookie, offer an apology that parts of the site are unusable. IMO, that’s perfectly acceptable for both users and designers and won’t lose any business that you couldn’t have gained.

You know, I thought I’d written in below, about how a service company (Yahoo) and a product company (MS) would have integration problems – an issue I’ve seen before in startups where competent managers in one type of operation failed to build a company in the other sphere. This article spells that out and supplies other great detail.

I’ll have to re-read the Roughly Drafted article and see if I can pick some holes or offer some amplifications, but the general thrust is compelling. Right now, I still do see an opportunity for advertisers. But for the companies involved, a mutual cooperation agreement for an approach to advertisers looks healthier than an acquisition. For search agencies, the fewer the number of outlets, the lower the costs of managing them all and the larger the addressed audience… The cost savings of managing the combined share of market through only one interface, MSN or Yahoo, could improve the attractiveness, especially for smaller accounts in Europe. As the Rough Draft article spells out, that’s probably not enough to make the business case.

In the UK, Google has over 80% of search share. But, as I’ve said before, the Content Network is a problem and it is often ignored by larger advertisers. Even worse, for Google, non-textual (display ads) are a small fraction of the Content Network. Google’s recent advice to advertisers will improve performance, helping increase advertising, but at the expense of higher costs to advertisers through building additional AdGroups with specialist techniques that may be uneconomic if recognised in accounting as a cost of sale – coupled with a lower click CTR and conversion rate, it’s usually a lethal combination for a small advertiser.

One of my biggest problems with advertisers and the Content Network, has been proving that the Content Network makes a difference. I have a feeling that Display Advert impressions make a difference to awareness. This may translate later into an increased CTR for paid search, or organic adverts, or affiliates, or even direct access to the site – but it is an invisible story unrevealed by attributable and measurable lead generation. This is a major problem for Google – they don’t like passing on information about users and have shown no indication of any willingness to pass on information about impressions (post impression tracking). Statistical treatments to prove that impression volumes have an effect, also need to take into account which sites are being used, and need large volumes of data to demonstrate statistical correlations. And it all gets much more complex when you consider latency (the time delay between beginning to think about buying something to actually getting around to it – can be measured in months or even years in parts of the travel industry).

Microsoft and Yahoo could steal advertiser hearts and establish a new standard in internet advertising by offering cookie tracking on impressions for display advertising. If I can show my clients that Display Adverts result in increased awareness, I can sell more advertising. I don’t think that MS and Yahoo would have the qualms that Google would have about this. As usual, I don’t care about identifying individuals – the goal of post impression tracking is to look for statistical correlations between adverts and response and offer ways to tune the delivery to users that are more likely to want to see the advert (win-win for user and advertiser). If I can deliver post impression tracking results and analysis, whew, I’ll be a much happier bunny.

The increased targeting options for Microsoft also play to segmentation concerns. I have huge doubts that Google’s content matching alone is enough to provide adequate segmentation. I’ve seen too much of “(named software) download” being matched to “(ripped music) download” and “(ripped video programme) download”, or other equivalent mismatching of content. Google’s pretty much unexplained “search behaviour” targeting, where they claim to improve performance through optimising based on recent searches… well, if I don’t understand it and can’t control it, I don’t trust it. I certainly don’t trust Google to do more than optimise their own revenue.

Microsofts targeting models also leave me a little concerned – but they are a step on the right path and frankly, without real information from Google about search history and targeting, I’d rather have the controls that MS offers, than the lack of control offered by Google. I can explain to clients what my errors are – but if Google is changing the game behind my back, I can’t justify changing response to clients. So a Microsoft model is much more interesting.

Much as I loathe Microsoft as a company (I’ve detested using Windows, for years), I do see real opportunities for Microsoft to work with Yahoo to improve targeting and measurement, and to steal advertising share from Google. Whether the world is better served with a Google near-monopoly of search advertising and a Microsoft near-monopoly of display advertising is a different question. But in the short term at least, this may require that Google treats advertisers with a little more respect, and that can’t be a bad thing from where I sit.

Particularly vexatious is that the screen saver keeps running. As you are trying to login, you can’t relax, secure in the knowledge that the next screen you see will be the desktop. You have to keep vigilant, and move the mouse, just a bit, every few minutes. It gets to be about as exciting as watching paint dry. Someone really needs to develop “Paint TV” that shows a series of images as it dries… Anyway, it spoils attention on any other serious projects, while you wait to login and evade the screen saver.

I wasted some time – which I won’t count, as it was my stupidity and ignorance – trying to run the AntiVirus package on all files on the system. After some time, it was estimating completion in between 6 hours and 205 days (or some obscenely large time). I killed the process, eventually, and started investigating the idea of reformatting the disk and re-installing. In doing so, I noticed an article that set me thinking…

After three hours, the Windows Task Manager finally came up, and showed “Spoolsv.exe” consuming 99% of the CPU, doing nothing visibly useful. OK, so this is a 2GHz laptop, 2GB RAM. Should be able to handle the kind of stuff I do (using MSIE7, usually, and some web analytics packages whose vendors are so perverted that the packages only work on MSIE). Four hours before I can even terminate the stupid program that is stopping me from working? Pathetic.

Searching for “spoolsv.exe” showed some promising links. The highest ranked result that was obviously about troubleshooting this problem was kinda OK. But the best answer was a lower ranked link. Here’s what to do if your system goes dog-slow as soon as it boots and you have a process called “spoolsv.exe” hogging the CPU. I’m putting in my link vote for this resource – easy to follow, and it has worked.

What caused the problem? No idea. Can’t find anyone describing how something as stupid as this survives in an operating system that is around twenty years old at this point. It’s seems like bad engineering and inattention to user difficulties. Which is why I loathe Windows. I detest badly designed stuff. I’m biased that way :) I use a Mac for a desk or laptop, or Linux as a server, when I can.

Having got the system back, I re-ran the AntiVirus and “Malicious Software Removal Tools” packages. No problems detected. Except that now the system won’t properly connect to WiFi. It keeps saying that it isn’t connected. Right clicking on the WiFi icon in the System Tray (lower right), lets me see that it has detected one wireless network (the right one) and that it is set to “Automatic Connection”. Clicking on the network is supposed to join it… and it ends up flashing a window that says “Connected!”. Then the “Connect” button changes to “Disconnect”, while the network connection status changes to…. “Not Connected”.

The Windows system will no longer connect to the WiFi that it used to use. I haven’t changed the WiFi settings and and all other equipment (a collection of Apple kit, a mobile phone using Microsoft’s PocketPC with WiFi, etc) is using the WiFi properly. Something has damaged or changed the machine’s WiFi, in some unknown way. Can’t see anything in system logs. More time spent trying to debug. Eventually I give up and run an ethernet cable from a Airport Extreme. More hours wasted trying to find out out why a subsystem that I have not consciously touched, no longer works, and creating a workround.

Windows just drives me nuts.

Let’s pretend that you are one of multiple players using an advertising platform. That platform is shared with other paid search outlets and has a significant volume of advertisers. You then create your own advertising platform which starts with no advertisers. What will happen?

Your cost per click from the few advertisers that you initially gain, will be lower. Even if you offer a great ROI, the number of competitors is not enough to allow the CPC to drift up, fast. So you could serve as many adverts, but you won’t get the revenue stream, until you’ve recruited more advertisers to drive up the competition and hence make the prices rise.

What brings in advertisers? A good ROI, for sure, but also users. Usage stats suggest that Live has a small but not negligible proportion of the market. Enough of a portion that advertisers might include them in the Pareto (80/20) rule to include as a target. But that’s a long way from a mandatory requirement, such as Google SERPs and AdWords. The advertiser volume is low, so ROI is good. User volume is low, meaning that the marginal cost of using the system is relatively high (if it takes the same management time to manage 10% of the business as to manage 60-80% of the business, what should you prioritise?).

That all says “low revenue” to me. If the advertising platform were rolled out to appear on more sites, then the user base would increase. Well, looky here – John’s already spotted the next installment in the thinking!

By offering more impressions, Microsoft can increase the user base. The low cost of adverts makes that attractive for advertisers. But the low revenue potential makes that a no-brainer no-way for the affiliate – why publish low revenue adverts from Microsoft when the higher competition on Google offers higher priced adverts?

So… can we expect that those people publishing Microsoft adverts will see a better share of revenue? How much share of revenue does it take to make a Microsoft advert as valuable as a Google advert? And given the difficulties that Google has had in targeting contextual match in 2006, do you expect advertisers to be happy with what Microsofts weaker search technology can do with contextual match?

I’m expecting that Microsoft will offer a greater share of the advert cost with searchers. That they won’t use Google-style “smart pricing” on contextual match and then that the contextual match will offer a weaker ROI. That’ll drive up revenues, but eat into profits. Given that Microsoft sees crocodiles anywhere it moves in the swamp, it’ll probably throw some cash to see if it can drive away competitors. Dunno that anyone has ever proven that throwing money at crocs makes the world safer, but it is a common tactic.

Is contextual match advertising revenue a problem that you can make go away by throwing cash at it? Probably not. The fundamental problems are that keyword search is relatively simple – users tell you what they want to know, and your problem is to crawl and index the existing pages and apply a bunch of filters to prevent spam sites from getting too high a rank. If you have to infer the conceptual meaning of a web page to deliver suitable adverts, then you are in a different domain of expertise. It could be that somehow Microsoft have jumped Google on this, but it seems… unlikely. So I’m expecting that advertisers will complain about the quality of site matching. I can think of an approach to beating Google, or at least matching them, but it relies on some wierd stats that are most popular in the old Soviet bloc and doesn’t rely on much of a conceptual matching system. But given that content-based publishing system is part of the Search Empire, that is an unlikely path…

So far, none of our clients have expressed any interest in testing or using Microsofts’ contextual matching. They have enough difficulty with the more mature, and improved over the 2006 performance, Google contextual matching. And they’ve seen Microsoft Live Search… Even those that enthuse about the ROI offered on Microsoft AdCenter are still using Google as their default search engine. So we have no evidence that Microsofts’ contextual search is strong or weak. But would you think that it might be strong? Care to gamble your next bonus on that?

This all suggests to me that publishers may be in for a good year, with Microsoft. If you have to burn money to grow, then buying new outlets for adverts has to be good news. It drives up usage, makes advertisers more competitive and that will ultimately drive up the revenue. But that’s a slow, uphill drive. Especially slow when the engine isn’t working properly and when you risk increasing advertiser defection through poor matching. Hard times for whoever is currently in the hot seat at Microsoft Live Search.

[Jim] Allchin [Microsoft VP] Suggests Vista Won’t Need Antivirus

During a telephone conference with reporters yesterday, outgoing Microsoft co-president Jim Allchin, while touting the new security features of Windows Vista, which was released to manufacturing yesterday, told a reporter that the system’s new lockdown features are so capable and thorough that he was comfortable with his own seven-year-old son using Vista without antivirus software installed.

(source)

Then: 6 Sep 2001

Jim Allchin: “Microsoft stamps out XP buffer overflows”

Microsoft has said it has stamped out buffer overflows with the upcoming release of Windows XP. Jim Allchin, vice president, claimed the company has done a complete code review of its operating system and removed all buffers which could overflow.

(source)