John Naughton: Google's worrying wizardry can make you invisible

On June 24, Nicholas Carr, a prominent blogger, author and technological contrarian, woke up to find the contents of his widely read blog (www.roughtype.com) had apparently vanished.

If anybody typed "google site:www.roughtype.com" (the syntax you need to get Google to search for a specific site), they got the message "Your search did not match any documents".

The entire contents of his blog had been erased from Google's index. "Every post," wrote Carr mournfully. "Every last bon mot. Gone. Without a trace. I'm on my knees. Please, Google, I beg of you, let me back into the promised land. I swear I'll never use Bing again."

Because Carr is an uber-blogger, someone at Google noticed and wrote to him.

"It looks like your site was removed because it has been hacked," explained a member of the company's search team. "We tried to send you an email last Friday with information on what happened, but it was difficult to find a contact." Further investigations were launched and eventually an explanation emerged.

Apparently, Carr wrote later, "there were some hidden spam links [to various lame mp3 and lyrics pages] at the bottom of the index page of an old blog I had called Rough Sort that's been dormant for a couple of years [it was set up as a separate Movable Type blog but within the Rough Type domain]".

"I don't know how the links got there, but apparently it may have been through sort of hackery.

"Anyway, I rebuilt the page, the offending links disappeared, and now Google says I'm once again fit for consumption. I should come back into existence in the next few days. Whew."

And he is indeed back, ploughing his contrarian furrow. But this episode is a sobering reminder of the power Google now wields. Most of the time we don't think about this because the company provides such a useful service to the average web user.

But someone who runs an online business sees it very differently. In The Search, an excellent book about Google's irresistible rise, author John Battelle has a tale that illustrates this.

In 2000 Neil Montcrief founded a small e-commerce business (www.2bigfeet.com), selling outsize shoes on the net. For a time, the business did well because of the traffic Google drove to the site. But in November 2003, everything changed. Traffic to his site shrivelled and cash flow plummeted.

But Google had not targeted Montcrief: the vaporisation of his little business turned out to be just collateral damage in the ceaseless war between Google and the armies of people who try to "game" its search results. As part of its strategy to beat such people, Google periodically adjusts its algorithms so the gamers are thrown off balance.

As a result the web is swept by regular "Google waves", some of which can drown businesses like Montcrief's.

As it happens, 2bigfeet.com eventually weathered the storm. But the story illustrates the consequences of having a dominant search engine such as Google. In a networked world, it can effectively wipe you out because if people can't find you on the web, it's as if you don't exist.

This is real power - and at the moment it's completely unregulated.

Which brings us to Microsoft and its new, improved search engine, Bing.

It seems to me it's steadily improving. Which is promising because it now looks as though there are only two ways of curbing Google's dominance: via the US Department of Justice or via Bing. Not much of a choice, is it?

- OBSERVER