KEY POINTS:
Since about 1995 there has been an ongoing argument in our house. I maintain the internet is one of the most important inventions ever and that its effect is far more significant than that of the printing press. It's a view that sees the net as the most devastating weapon for democracy of all time. Whenever a new web trend - blogs, YouTube, myspace - hovers into view, I push home my argument with sophisticated logic: "See I was right, now do you believe me?"
My partner remains unconvinced. I point out that unlike the printing press, which allowed a few to publish and a lot to read, the net allows a lot to both read and publish plus broadcast.
"And that's a good thing?" she retorts. "Everyone publishing and broadcasting an increasing pile of rubbish."
I try to explain that yes, a world where everyone publishes/broadcasts - spews forth in an uncontrollable torrent - is indeed a good thing.
Why? Because everybody gets - or rather has the freedom - to have a say. Whether it's the equivalent of talkback radio or reality TV doesn't matter. The point is they can.
When mass multifaceted discourse thrives, democracy is alive. My partner counters with an eloquent. "You're talking complete crap."
It was shortly after one such fruitful interchange that I read Web Dragons: Inside the Myths of Search Engine Technology (www.webdragons.net) and found I had to reassess my position.
The book, written by three academics including Waikato University's Ian Witten, pulled me up with a start on page 146: "Likewise, those trawling the web for information will meet nothing but disappointment if it becomes infested with low-quality documents that stifle everything else with their overwhelming rate of reproduction."
Witten et al were comparing the riff-raff invasion of the web with that of the killer lobster which so devastated the ecosystem of the Versilia area off the coast of Pisa in the early 1990s.
The argument goes like this. Unless we want a lobster-only diet, the web must have diversity. But it also needs a way to access the knowledge. Herein lies the problem. The librarians of the web are not people, but machines - search engines, also known as web dragons. And "it is they who fight spam, the killer lobsters of the web".
Spam, in this instance, is not the hordes of junkmail we all get in our inbox, but web pages created for the sole purpose of attracting search engine referrals. In 2004 it's estimated up to 15 per cent of web pages were spam and as Web Dragons points out "once indexed by search engines, they pollute their results".
So begins an endless game of cat and mouse - spammers employing tricks to increase their web visibility and the search engines reacting by changing their page ranking algorithms to cut spam. In the process there's always collateral damage.
As Dragons puts it: "Sometimes one person's spam is other person's ham."
Witten calls search engines dragons because they are "the gatekeepers of our society's treasure trove of information". It's a good metaphor because dragons, like search engines, are magic, seemingly independent and unpredictable. What worries me is that control is in the hands of so few - uber-dragon, Google, deputy Yahoo and hangers-on MSN Search, AOL Search and Ask.
Because of the spammers all the dragons keep their formulas for assigning page ranking secret.
But what's also secret is how much these page ranks are influenced by commercial and political agendas. We do know that in the search engine economy the most prestigious pages are the ones that are most popular.
So on the one hand the hoi polloi threatens to overrun the web with rampant banality. On the other the search engines select from the great unwashed and present pages on the basis of their popularity.
Which brings us to an outcome Web Dragons seems a little unhappy with. "In a global village that spans the entire world, the tyranny of the majority reigns."
But as I see it, the more people publish the greater the diversity - even if much is of the low-rent kind. Hopefully that's enough to keep the killer lobsters at bay.
