Spammers being driven out

If you are still getting large amounts of unsolicited commercial emails in your in-box, blame your internet service provider, blame your boss, because the problem is solvable.

That is the claim from Auckland-based Richard Jowsey, creator of death2spam.

Jowsey says the amount of spam as a percentage of all emails seems to be stabilising at around 50 to 60 per cent of all emails, although some well established addresses may be getting much higher noise to signal.

"We seem to have the spammers on the run. The tide started to turn about six months ago and they are starting to give up now. It's too hard," Jowsey says.

It's a matter of economics. Send out millions of emails using stolen servers, stolen bandwidth, and if 10 people per million send off their money for phony viagra or counterfeit, virus-ridden software, you are making money. If filters cut out 90 per cent of your messages before they get to the mugs, your profit margins disappear.

"Third generation statistical filters are very hard to beat. A well trained system like death2spam is seeing to a 99.5 per cent or better capture rate with better than one in 1000 false positives," Jowsey says.

"We are also seeing a psychological effect which is the tipping point where the accuracy is so good, people just hit the delete on spam, they don't train the system any more."

Jowsey's system uses Bayesian filtering, a branch of statistics which allows the computer to learn what is a junk message, a virus worm or other malware and what may be legitimate.

With servers around the world filtering millions of messages a day, the system can learn pretty quickly what people want to read and what they delete.

Statistical filtering has superseded rules-based systems, which the spammers worked out how to get around two or three years ago.

The two useful tests of a spam filter are the capture rate, what is stopped, and the false positives, how many of the messages stopped shouldn't be.

Jowsey says for a home user, a false positive rate of one message in 1000 is probably acceptable. For a business he sets one in 10,000 as a benchmark. "It is more likely to be mission critical.

Symantec, one of the biggest anti-spam firms through its acquisition of Brightmail, claims a 95 per cent effectiveness rate and less than one false positive in every million messages - the price of having five per cent of junk mail getting to your desktop.

MailWasher, a spam filter developed in Christchurch, also claims a 95 per cent capture rate and negligible false positives.

MailWasher developer Nick Bolton says he aims his product at the consumer market, with a free open source server-side version for businesses.

MailWasher uses Bayesian filtering, community filters where if someone clicks the button saying something is spam, a fingerprint is taken so no-one else on the system gets it, real time blacklists which block mail from servers known top be used by known spammers, and other filters.

Another spam product used widely in New Zealand is Mail Marshall, which was developed here and sold to United States company NetIQ.

However, it is still primarily what Jowsey calls a second generation, rules-based filter, with some newer technologies bolted on.

Jowsey ignores the consumer market, aiming what he calls his Rolls Royce system at internet services providers and large companies.

Many of his customers are around Seattle, where he used to work, and the San Francisco area. In New Zealand it is offered as an option by Iconz and used by Harcourts Real Estate and other corporates.

Jowsey says even though people complain about spam, it can be hard to convince them they need a filter.

"The ISPs say they can't charge extra for spam filters because it is not a profit centre, so they get something cheap and nasty and say they have a spam filter," Jowsey says.

"The end result is a system which does next to nothing. Few ISPs give a damn about accuracy and capture rate."

That is why Jowsey prefers to concentrate on enterprises with more than 1000 email users, where he can make a business case for filtering.

"The average spam takes eight seconds for the user to delete, and the average business we see is getting at least 50 per cent spam.

"At eight seconds, 100 spams is 800 seconds, so typically a person has to spend 12 to 15 minutes a day clearing spam. Multiply that by a few hundred employees, and there is a business case for a specialised system."

Jowsey says anti-spam technology will come to be seen as part of the plumbing of the internet.

"It's like the filter in the kitchen sink, it is going into the walls of the web."