Are spam 'blocklists' going too far?

Like a growing number of Web surfers, Audrie Krause faces a new uncertainty when she hits the send button on her e-mail these days: Will the message get through?

As the head of a political action group, Krause uses members-only e-mail lists to help educate and organise fellow activists. So she was jarred recently when one message bounced back with a note accusing her of spreading unsolicited junk e-mail, or spam.

Without warning, Krause's NetAction site had been blacklisted--an increasingly common occurrence as companies seek to block crushing loads of unwanted e-mail by any means necessary.

"It's ironic because the work we do as an organisation includes helping get the message out to other activists and nonprofits about how to use e-mail and the Net for outreach...without spamming," Krause said. "I'm sure it was a mistake."

The incident, which was fixed within a day, highlights a growing problem for ordinary e-mail users now that sometimes-indiscriminate blacklists have become a key weapon in the war against unsolicited bulk e-mail.

Blacklists--also known as blocklists--keep tabs on sites and numeric IP (Internet Protocol) addresses suspected of sending spam. Internet service providers, companies and individual Web site operators subscribe to the lists, bouncing any traffic directed to their servers that originates from those addresses. The result is that all blacklisted e-mail--legitimate or not--is returned to the sender.

Blacklists are as old as the Internet, but their number has multiplied in recent years. Many on the receiving end are now adopting tougher policies as spam has grown to epidemic proportions. At the same time, more companies and Web site operators are using blocklists as a mainline defence against vast volumes of spam that can cripple their systems if left unchecked. The need is so great that some companies now are turning a blind eye toward militant tactics that may do too little to sort legitimate from illegitimate sites.

"Almost every company now is looking at using blocklists because there's no choice--there's too much spam coming in," said Steve Linford, who maintains a London-based blacklist of mass e-mailers called the Spamhaus Block List. "The blocklists need to be run with an amount of responsibility and ensure that if any innocent user is caught on a blocklist there's a means to get off quickly."

Spam invasion
Most people are enraged by the exponential growth of spam in the past year but baffled when it comes to looking for answers. Worldwide spam attacks have grown by nearly five times in the last year, from about 1 million last June to just under 5 million this year, ISP filtering company Brightmail noted in a report published this week.

Part of the problem stems from the economics of e-mail, which provides no incentive for marketers to cap the volume of messages they attempt to deliver.

Blocklists such as Spamhaus, the Realtime Blackhole List, SPEWS and SpamCop.net have grown as a response to the resulting flood. But they are increasingly coming under fire for high incidents of "false positives," in which non-spammers are added to the lists.

Recent complaints about blocklists have come from companies and organizations, including British Telecom, the Libertarian Party and ZDNet News publisher CNET Networks, among others.

In general, blocklists are simple databases of spam-generating IP addresses. Most use the DNS (domain name system) protocol to block a IP address in real time so that if a number is added it will have an immediate effect on spam delivery.

The blocklists rely heavily on each other to locate spammers and create their lists. Many lists go to SpamCop to see if a piece of e-mail has been reported and to determine the offending IP address. Others use a Usenet newsgroup called news.admin.net-abuse.sightings (NANA) to root out sources. Once the mail is verified as spam, the blocklist will add its originating IP address and, typically, that of any Web site advertised in the message.

While the blocklists target spammers, legitimate sites such as NetAction.org can easily be caught in the net.

Sites may find themselves on blocklists because of e-mail viruses or other tricks that spammers use to "spoof" or mimic addresses. The Klez virus, for example, caused at least one site to be listed by mistake on Relays.osirusoft.com, according to Joe Jared, who runs that list.

Jared operates a blocklist database that carries SPEWS and other spam listings.

Organisations running the blocklists have different policies for adding an IP address to the list. But many are now adopting an attitude of list-first-ask-questions-later, capturing an ever-widening circle of suspected offenders, guilty or not.

Jared, for one, downplayed concerns about catching legitimate e-mail, saying that if an e-mail "looks like spam and it smells like spam, then it will get listed."

Room for mistakesSpamCop, which started in the last year, this week incorrectly listed the main e-mail hub for British Telecom, ruffling a few feathers. Because the system is automatic and doesn't use a person to flesh out whether an IP address belongs on the list, it can mistakenly add a company, according to operator Julian Haight. In British Telecom's case, its mail hub had an inconsistency in its DNS information, which caused the listing. Haight corrected the mistake by listing the individual spammers on the telecommunications company's network.

"Every form of filtering has false positives. As soon as you start to use filtering, you accept that you're going to block some legitimate e-mail; it's just a question of how much," Haight said, who advises site operators to give their users a choice about blocking.

"People in the past were opposed to filtering at all, but more and more system administrators have to be aggressive because they have no choice."

He said that if innocents are listed, it takes a week to become automatically de-listed.

One of the most controversial tactics involves adding entire ranges of IP addresses to a databases, even when it's clear that some legitimate Web sites may be affected--an outcome dismissed as "collateral damage" in the trade.

Some militant blocklists have been accused of actively using collateral damage as a tool to spur legitimate sites into the battle against spam.

Magdalena Donea, a system administrator at Web hosting company KIA Internet Solutions, found a set of her company's IP addresses blacklisted recently on SPEWS. She successfully lobbied to get the listing removed, but it was re-listed a second time with additional IP addresses, a move that also affected a company client, the Libertarian Party.

"The SPEWS system is unapologetic about false positives and even regards them as a plus. They've taken the 'ends justify the means' argument way farther than I've seen anyone else take it," Donea said.

"Their philosophy appears to be that if innocent businesses and individuals on the periphery of spam-house blocklists are affected, then those innocents will have no other choice but to pressure their upstream provider to remove the spammers from their blocks, thereby solving the spam problem bit by a bit. Draconian, yes. Effective? Sure."

The people who run SPEWS are anonymous and could not be reached for comment. Many blocklist operators seek the shadows because they are constantly slammed with complaints and requests for addresses to be removed.

"We get harassed all the time," said Relays' Jared. But he added that blocklists are winning more converts every day.

"There are lists that are very hard core and lists that are very liberal," he said. "But basically the tolerance for spam is decreasing in direct proportion to the increase in spam."