 |
||||
|
|
|
||
|
Contents 
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT |
|
||
|
|
||||
|
|
||||
To do a complete and thorough live accuracy test would take at least two to three months. It would involve the setting up of two concurrent running mail servers and applications per product on test, one set to default vendor baseline static -out of the box" settings and the other as a dynamic -tweakable" system to ensure that benefits were being derived on a day-to-day and week-to-week basis between the static and dynamic machines. So for the 11 products in this review, 22 mail servers would be required.
We would then have to select the combination from our live honeypot domains which would provide the best mix of unique spam messages for the testing.
Honeypots are live mail servers with valid domains and user accounts that we constantly have running, which attract and collect spam. It takes considerable time to build these honeypots up as you can't simply subscribe or add your e-mail address to the spammers database -- then you are inviting or entrapping the spammer and those messages would have to be classed as -grey" not -spam".
We have to ensure that our test e-mail accounts are harvested via normal spammers means, like domain and address harvesting from live Web sites, name database additions etc. Our honeypots have a history of almost three years now.
Once we had our live spam feed we then need to inject a live ham/grey mail feed too. We have modified a centralised mail server to enable us to perform this initial combination then aggregation of the message stream to ensure that each mail server receives exactly the same feed live from the Internet to one address on the mail server as though the message had come straight from the source to the final destination.
This is very difficult to achieve particularly considering that many anti-spam applications rely on the original e-mail header information being intact. Something that mail applications like Novell Groupwise, Microsoft Exchange and Outlook do not do.
So while this is all good and well, we would then have 22 mail servers and anti-spam applications up and running with live feeds of spam, ham and grey mail. We would then set up a machine or groups of machines to POP the e-mail messages from the respective servers using Outlook Express. OE keeps the headers intact for later reference or use via our controlled/static test, yes once the live testing is over we have developed a methodology for mass -re-testing" under a controlled environment.
Once the messages are in OE the hard part starts sorting out the missed spam, the canned grey mail, and dare we say it the false positives. This is the most labour resource intensive part of the testing. I couldn't bear to imagine how many hours/days it would take to go through 22 servers results every week for two or three months.
So the resources like time, budget and labour are against us to complete such a test for this review.
We can however let you in on a few -generic" results derived from the various private anti-spam product testing contracts that we have completed in the past 18 months.
This spam testing experience has shown us that most of the applications we have performed private testing on in an -out of the box" configuration rate at around 65 percent to 70 percent spam catch accuracy with very low to zero false positives. You may well appreciate as with most things in life there are no two e-mail environments which are identical, therefore the anti-spam vendors have a difficult time deciding on default baseline settings to achieve the best spam catch rate with a low to zero false positive rate for the widest possible base of users.
Administrators should expect a certain amount of fine tuning or tweaking to achieve higher catch rates and possibly lower false-positives depending on the industry . Imagine applying an anti-spam filter if you were in the pharmaceutical industry or the porn industry! Both naturally require e-mails to be sent and received containing information about their respective businesses.
A false positive as its name suggests is a legitimate e-mail message, (ham), that a recipient should receive that gets filtered incorrectly and flagged as spam. This is particularly bad if the filter has been set to drop all messages determined to be spam as it means that ultimately the recipient would potentially have no idea that the message was ever sent.
To add another level of complexity to the mix there are also messages which fall into the area between ham and spam these are generally classed as -grey" mail and are the newsletters, circulars etc that some recipients may subscribe to and need to receive. These have many common characteristics of spam messages and are very hard to filter correctly. So at the end of the day there are three bodies of messages to be concerned with, spam, ham and grey. In a perfect world all spam should be dropped, all ham and grey mail should be delivered.
With a few weeks of concerted tweaking and testing by the mail administrator, we managed to increase the spam catch rate to somewhere between 85 percent to 92 percent while still maintaining a zero false positive rate for most of the applications that were tested in private.
Grey mail on the other hand is a very different beast. The best ways we have found to deal with the sensitivity of the filters in respect to these messages is to include them explicitly on a white list.
Why not just use the 300 products for Linux?