How we tested
 |
|
|
|||
| |||||
To test participating vendors' anti-spam software applications and their own technical configuration of the same in an effort to provide some statistics on the effectiveness of the current technologies ability to successfully filter desirable and undesirable e-mail messages.
How it worked
We provided each vendor with a server running Microsoft Exchange Server 2000 on Windows 2000 Server. All vendors were invited to send a technician to the Labs on the same day to install and configure their own products and the Exchange servers. The technicians could either install their product on the same server or on a separate server also running Windows 2000 Server.
All test servers were connected via the same switch and the Lab gateway to the Internet. Each server was allocated a pre-defined static publicly accessible IP address and each mail server was assigned a fully qualified sub-domain name. Each server was also assigned an external e-mail account.
Vendors were encouraged to implement their rule sets so they were "tight to catch as much spam as that package is capable of", but not too tight to block everything--false positives were to be avoided as much as possible. The use of current black and white lists was acceptable. Once the install/configuration period was over, the vendors were not allowed to see or access their systems again.
Static tests
A pre-collected set of over 1800 e-mails were sent from a few static mail accounts, and the results collected. We ran through these tests several times to ensure their accuracy, and to determine the effect, if any, of the packages' learning capabilities.
We used a Microsoft internal testing tool to send the static control messages to the servers. This tool was initially developed to test mail servers under load. We adapted its use to allow us to take messages that we have collected (provided their headers have not been corrupted) and then send with original or new headers.
The scores you see are based on the results of these static tests, although they are very similar to the results achieved in the live tests as well.
Live tests
Following the static tests, the packages were run on a number of live external mailboxes, providing a live test scenario with real-world mailboxes and real world external mail servers.
We used a Linux-based Sendmail server to combine all messages to a single account, then forward them to the multiple test accounts, which left the headers as if the messages had been sent directly from the spammers.
After running these two tests using the vendor's suggested configurations, we spent a bit of time altering the vendors' rule configurations to see if tweaking the products could alter their results. Although this did not contribute to the overall scores, because of the subjective and human factors involved, it gave us some valuable information on the ease of use and effectiveness for administrators who will need to constantly tweak the systems once they are in use.
A note on results
Any results achieved by this testing only have a limited lifespan before becoming redundant, due to the nature of the applications themselves, and the dynamic nature of the spam they are trying to filter. The bad guys will always be trying new ways to get around the good guys' defences, which the good guys will always be trying to improve.
A note on servers
Due to the large number of servers required for this test, since each vendor required up to two servers for their products, we were unable to provide enough identical servers to run the tests. We therefore hired all the servers from PC Hire. Each vendor contributed the costs of hiring their own server(s), and we'd like to thank PC Hire for helping us keep these costs to a minimum. We'd like to make it clear that--as always--vendors were not charged to participate in the tests, and were only required to pay the cost of the server hire.
Test bench
- Interoperability: Will the software work with your existing mail server and operating system?
Futureproofing: Is the software accurate and flexible enough to suit your needs into the future?
ROI: Does the price justify the accuracy and will you achieve productivity gains by using the software?
Service: What options are available for service and support, and how much do they cost?
- Easily deployed packages that work with your existing mail applications (mail servers and mail clients)
- For "self managed" solutions look for apps with the ability to create, manage, and manipulate quite complex rules
- Evaluate, evaluate, evaluate. If you are a larger organisation, most of the filtering vendors on the market would be more than happy to arrange a demo/test in your given environment, this should assist you in making the decision that is right for your enterprise.
Final Words
Naturally with all these applications, a system administrator can configure and change the rules and create custom rules also to adapt the system to their own particular organisation. Therefore it is not sufficient to take this accuracy test on its own, I would still advise any enterprise looking to implement any spam filtering application to perform their own tests in-house on any of these or the many other applications available to filter spam. Spam is certainly a hot topic at the moment, and doesn't look at being easily remedied in the near future. The innovations and implementation of each of these applications is quite impressive and very exciting--certainly much more involving than your average word processor or antivirus app.
Subscribe now to Australian Technology & Business magazine.
I am interested in knowing how the market is for spam filters in Australia, I work for FrontBridge Technologies (in Marina Del Rey, California, U.S.A.) and I recently followed up on an inquiry on us from a Sydney based company. As we spoke they mentioned that there are no perimeter based solutions doing business in Australia like ours.
Just curious and thought the author of this article might like to discuss.