Interoperability
Is the software compatible with your existing operating system?
Futureproofing
Is the software accurate and flexible enough to suit your needs into the future?
ROI
Does the price justify the performance and will you achieve productivity gains by using the software?
Service
What options are available for service and support, and how much do they cost?
How we tested
Aim
To test participating vendors' anti-spam software applications and their own technical configuration of the same in an effort to provide some statistics on the effectiveness of the current technologies ability to successfully filter desirable and undesirable e-mail messages.
How it worked
We provided each vendor with a server running Microsoft Exchange Server 2000 on Windows 2000 Server. All vendors were invited to send a technician to the Labs on the same day to install and configure their own products and the Exchange servers. The technicians could either install their product on the same server or on a separate server also running Windows 2000 Server.
All test servers were connected via the same switch and the Lab gateway to the Internet. Each server was allocated a pre-defined static publicly accessible IP address and each mail server was assigned a fully qualified sub-domain name. Each server was also assigned an external e-mail account.
Vendors were encouraged to implement their rule sets so they were "tight to catch as much spam as that package is capable of", but not too tight to block everything--false positives were to be avoided as much as possible. The use of current black and white lists was acceptable. Once the install/configuration period was over, the vendors were not allowed to see or access their systems again.
Static tests
A pre-collected set of over 1800 e-mails were sent from a few static mail accounts, and the results collected. We ran through these tests several times to ensure their accuracy, and to determine the effect, if any, of the packages' learning capabilities.
We used a Microsoft internal testing tool to send the static control messages to the servers. This tool was initially developed to test mail servers under load. We adapted its use to allow us to take messages that we have collected (provided their headers have not been corrupted) and then send with original or new headers.
The scores you see are based on the results of these static tests, although they are very similar to the results achieved in the live tests as well.
Live tests
Following the static tests, the packages were run on a number of live external mailboxes, providing a live test scenario with real-world mailboxes and real world external mail servers.
We used a Linux-based Sendmail server to combine all messages to a single account, then forward them to the multiple test accounts, which left the headers as if the messages had been sent directly from the spammers.
After running these two tests using the vendor's suggested configurations, we spent a bit of time altering the vendors' rule configurations to see if tweaking the products could alter their results. Although this did not contribute to the overall scores, because of the subjective and human factors involved, it gave us some valuable information on the ease of use and effectiveness for administrators who will need to constantly tweak the systems once they are in use.
A note on results
Any results achieved by this testing only have a limited lifespan before becoming redundant, due to the nature of the applications themselves, and the dynamic nature of the spam they are trying to filter. The bad guys will always be trying new ways to get around the good guys' defences, which the good guys will always be trying to improve.
A note on servers
Due to the large number of servers required for this test, since each vendor required up to two servers for their products, we were unable to provide enough identical servers to run the tests. We therefore hired all the servers from PC Hire. Each vendor contributed the costs of hiring their own server(s), and we'd like to thank PC Hire for helping us keep these costs to a minimum. We'd like to make it clear that--as always--vendors were not charged to participate in the tests, and were only required to pay the cost of the server hire.
Company: Michalak Manufacturing. This company wants to roll out antivirus protection to its corporate desktops and servers.
Approximate Budget: Open.
Requires: Antivirus software for desktops and servers, and deployment software to manage the distribution of definitions.
Concerns: The company is most concerned with the ability to centrally manage the software and to control the distribution of virus definition files so that systems are upgraded as quickly as possible. The software's ability to block viruses in desktop e-mail clients is also a big concern.
Trend Micro Antivirus
While arguably not the most powerful enterprise AV package overall, Trend Micro's OfficeProtect Corporate Edition is nevertheless feature packed and very flexible. What sets OfficeProtect--and its attendant applications ServerProtect and Control Manager--apart when compared to the other packages reviewed is the wonderful simplicity of its interface. The Web-based user interface for product deployment is so easy to drive that for basic rollout functions, and many of the complex functions as well, the documentation can be ignored completely. This cannot be said for any of the other packages tested.
eTrust AntiVirus V7 is also a very powerful and easy-to-use package that may also be worth a close look, and it gets an Honourable Mention.
Subscribe now to Australian Technology & Business magazine.
