We used an AMD Athlon 64 3400+ with 2GB RAM running Windows 2000 as a host operating system and VMware GSX Server on top of that. For each test, the VM image we used was 512MB RAM and a patched up-to-date Windows 2000 Professional install. All the testing was conducted within a single day and each package had the latest updates and definition files available applied. Testing was conducted with a range of spyware as well as adware applications. Centrally managed systems, where possible, were set to monitor a separate virtual machine.
Our rationalisation methodology means we don't count cookies as individual infections. This discounts false positives. We then ensure that no single threat has been counted multiple times. Sometimes minor variances in the signatures can cause the application to register the same infection twice.
- Test 1.1 -- Accuracy, clean machine. We created clean images for each product and installed each anti-spyware application. We then ran the scans and had a look to see if anything was reported on a clean system with a fresh and patched operating system. The purpose of this was to see if any applications generate false positives out of the box.
- Test 1.2 -- Accuracy, infected machine. We created clean images for each program on test, and prior to installing the anti-spyware application we infected the machine with spyware applications and installed some adware applications. Once the anti-spyware applications were installed we ran the default scans and recorded the results, signatures detected, and instances detected. We then rationalised the results (removed false positives, excluded cookies from individual infection count, and ensured no double counting of single instances). This test will show how well the applications perform detecting spyware and adware on machines that are already infected with spyware and have adware applications installed.
- Test 2.1 -- Effectiveness. Using the same scan results of Test 1.2, we took the top three applications that had detected the most infections and instructed each one to remove everything they found in the scan from the PC. Once cleaned, we performed a scan with that application to ensure no further or residual items were found.
- Test 3.1 -- Performance clean machine. We installed all anti-spyware applications onto one clean image and ensured that all automated scanning was disabled and none of the anti-spyware applications were memory resident--this provides us with our system baseline resource usage information. We then ran the default scans recommended by the vendors and recorded the times. This test was repeated three times and the scores averaged.
- Test 3.2 -- Performance infected machine. We installed all anti-spyware applications onto one clean image and ensured that all automated scanning was disabled and none of the anti-spyware applications were memory resident. We then installed the same spyware and adware used in Test 1.2, but this time ran the default scans recommended by the vendors and recorded the times. This test was repeated three times and the scores averaged. We disabled the "Auto Clean" function of the applications to ensure we could perform repeated scans to measure the performance. We also ensured that on each pass the anti-spyware applications returned the same results.
What to look for
- Accuracy. This is key; ensure the application can detect what your enterprise sees as its biggest threat from spyware. Try to evaluate to see if resident threats are catered for as well as new incoming threats.
- Management. Smaller organisations will not be affected as much as larger organisations but take a look at your antivirus needs and you will see that anti-spyware should be closely aligned.
- Performance. This is important, especially in an organisation requiring regular scanning. The quicker the scan the less interruption staff will experience.
- Effectiveness. Test to make sure the application is capable of cleaning all instances of detected spyware from the machine, so that engineers and administrators do not waste valuable hours manually cleaning or rebuilding infected PCs.
Good to see effort was put into actually testing these products. Great Work !