The last firewall review that we performed in the March 2004 edition of Technology & Business Magazine included a basic ability test. We ran a simple leaktest which is a simulated Trojan from the inside of the network. I also ran NMAP port scans against the inside and outside of the devices and a remote vulnerability scan on each device and devices situated on the network, so that we could be assured the scan was coming in remotely.
For internal purposes there are a plethora of capable vulnerability scanning and reporting software tools and hardware appliances such as Computer Associates' eTrust Vulnerability Manager Appliance or NetIQ's Vulnerability Manager Software.
This type of basic testing is a necessary to evaluate a firewall for the specific configuration a company will have in its environment. Once purchased and deployed it is necessary to regularly run similar pen-tests (penetration tests) and vulnerability scans to highlight any new found threats and take action.
This is not to say that security administrators can take a back seat approach and let the vendors drive them. Businesses should regularly stage and perform their own pen-testing of their network devices and resources to ensure that they are up-to-date and their network is as secure as they can make it at that point in time.
The next test that can be performed on these devices is a loaded throughput or performance test, while the lab is more than capable of performing this type of testing, the variance in potential environments, devices and configurations hinders a truly comparative performance test. At the end of the day each company will have their own varying amount of filtering policies, procedures that need to be applied, not to mention unique network load -- a performance test in this instance would be purely academic and of no real relevance.
We therefore decided to setup and run the individual firewalls and take a look at the devices and their management and identify strengths and weaknesses with an emphasis on the unique features between vendors. Also to take a look at the logging and reporting systems of each vendor and see how well they interoperated with third party reporting and analysis systems.
The benefit of having our own public facing IP range enabled us to set up each device on an open address and log all the traffic against that machine from the outside -- Script Kiddies running port scans and various other foot printing tools.
How can you test all firewalls and leave the marketleader out ? All these are toys ! :)checkpoint rules