 |
||||
|
|
|
||
|
Contents 
Introduction
Mercury Load Runner
IBM Rational Tester
Segue SilkPerformer
NeoLoad
Specifications
How we tested
Editor's choice
About RMIT |
|
||
|
|
||||
|
|
||||
Mercury is without a doubt the biggest player in software automation. They won more than 50 percent of the market with the company recording revenues of $685 million last year. None of the other vendors in this space even came close.
We had a Systems Engineer come to the TestLab to help install Load Runner, but it's actually a piece of cake (it was the only way Mercury was going to be involved in this evaluation). They also recorded the scripts for us, played them back, found the faults on our Web server, did the analysis and then created a report for us. They pretty much did our job then uninstalled Load Runner from our PC before leaving the Test Lab.
This made it a bit hard for us to run some more tests later on as we were restricted to five users on the 10-day trial version you can download from their site. But luckily we've used Load Runner on a number of occasions before so we can make some fair comments on this product based on previous experience.
Load Runner gives you three main options when you first start the application. You can choose to create/edit scripts, run load tests or analyse load tests. Clicking on create/edit scripts opens up the Mercury Virtual User Generator, which gives you a variety of protocols or technologies to choose from. We chose the Web (HTTP/ HTML), which emulates the communication between a browser and Web-server.
We opened up the controller, which is used to execute and manage the test scripts. From the Design screen we were able to set the run time settings like the number of iterations we wanted the script to run through.
From the Controller we set up a schedule for the load test, deciding to take the ramp up approach. There are other options available like ramp down or load all users simultaneously, run the test for a set duration, or set the users to run indefinitely. Again there is a good amount of flexibility here.
Clicking the start scenario button kicks off the virtual users and from the run screen you can watch in real time where the tests are up to and how your system is performing under the load. The layout of the run screen was excellent. Every bit of information you need to monitor is on this screen. We were mainly concentrating on the response times for our server. By clicking on these graphs or on a particular transaction we could drill down to see what was causing the long delays. You can get down to the method and even SQL call.
We could quickly see that after 10 users the performance of the system dropped dramatically, and after examining the information that was coming back from the Apache monitors we realised the number of threads was limited on the Web server. We were able to correct this configuration and after we ran the test a second time discovered the database was a bottleneck. More is explained about this in the "How we tested" section.
The next step was to analyse the results. We were able to build custom graphs, adjust the granularity, scale, apply our own filtering, and create a Web page breakdown of the tests. We annotated the graphs and saved the results as a template so next time we wouldn't have to customise the report again. We could export the results into Word or in HTML.
The great part was how each of the graphs was supported by a brief explanation. The report also provided a glossary of the technical terms used.
Mercury indicated that there are options for more comprehensive type reporting but we were impressed with what we saw in the standard package.
|
| ||||||||||||||||||||||||||
I think you missed iMacros in your review. We use it to test our websites. It's great! It has by far the best support for AJAX and Flash applets and the scripting interface is extremely powerful.