 |
||||
|
|
|
||
|
Contents 
Introduction
Mercury Load Runner
IBM Rational Tester
Segue SilkPerformer
NeoLoad
Specifications
How we tested
Editor's choice
About RMIT |
|
||
|
|
||||
|
|
||||
We setup two Pentium 4 2.6 GHz machines with 1GB of RAM running Windows 2000 Professional. These machines were also connected to each other through a 10/100 switch.
On PC 1 we installed an Apache Web Server, which powered a mock travel Web site. While on PC 2 we installed each of the Load Testing Tools. We had the vendors send down an engineer to help us install and run up some basic load tests just so we could shorten the time spent to learn each product. From the mock tours Web site we registered a couple of user accounts. These accounts allowed us to later book a flight through the tours system. When creating a new account we had to fill out a form with some basic information about ourselves such as first name, last name, phone, e-mail, address, city, state, post code, country, and we choose a user name and password. Once we set up these users we were able to book a flight. We did this using the record feature in each of the applications. Hitting the record button on each of the tools launched an IE browser, which we used to navigate to the destination URL.
We could then see the main page on the tours Web site -- we logged into the system and booked a return flight from Sydney to San Francisco. We then filled out our credit card details, confirmed the flight, and logged out of the system.
Up until now it was quite easy, so to make it more interesting we edited each of the test scripts or recorded sessions and looked at the way each of the apps displayed the script and checked to see correlations they made.
We parameterised the user name and password and randomised the credit card field. We sourced a list of user names and passwords, which the script used to simulate a large number of different users logging into the tours system.
As part of our test we had to decide how to place the load on the test server. The vendors each have their own opinion on how this should be done. We took the ramp up approach where we slowly increased the load by one every 10 seconds until we got to 50 users, then we ran the test for another five minutes.
We quickly discovered that we only had to simulate 40 users before we found the tours Web site started to fall down. We didn't get any failed transactions,which was good, but the delays were too long to consider this site acceptable for 40 users to use at the same time. The tours Web site uses flat files instead of a proper database to store booked flights -- this became the bottleneck. It particularly occurred when we had to secure the purchase of the flight or when the application was writing our flight details to the flat file that the response times increased to around about 70 seconds. Things like logging in and getting to the main page took up to 45 seconds.
Before we moved on to pinpointing the database as being the bottleneck we preconfigured the Apache Web Server so it would only accept 20 concurrent threads (ie. requests) at any given time. In Internet Explorer, when one window is open it uses two threads, so this means that only after 10 users will the system start to drop off in performance, leaving many users would get stuck on the main page. This is exactly what happened to us -- there was a large delay trying to access the main URL. As more users entered the mix, the delays became longer. We put this in there just to see if the vendor's software could pick up this problem -- we also ran the tests with 200 supported threads and checked out the reporting. We looked for any unique features that separated them and evaluated the platforms and type of applications they can support, the general setup of the controller, virtual clients, monitors, and ease of use.
I think you missed iMacros in your review. We use it to test our websites. It's great! It has by far the best support for AJAX and Flash applets and the scripting interface is extremely powerful.