Test results
Test bench
Interoperability
How well will the server work with your existing applications and other systems?
Futureproofing
Performance and upgrade capabilities, and redundant components to improve uptime.
ROI
Just the age-old price, performance, and features.
Service
Decent warranties and prompt service availability at a reasonable cost.
How we tested
Firstly, some more insight into the variety of machines that were supplied and the difficulties associated in doing a comparitive performance test. We received units from Apple (a dual G5 running OS X Panther), AMD (a quad Opteron running Windows 2000 Server), Ipex (a quad Itanium 2 running Windows 2003 Advanced Server 64-bit) and Sun (dual UltraSPARC IIIi running Solaris 9).
It should also be noted that NEC was more than happy to submit a server for us to test, but the company had sold out of its quad-capable servers at the time and were on backorder. While NEC offered to put one together from a spare parts kit, we made a mutual decision not to go to that extent, so full marks go to NEC for assistance in the face of deadlines.
Testing worries
Now for the conflicts with the testing, for many years now CPU manufacturers such as Cyrix, AMD, and IBM have been decrying the clock speed (MHz and GHz) ratings of CPUs as mere numbers which have no relative value to the true outright performance of the system, or indeed the CPU itself. We remember the outcry when AMD started shipping CPUs labelled 1100+ and 1300+ as an indication of their performance even though the actual rating was quite slower.
Until relatively recently, Intel has been pushing the clock speed as the be all and end all. I say recently, because Intel is now going the other way with its mobile CPUs--the latest 1.3GHz version we recently tested beat the previous 1.5GHz model through the use of a larger internal cache. While it is true a system's performance is only as fast as the weakest link in the chain--for instance bus speed, video speed, I/O rate, memory, or CPU--with servers there are many other factors to take into consideration. So with this test, given the disparate array of processors and operating systems, we decided to set up a test rig to hammer the servers using one application that they should all be more than capable of running and that the client systems could hit in a similar uniform fashion. That, folks, is a Web server application.
Yes, we have already just finished explaining that these servers are far and beyond the relatively menial task of serving out Web pages, however for the purposes of this test a Web server and its very cross-platform
Naturally our test rig did not entail simply downloading a page via a 56Kbps modem. We borrowed a 12-port gigabit Ethernet switch from Dell, which allowed us to connect 10 client systems each running a 100Mbps card. Therefore, with the controller system and the test server on the switch at the same time, we could run WebBench and get almost 100 percent utilisation over the test server's gigabit Ethernet network link.
Each of the 10 client systems could simulate up to 20 virtual clients hitting the server simultaneously with thousands of multiple transactions, so we effectively had a network maximum of 200 clients connected to the gigabit Ethernet NIC on the server and capable of hitting it with hundreds of thousands of simultaneous Web transactions per second.
We then took the results data collected by the controller system of WebBench and translated them to match the clock rate of the CPU. Similar to a kilogram per kilowatt (Kg/Kw) performance rating for cars, we obtained a request per second per million clock cycles (MHz). The kilowatt per kilogram rating for a car is a virtual equaliser that shows while one car may have a five-litre V8 engine producing 300KW and another car may have a three-litre six-cylinder engine producing 130KW. Most rev heads would say give me the 300KW car anyday as it is obviously far more powerful than the 130KW car. However if the car with the V8 weighs near 2300kg and the six-cylinder vehicle only weighs near 800kg then in reality the 130KW vehicle has a power-to-weight ratio of 6.15kg per KW and the V8 vehicle translates to a hefty 7.6kg per KW. In reality, the performance on paper when compared in this way is more realistic because it takes the weight into consideration. Sure there are several hundred variables and arguments that can then be brought up given that scenario and we are a technology magazine not a motoring magazine. However, we are purely using this as an example of where we are going with this server comparison, given the major physical hardware and software differences between these platforms that these vendors are proposing can all perform the same or similar high-end server tasks.
| Test Results (RPS divided by quoted/rated MHz = RPS/MHz ratio) |
||||
| System | Requests per second | Clock speed | Number of processors | RPS/MHz |
| AMD | 8213.82 | 2000MHz | 4 | 1.02 |
| Apple | 1339.04 | 2000MHz | 2 | 0.33 |
| Ipex | 7629.56 | 1200MHz | 4 | 1.58 |
| Sun | 1751.22 | 1000MHz | 2 | 0.87 |
As you can see from the results, the AMD seemed to sweep the field with over 8200 requests per second with the Ipex following it up, however when divided per MHz the Ipex comes out on top with 1.58 RPS/MHz as opposed to the AMD with 1.02 RPS per MHz. It will be very interesting to see the results of the dual and quad Xeons in part two of this review next month.
These RPS per MHz results should be used as a guideline only to the systems' overall performance, not the CPUs themselves as there are more than just the CPU factor driving the system performance. Components such as I/O, memory speed/capacity, disk speed/availability, network performance, OS, and drivers all make a difference, not to mention individual system tweaking and the like.
Sample scenario
Company: CVP Fabrics. This business wants to install some mid-level servers for database and e-mail use.
Approximate budget: Open.
Requires: Six servers capable of supporting up to four processors, with two processors installed. 1GB of memory, at least 150GB of disk storage. At least one gigabit Ethernet (copper) port.
Concerns: The company believes at this stage dual-processor servers will support its needs, but with the expected growth over the next few years, wants the option to install additional processors and memory. Redundancy and other features for improving uptime will be very highly regarded.
Best solution: Unfortunately as the Apple and Sun are only dual-processor systems they are not in contention for this scenario, therefore it is down to AMD and Ipex. At this stage the AMD has the upper hand as it supports both 32-bit and 64-bit OSes and applications, and also has a significant price advantage over the Ipex system.
Editor's Choice: None.
It would be unfair to try and judge one of these systems above another. There are benefits in each system that would be attractive to different companies for different applications. The ultimate power network would comprise Ipex servers running the back end databases with the AMDs following up supporting the legacy apps and then later migrating across to the 64-bit apps. With G5s on the desktops and the Suns bringing up the security and Web infrastructure. Throw in some decent network switching like the Dell Gigabit switches. Now there is a dream network that hopefully isn't going to go out of date in a hurry.
Final words
The Test Lab is currently working with the manufacturers to develop further tests and scripts that will be able to easily put these types of disparate servers through their paces using database transactions and other variable loads, so future tests will use more real world applications one would expect to run on these types of machines yet still maintain a uniformity over platform hardware and software. Naturally the time and cost to develop these tests is more than could be done in just a few weeks and on our reviewing budget, however by the time the next server review rolls around for the magazine we hope to have added these to our arsenal of test routines.
Subscribe now to Australian Technology & Business magazine.
I saw a better Quad Opteron server platform from a manufacturer in Taiwan, looks like a great server, you can visit http://www.genesysrack.com.tw for a look, I hope somebody can supply this chaassis in Sydney.