Browser faceoff: IE vs Firefox vs Opera vs Safari

Web 2.0, with its complex sites and rich Ajax applications, is an increasingly demanding platform for a browser. In this review feature, we look at how the leading browsers measure up.

When Tim Berners-Lee presented his employer CERN with the first browser, World Wide Web, to facilitate information flow between the different departments in the European nuclear research centre in Geneva, he little suspected that it would cause a revolution in the information age. Today, the browser is probably the most widely used computer application.

However, the tasks performed by a web browser have changed significantly. As well as displaying text and images, the modern browser needs to accommodate technologies such as JavaScript, DOM and XML in Ajax-based programs. Even if you're not familiar with Ajax, you'll probably have used it via Google Maps, Google Mail or AjaxWrite; sites such as Flickr and Last.fm also make intense use of this technology. Ajax has even taken root in the business environment: for example, 24SevenOffice is an Enterprise Resource Planning (ERP) and Customer Relationship Management (CRM) solution that runs in a web browser. The advantage of this Web 2.0 technology is platform independence: you don't need a specific operating system or processor to run Ajax applications — just a compatible browser.

Web 2.0 applications and sites place the focus firmly on browser performance. Anyone who still believes that the speed of your DSL connection is the only potential bottleneck is gravely mistaken. Key parts of Ajax applications run locally, which means that — all other things being equal — the speed of the browser will be crucial in determining the user experience. For Ajax-based business applications, the browser becomes even more important because data will be accessed from within-firewall servers rather than the internet. Companies deploying such solutions will be able to improve employee productivity by paying attention to browser performance.

To test the performance of browsers when handling Web 2.0 technologies, we used the iBench 5.0 test suite and SunSpider, a Javascript benchmark.

iBench evaluates browser performance by measuring how fast HTML, XML and JavaScript is handled. The web pages are held on a local web server. SunSpider, which (unlike iBench) is available online, concentrates solely on JavaScript performance. Tests are grouped into nine categories, including 3D, bit operations, cryptography and string processing; there are several tests within each group. The benchmark runs each test multiple times and calculates an error range.

Obviously the speed of the hardware platform is a crucial variable. In our browser tests under Windows Vista and Mac OS 10.5.2 Leopard, we used the following components:

The seven browsers we tested load HTML and XML/CSS pages under Windows Vista at a variety of speeds. According to iBench 5.0, Safari 3.1 is the fastest browser and Opera 9.27 the slowest; the latter takes over twice as long as any other browser to load HTML pages. However, the beta version of Opera 9.5 performs significantly better, and is on a par with the middle-ranking browsers.

In the iBench 5.0 JavaScript tests, Safari 3.1 is again the fastest browser, while Microsoft's browsers bring up the rear in JavaScript/HTML DOM tests. The JavaScript tests also expose greater performance differences among the browsers than the HTML/XML tests: in the JavaScript tests, the slowest browser is about 10 times slower than the fastest.

The various beta versions show significant performance improvements in some tests. For example, Firefox 2.0.13 completed the JavaScript/HTML DOM test in 3.1 seconds, while the Firefox 3 Beta 5 took only 0.65 seconds. The two Opera browsers show a similar pattern: Opera 9.27 was slowest in the JavaScript test (2.54s), while Opera 9.5 Beta 4758 delivered the second best result (0.36s).

It should be noted that iBench 5.0's reported values for the Safari browser are underestimates of the actual timings, which we checked manually. The reason is the measuring methodology in iBench 5.0, which uses the 'onload' event to signal that a page has been loaded: most browsers load the page, decode images and run stylesheets and scripts before firing 'onload'. Safari does not do this. There's no doubt that Apple's browser is fast, but it's not as far ahead of its competitors as iBench 5.0 suggests — as our next test shows.

Timings in seconds: shorter bars are better.

Timings in seconds: shorter bars are better.

The SunSpider JavaScript benchmark shows Microsoft's current browser in a poor light: as in the iBench JavaScript/HTML DOM test, Internet Explorer 7 is by far the slowest of the group. Firefox fans will be pleased with the speed of version 2.0.0.13 compared to IE 7, but Opera 9.27 and Safari 3.1.0 are the real winners here.

Things look different when we consider the beta versions. Microsoft improves its position significantly with IE 8 Beta 1, although it's still the slowest of the betas. Meanwhile, Firefox, whose current version only manages second-last place, leaps to the top of the class ahead of Safari with version 3 Beta 5.

The individual SunSpider JavaScript benchmarks show the strengths and weaknesses of particular browsers, and where the beta versions show the most improvements. The poor performance of Internet Explorer 7, for example, is due to its slow string processing. IE 8 Beta 1 delivers a significantly better result in this test and reels in some of the competition, although it remains slow compared to the other betas and Safari. The current version of Firefox is weak in string and 3D processing and bit operations, but version 3 Beta 5 overcomes these weaknesses to take first place in all of the tests.

Timings in milliseconds: shorter bars are better.

Timings in milliseconds: shorter bars are better.

According to Apple boss Steve Jobs, the company's Safari browser is the fastest on the market. However, Apple has something of a history with benchmarks: when the PowerMac G5 was introduced in 2003, for example, the company was accused of tweaking its test system and hobbling rival Intel-based machines. Since then, of course, Apple has come round to the attractions of Intel processors.

Jobs' statement that the fastest Safari browser is based on tests the company has carried out using iBench 5.0. However, these benchmark results should be treated with caution: manual measurements with a stopwatch show that Safari is not quite as fast as iBench suggests. This is because, as mentioned earlier, iBench uses the JavaScript 'onload' event to determine page loading time, which Safari triggers before the page has in fact finished loading. Even so, despite these caveats, there's no doubt that Safari is indeed a fast browser.

Safari 3.1.0 is quickest in the HTML tests with a time of under 10 seconds, placing it well ahead of Opera 9.27's 48 seconds. However, the STAND Safari plug-in extends the load time to over 18 seconds — perhaps because it prevents the premature page-loading statement normally reported by iBench. The beta version of Opera 9.5 manages to reduce the load time from over 48 seconds in version 9.27 to less than 20 seconds. In the XML/CSS test, Safari again takes first place — and here, the STAND plug-in has no significant impact on performance. Opera brings up the rear in this test, and this time the 9.5 beta brings no improvement.

Safari is the winner in the JavaScript tests too. These tests also demonstrate the benefits that browser development can deliver. Among the beta versions, Firefox 3 Beta 5 is particularly impressive compared to its predecessor, reducing Firefox 2.0.13's 3.91 seconds to just 1.07 seconds in the HTML/JavaScript DOM test. Opera and Safari also deliver significant performance improvements in their respective betas.

Timings in seconds: shorter bars are better.

Timings in seconds: shorter bars are better.

Of the current Mac OS X browsers, Safari 3.1 is clearly the fastest in the SunSpider JavaScript benchmark. However, Firefox 3 Beta 5 beats Apple's browser by a noticeable margin. Elsewhere, the Opera 9.5 beta improves on its predecessor, although it cannot match the performance of Safari and Firefox 3 Beta 5.

The individual tests show exactly how Firefox 3 Beta 5 has overhauled Safari, and where Opera needs to improve in order to catch Firefox and Safari. For example, Firefox 3 Beta 5 took 187 milliseconds for the bit operations test, compared to the current version's 2,241ms. Opera 9.5 Beta also improves on the current version, but not to the same extent.

Timings in milliseconds: shorter bars are better.

Timings in milliseconds: shorter bars are better.

As well as pure performance, it's important to consider the browsers' resource requirements. Although the CPU load is pretty consistent among the browsers, this is not the case with memory consumption.

In the first test, a web site is loaded and memory consumption measured. A second test measures the resources needed to load 10 tabbed sites. We chose sites without advertising, since the constantly changing content of such sites does not allow for reproducible results: for example, one time you may get a simple image, while the next time the page is loaded a Flash animation may appear.

With only one web site to display, the browsers use between 18.5MB and 35.3MB of memory under Mac OS X 10.5.2. Safari is the most memory-frugal, while Firefox is the biggest memory-hog. However, the picture changes if ten sites are loaded at the same time. Now, Firefox 3 Beta 5 leads the field with 76.2MB, followed by Safari with 93.7MB. Safari uses slightly more memory (98.1MB) if the STAND plug-in is installed. Opera 9.27 uses 134.5MB, while Opera 9.5 Beta uses 139.3MB — slightly more than Firefox 2.0.0.13 with 138.8MB.

Memory usage patterns vary widely under Windows Vista too. With only one website, Microsoft's browsers perform well: Firefox uses around twice as much RAM as Internet Explorer, while Opera and Safari use around three times as much. With ten sites loaded, the results are almost reversed. Now IE brings up the rear with 143.7MB, while Firefox 3 Beta 5 only uses 50.8MB. As with Mac OS, Firefox 3 Beta 5 is the most memory efficient. The Opera versions use 66.8MB (9.27) and 77.8MB (9.5 Beta), while Safari and Firefox 2.0.13, with 113.3MB and 94MB respectively, need much more memory — but still less than IE.

Memory use in megabytes: shorter bars are better.

Memory use in megabytes: shorter bars are better.

Timings in milliseconds: shorter bars are better.

Timings in milliseconds: shorter bars are better.

Timings in milliseconds: shorter bars are better.

Timings in milliseconds: shorter bars are better.

Safari is undoubtedly one of the fastest browsers available, even if iBench 5.0 delivers slightly optimistic scores (the SunSpider JavaScript benchmark does not seem to be affected). Under both Mac OS X 10.5.2 and Windows Vista SP1, Apple's browser is a front-runner.

The worst-performing browser in our tests is Internet Explorer 7 — The SunSpider JavaScript benchmark, in particular, shows up the deficiencies of Microsoft's current browser. Things may change with Internet Explorer 8, as the first beta version shows a significant performance improvement. However, the competition is not standing still, with new versions of Firefox (3) and Opera (9.5) closer to release than IE 8.

Overall, and taking the performance of the beta versions into consideration, the browser performance rankings are as follows: Firefox and Safari in a clear lead ahead of Opera and finally Internet Explorer. Firefox 3, due in June, promises very good performance and efficient memory usage. No browser used less memory to open multiple sites than the Beta 5 version of Firefox 3, and only a few sites failed to display correctly. Safari and Opera, however, are more likely to exhibit incompatibilities.

These tests show that when it comes to browser performance, there's plenty of scope for improvement. In some tests, we observed speed boosts of the order of 100 per cent between the current version and the latest beta.