Keeping track of what's happening with your Web site is a big ask. There are several dimensions to the issue, and in some aspects you can quickly collect many megabytes of data that must be turned into information.

Infrastructure

IBM Australia's market manager for WebSphere, Jack Verdins, says the company has a four-step methodology for Web site performance measurement: establish performance objectives, monitor and measure the site, analyse and tune components, and predict and plan for the future.

Performance objectives can include business-oriented criteria such as browse-to-buy ratios as well as technical criteria including application availability and resource utilisation.

Monitoring and measuring should be done in terms of the objectives set in the previous step, so you are aware of any metric that goes outside the established norms.

Component analysis can be broken into four main areas, namely edge servers (caches, etc), Web servers, application servers, and database servers and legacy systems.

"Each has their unique tuning opportunity," says Verdins. Viewing the system in this way makes it easier to identify the source of any problems, and the appropriate testing tool can be brought to bear on the affected subsystem.

Examples of real-life problems identified by this methodology include a failure to provide enough connections between the application server and the back-end database, an inappropriately indexed database, and a firewall configured to allow too few simultaneous connections to the Web server.

"IBM has a good history at the back end," Verdins says, but moving from traditional applications to the Web "increases the number of performance-related matters." Performance issues tend to be a concern mainly with transactional systems, Verdins says, adding "we've done most of the Wall Street sites" and those systems managed to cope with peaks in transaction volumes.

Planning for the future can be tricky. "The Web introduces 'interesting' volumes and unpredictability for marketing campaigns...very unpredictable things can happen," he says. That may be true, but it is still worth looking for fluctuations in transaction levels according to the time of day, day of the week, or the season, and planning accordingly.

Approximately one-third of performance bottlenecks can only be found by testing from outside the firewall, according to Mercury Interactive.

While the majority of problems can be traced to internal subsystems such as those identified by IBM, 35 percent are due to tuning and configuration issues with external routers, gate ways and switches, bandwidth constraints, and ISP peering point problems.

Company officials claimed that "performance problems occur 98 percent of the time simply because the infrastructure components have not been tuned or configured properly for a specific application."

Mercury Interactive provides a hosted service to load-test Web sites by emulating as many users as appropriate--1.35 million concurrent connections in one case--while measuring the performance of infrastructure components.

"Often, solving a performance problem with the one component will reveal previously hidden problems with another component," says Graham Sowden, managing director of Mercury Interactive in Australia.

"Proper load testing is an iterative process. It's like peeling an onion and examining each layer looking for ways to improve performance. By taking this approach, our ActiveTest experts have typically helped our customers increase the performance of their Web applications by more than 400 percent, without adding more hardware," he says.

Over the last few years, network appliances--pre-packaged Web servers, firewalls, and so on--have become increasingly commonplace. This trend has even extended to performance monitoring in the shape of Sniffer Technologies' Sniffer Pulse which monitors and analyses Web traffic in real time.

Sniffer Pulse has a Web-based interface to present detailed analyses of Web site components, and custom views can be created to focus on aspects of particular interest.

According to the company, it helps answer questions such as "How long are visitors waiting to view critical Web pages?" or "Are certain objects on my Web page degrading the performance?".

It can help determine whether complaints of poor performance are due to a site problem or bottlenecks at the user's end, and when installed between a load balancer and the Web server farm, it can verify that load balancing is being carried out appropriately.

In addition to displaying data on request, Sniffer Pulse can generate e-mail alerts on over 50 metrics including response time, page download time, and server error codes.

Specific alarm thresholds can be set, or they can be derived in terms of deviations from automatically generated baselines. These baselines identify the times when particular resources are heavily used, and can reveal when they are reaching capacity.

Usability

When IBM addressed usability issues on its e-commerce site, sales went up by 400 percent, he says, though doubling sales would be a more common outcome.

"If the customer can't find it, they can't buy it," he observed. Similar results apply to intranets: the easier it is to find the information people need, the more likely they are to use the intranet. One client recently implemented the results of the firm's usability testing an intranet usage went up by 98 percent.

Nielsen suggests a two-phase approach to ensuring the usability of a site. The first step is to base the design of the site on established guidelines, or to apply those guidelines to an existing site.

For example, his recent book Homepage Usability: 50 Websites Deconstructed provides a checklist of 113 guidelines for home page design.

"People like me have been looking for years at how people use technology...we have thousands of examples of sites that do or do not work," says Nielsen.

While he stresses that guidelines are exactly that--not hard and fast rules--they provide "a fast way of getting a first cut at which aspects of the Web site need attention."

The second step addresses site-specific usability issues through user testing. Nielsen recommends taking one user at a time and watching their attempts to perform tasks.

Even though his firm carries out usability testing for its clients, he says "it's very practical to do it yourself." While large, important projects would normally turn to professionals, "it's more appropriate to do a smaller, simpler test yourself."

Watching just five users working through some tasks is likely to produce a list of between 50 and 100 things that need changing, he suggests.

Learning the basics of usability testing only takes a few days. "It really is pretty easy to do," he says, though you get better with experience. This June, Nielsen Norman Group is running a five-day conference on the subject in Sydney, and one day is devoted to the basics of testing.

"There's no excuse for not doing usability testing," he says. Developers can't judge usability for themselves, because they know what each button means, so they never go down the wrong path. "You cannot wipe your brain...you will [always] think your own Web site is easy [to use]."

Focus groups are "worthless" for determining usability, he says; the question isn't whether people like the site, it's whether they can achieve a desired result.

Catering for people with some kind of disability is "a very important issue," according to Nielsen, and its importance is growing as the population ages and the user base widens.

Failing to accommodate the blind (through designs that are compatible with speech browsers), those with low vision, and even those who have trouble with small print is an example of showing "contempt for your customer," he says.

Similarly, the use of scrolling or drop-down menus can be a problem for those with physical impairments as minor as shaky hands. "There are well established guidelines for designing inclusive Web sites," he says, such as the Nielsen Norman Group's Beyond ALT Text: Making the Web Easy to Use for Users With Disabilities.

As before, the next step is user testing, but although that's harder to do, the guidelines in this area are more prescriptive and so testing might not be as crucial. Nielsen recommends professional testing for large e-commerce and government sites.

Automated testing tools can be of some help. Nielsen describes the well-known Bobby Web-based service as "a nice early attempt," he currently favours the Lift software and online service from UsableNet.

However, while such tools can spot that you have provided a text alternative for each graphic, they cannot determine whether the text is meaningful.

User interaction

Martin Aungle, online marketing manager at Dimension Data Australia, uses WebTrends to measure the usage of the company's various Web sites. (Dimension Data is also a reseller of this product.)

The reports he gets from WebTrends let him keep an eye on error rates to help identify and correct specific problems, and monitor the effect of changes to the site. For example, he can see the clickthrough rate for new items added to the home page.

Aungle also uses WebTrends to identify usage patterns and trends over time. Intranet-based business reporting tools are used heavily towards the end of the month, but the information he gets from WebTrends helps ensure everyone has access to these tools without degradation during busy times.

The software also identifies referring sites, top search terms (which aren't always the words you expected, providing clues for search engine optimisation), and platform statistics (ongoing development costs can be reduced by not testing against an obsolete browser once you know hardly any visitors are still using that version).

Keeping it in-house

You might be tempted to write your own log analyser, and most likely some of the commercial products started that way. As your operation grows, so does the complexity of the problem. Server logging generates a large amount of data, so efficient processing is necessary to turn that data into information in a timely manner.

If you have a fairly simple operation a free program such as FunnelWeb Analyzer StandardEdition is most likely adequate; in complex environments, you'd have to question the wisdom of trying to reinvent the wheel.

The appliance market also extends to traffic analysis. Evolvo System's AIM (Advanced Internet Monitor) appliance avoids the need for log file analysis or the use of cookies or JavaScript to collect information about users. According to the company, the device collects, interprets and consolidates the Internet traffic data.

A single AIM appliance can handle up to 100 Web servers, and uses a patented system to identify individual visitors despite the presence of corporate firewalls, ISP proxies, or dynamic IP allocation systems. Visitors' ISPs are identified to show their geographic location.

AIM provides a variety of reports including visitor sessions, page hits, clicks, bandwidth, countries of origin, and the most and least requested URLs. Custom reports can be defined by exploiting AIM's SQL 92 compatibility.

These reports are available through the browser interface, or they can be delivered by e-mail or to a mobile phone via SMS, which is especially useful for error reports.

Three models are available, catering for 100, 500, or 250,000 concurrent Web visitors. Web servers are automatically identified when the appliance is connected to a network, and when a new Web server is brought online it is automatically added to the analysis.

The collected data is stored within the unit for privacy and security reasons, though backup agents are available for use with Legato, HP, IBM, and Microsoft software.

Outsourced analysis

Some of these services rely at least partly on using panels of users, similar to the way TV ratings are calculated. Panel-based approaches to tracking have the advantage of providing demographic information about users, though some demographics may be available with other methods.

On the other hand, panels are very small compared with the population of Internet users. While panels can give satisfactory results where there are limited choices (eg, TV stations), the Internet's diversity means only the sites with the heaviest traffic are likely to be accurately measured.

Generally speaking, panels comprise home users, overlooking the substantial Web use in businesses, schools and other educational institutions, and government departments. Quite apart from the privacy implications, few IT managers are likely to entertain the idea of adding a research company's metering software to the organisation's standard operating environment.

Proxy analysis takes care of one of the shortcomings of log analysis: if a user request for a page is satisfied from the ISP's proxy server, it will not show up in a server log.

Most ISPs route Web traffic through proxies. Analysing the proxy logs from a number of ISPs gets around this difficulty, and potentially covers enough users in various categories (home, business, education, etc) to give meaningful figures.

Hitwise--one of the foremost proponents of proxy analysis--covers around 31 percent of Australian Internet users, and an even larger proportion in some other countries: approximately 43 percent in Singapore and 47 percent in Hong Kong. Although some demographic information may be available, it is unlikely to be as detailed or exact as that collected from a panel.

Like log analysis, proxy analysis has the advantage of being unobtrusive. Since proxies are now generally implemented in a transparent manner, users are no longer required to configure their software to take advantage of a proxy.

The other main method of collecting data is to embed code (usually JavaScript or Java) in each page that notifies a server whenever the page is loaded.

While this gets around the various caches that exist between the user's screen and the original server, it does raise privacy issues and can be circumvented by settings in the browser or in a firewall.

What does the information mean?

Keeping track of referring sites can be important as it can reveal whether you have done enough to optimise the site's ranking by search engines (whether that's achieved through technical means or by paying for a prominent position). It also shows how much traffic is being driven to your site by marketing partners, banner ads and so on.

Not every visitor will arrive at your home page. Many will go directly to pages deep within your site's structure from search engines, links from other sites, or from bookmarks.

Popular entry points deserve attention to ensure they deliver a good user experience otherwise you risk losing visitors as soon as they arrive. If a substantial amount of traffic is arriving directly and without a referring site, users have probably bookmarked or memorised the page.

In that case it could be risky to make a significant change to the content of the page, or to move the information elsewhere on your site.

By following users' progress as they go from link to link within your site, you are able to determine how well its structure suits their needs. If a lot of people are talking the long way round to a particular piece of information, you may need to provide a more direct route or make an existing link more obvious.

When a series of pages correspond to steps in a transaction (such as obtaining basic information, more detailed information, adding the item to a shopping cart and then completing the purchase), you might spot that an unexpectedly high proportion of visitors are bailing out of the transaction at a particular stage.

You can then investigate the reason why it is happening--abandoned shopping carts could be due to high shipping charges, for example. Knowing where your users go when they leave your site can be valuable.

For example, if a company that sells computer consumables but not inkjet refill kits might discover that a significant proportion of visitors move from its site to those operated by refill specialists.

In that case, it might be sensible to expand the range of products to include refill kits or to form an alliance with a refill specialist. Shopping centre operators know that the longer people spend in a centre, the more money they will spend.

One reason there are so many cafes and other food outlets in the typical shopping centre is that they tend to extend the duration of the average visit. Some Web site operators use the similar reasoning, and assume the longer visits last, the better.

This might be true for some kinds of site, notably a portal that exists primarily to put advertising in front of visitors, but others will judge success by providing visitors with the information they need as quickly as possible.

What users say

In that time he's used various tools but concludes "[browser-based] solutions like Red Sheriff are the most flexible and the easiest to use...if I were a corporate [user], that's the way I would go." This approach puts an applet inside the page, so every access is measured.

"We've run Red Sheriff on a lot of sites and not had any problems with issues [such as cookies or JavaScript being disabled]," he says. With Red Sheriff "you can have a high degree of confidence in the number of first-time users versus returning users," for example.

The problem is that the service must set up the reports for you, and mapping the reports onto your needs can be difficult.

Forrester Research is another advocate of data collection through JavaScript or Java Web bugs, as it avoids proxy and browser caching issues. The firm recommends the use of content management tools to insert bugs on all pages, but warns that the site's privacy policy must reveal their use.

van Wyck doesn't think much of log file analysis. "There always seem to be problems," he says. If you lose one day's log, that throws the whole month's figures out, and you have to remember that month is no longer comparable to others. "I find log file reporting tools a damn nuisance," he says. "Most tools give you more data than you ever need--knowing which [data] to use is the trick."

Content management systems sometimes provide tracking tools, but "my experience is that they are somewhat over-engineered and don't work as well as advertised. They're difficult to get to work, and take a lot of maintenance," says van Wyk.

Hitwise is "dangerous" according to van Wyk, because he claims its Australian figures are skewed towards Melbourne, you don't get accurate figures that allow comparison, the numbers are based on impressions (so a site that makes visitors navigate through several pages to reach the information they want will return higher figures than one that provides more direct access), and you don't see the number of users.

Paul Strickland, Internet marketing consultant at Web developer Marketing For The Web has a more positive view of Hitwise. His company uses Hitwise figures when pitching for new business and when maintaining sites for its clients, which include the Accor Asia Pacific hotel chain, Delta Europcar, and Kea Designer Sportswear.

One of the main attractions is that Hitwise looks at enough traffic to allow a breakdown into detailed categories while restricting the figures to Australian users.

The ability to define your own categories is "very valuable," he says, as it means you can compare a site with those you regard as its competition rather than relying on someone else's categorisation. Strickland also spoke highly of Hitwise's ability to quickly implement requested features. "We find the service is very good," he says.

"We find the accuracy in rating visitation is . . . overall a good indication of what's happening on the sites," says Strickland. Marketing For The Web uses Hitwise figures in conjunction with the reports it receives from its hosting company.

For example, if the hosting reports show a site had 100 unique visitors per day and Hitwise says its share was 50 percent, you know the total market is only 200 visitors per day.

"We like to look at clickstreams," says Strickland. While a Hitwise report may show that a lot of visitors came from Hotmail, for example, he can distinguish users who opened a new browser window leaving Hotmail open in another from those who responded to links embedded within messages.

van Wyk also likes panel-based services like Nielsen//NetRatings because they produce figures that let him compare his sites with their competitors.

While panels might not be truly representative of the wider population, he doesn't think this really matters, as all sites within a category will probably be equally affected, so comparisons can still be made.

If you're operating a niche site that won't show up in a panel survey, "you're better off looking at your own data and traffic" rather than trying to compare it with others, he says.

It doesn't really matter if you have 50,000 or 51,000 visitors, he suggests, what's more important is knowing how many of those visitorsbuy from you and how many go on to your competitors' sites.

Clean, consistent data tells you about the trends even if the numbers aren't 100 percent right, he suggests.

Usability testing is important in van Wyk's book, but it must be kept in perspective. "Do little bits aimed at a specific outcome, and do it a lot," he suggests.

Research companies tend to turn testing into an end to itself, he says, and may confuse usability testing with opinion research. "I believe you have to be really careful about what you're testing, why you're testing it, and what you are going to do with the results. "Do as much research as you need to get an answer, then move on," he says. For example, if you get the same result from the first five people, stop testing.

"It's well worth spending the money on usability testing, no question at all," he says. Some of the results can be surprising. For example, small changes in wording can make a big difference to a site's success.

The Toyota site uses the phrase "Price Your Car" because that usability testing showed that wording better conveyed the function than other phrases. van Wyk has also found the word "buying" puts people off. "Looking," "browsing,"' "obligation-free," and the like appear to be less threatening.

What analysts say

Indeed, the proportion of organisations using this measure had increased slightly from 94 percent to 98 percent since 1999. Organisations were attaching greater importance to measurement, and a wider range of metrics such as clickstreams and survey feedback were reported.

Most respondents said they were dissatisfied with their measurement tools: "Frustrations include trouble getting at the right data, inability to integrate data from multiple sources, and standardised reports that don't meet their needs."

They really wanted to know about individual users--whether a visit to the site was successful, the visitor's intention, identity (an e-mail address, company, even a geographical location), and whether the user is profitable.

Perhaps the most telling observation is that "Metrics alone can't answer the right questions. Hits, page views, session lengths, and even clickstreams don't tell site managers what's wrong with their Web efforts, let alone how to fix problems."

If a visitor looks at a couple of pages then leaves, was it because she got the information she needed or because your site was irrelevant? Was a shopping cart abandoned due to high shipping charges, privacy concerns, or because the visitor was just window-shopping?

(We would also ask whether your attempts to collect information about visitors are driving them away. Cookies are easily rejected, and trying to force them onto visitors makes potential customers decide how much they want to see your site. You may not like the answer. As for JavaScript-based Web bugs, JavaScript is notoriously sensitive to differences between browsers--even different versions of the same browser. Indeed, while preparing this article we had to change browsers in order to access material from one organisation because its JavaScript was incompatible with the version of Internet Explorer we were using. Furthermore, it isn't unusual for people to disable JavaScript support for security reasons. Is there really enough value in monitoring how people use your site to justify alienating some of them?)

Forrester also observed that very few organisations were tracking visitors who cross channels, such as those who do their product research online but make the purchases in stores.

To overcome some of these deficiencies, Forrester suggests an approach that it has dubbed scenario completion: "the rate at which users are able to accomplish their key goals". The idea is to "only measure the most profitable scenarios enacted by key market segments".

This, claims Forrester, focuses on the customers and prospects that matter most, forces organisations to take a cross-channel view, and generates actionable results (either the customer succeeded, or you have identified the source of the problem).

Three categories of scenario are suggested: action (eg, customer places an order), information (customer browses information, but this does not culminate in action such as placing an order), and communication (eg, a bank customer checks an account balance online, calls about a missing deposit, then visits a branch to follow up). That latter example also illustrates the need to combine information captured by different systems.

Four types of tools are needed to measure scenario completion. "Usage analysis provides a foundation, usability studies answer 'Why?', performance monitors ensure reliable service, and opinion polls track satisfaction-but not behaviour."

These scenarios do not exist in a vacuum, and should be related to corporate goals such as revenue, cost savings, site ROI, and cross-channel ROI.

Usability studies, performance monitors, and usage analysis are equally applicable to action scenarios, but opinion polls aren't such a good fit because people's opinions are not necessarily congruent with their actions.

Information scenarios are well measured by useability studies and opinion polls, but usage analysis is not relevant because such scenarios rarely end with a measurable action. Performance monitors may provide some relevant information.

Communication scenarios are the worst served, because usage and performance tools must integrate data from non-Web systems. Usability testing is of little relevance because of the differences between online and in-person scenarios. Opinion polls may provide some information in this area.

An important issue is that Web analytics require unique user identification. One way of achieving this is to persuade important customers to log in to the site. Organisations must relate all customer activity-marketing automation, customer service, Web--to a single ID and use it to link disparate systems.

As for the question of doing the work in-house or outsourcing, Forrester's advice is "Firms just piloting data warehousing projects or struggling to get IT support should focus on outsourced analytics.

But companies with data warehousing and mining skills will duplicate effort and expense by outsourcing--they should install packaged solutions instead."

Either way, you should look for tools offering flexible reporting from OLAP (if you have data mining skills) or tools and services that provide highly customisable Web-based reports.

If you are doing the work internally, you'll need a way of archiving information from Web logs (the logs themselves are too big to keep), or of storing clickstreams, user IDs, etc in a database. A final tip from Forrester: "Design and build with measurement in mind."

META also believes in looking at the big picture. "Poor revenue numbers do not necessarily mean a Web commerce site is performing poorly. Companies must consider all the roles their Web presence plays to more accurately determine return on investment," says META's Carl Lehmann in his report "The Multidimensional ROI of Web Commerce."

"Other valuable business functions of a Web commerce site are often overlooked in ROI analysis. These include the Web site's role within existing advertising campaigns, as a direct marketing channel, and as an indirect sales support tool for sell-side business partners.

"We believe Web commerce sites should not stand alone as a distinct customer interaction channel. They must be aligned with existing customer interaction channels and governed within the context of a customer relationship management (CRM) strategy."

For example, organisations should look at cost of transactions (sales, support, etc) via the Web site, and compare the results with the costs for other channels.

If the Web really is cheaper--and it may be as little as 2 percent to 20 percent of the cost-they should seek to drive customers to online channels. Similarly, compare with other forms of advertising--use CPM (cost per thousand impressions). Also include contribution of Web to channel support functions such as lead management.

"Companies realise a greater return on investment when they consider the multidimensional role of a Web commerce site as also a mechanism for advertising, direct marketing, and channel relationship management."

The May 2002 issue of ZDNet Australia's Technology & Business Magazine contains information about about Web site analysis tools. For subscription information, visit Technology & Business.

Copyright © 2009 CBS Interactive, a CBS Company. All Rights Reserved.
ZDNET is a registered service mark of CBS Interactive. ZDNET Logo is a service mark of CBS Interactive.