What analysts say
Industry analysts warn of the importance of collecting the data that can answer important questions. A September 2001 study by Forrester Research found the most common metric for Web site success was still traffic, whether in terms of hits, page views, or unique visitors.
Indeed, the proportion of organisations using this measure had increased slightly from 94 percent to 98 percent since 1999. Organisations were attaching greater importance to measurement, and a wider range of metrics such as clickstreams and survey feedback were reported.
Most respondents said they were dissatisfied with their measurement tools: "Frustrations include trouble getting at the right data, inability to integrate data from multiple sources, and standardised reports that don't meet their needs."
They really wanted to know about individual users--whether a visit to the site was successful, the visitor's intention, identity (an e-mail address, company, even a geographical location), and whether the user is profitable.
Perhaps the most telling observation is that "Metrics alone can't answer the right questions. Hits, page views, session lengths, and even clickstreams don't tell site managers what's wrong with their Web efforts, let alone how to fix problems."
If a visitor looks at a couple of pages then leaves, was it because she got the information she needed or because your site was irrelevant? Was a shopping cart abandoned due to high shipping charges, privacy concerns, or because the visitor was just window-shopping?
(We would also ask whether your attempts to collect information about visitors are driving them away. Cookies are easily rejected, and trying to force them onto visitors makes potential customers decide how much they want to see your site. You may not like the answer. As for JavaScript-based Web bugs, JavaScript is notoriously sensitive to differences between browsers--even different versions of the same browser. Indeed, while preparing this article we had to change browsers in order to access material from one organisation because its JavaScript was incompatible with the version of Internet Explorer we were using. Furthermore, it isn't unusual for people to disable JavaScript support for security reasons. Is there really enough value in monitoring how people use your site to justify alienating some of them?)
Forrester also observed that very few organisations were tracking visitors who cross channels, such as those who do their product research online but make the purchases in stores.
To overcome some of these deficiencies, Forrester suggests an approach that it has dubbed scenario completion: "the rate at which users are able to accomplish their key goals". The idea is to "only measure the most profitable scenarios enacted by key market segments".
This, claims Forrester, focuses on the customers and prospects that matter most, forces organisations to take a cross-channel view, and generates actionable results (either the customer succeeded, or you have identified the source of the problem).
Three categories of scenario are suggested: action (eg, customer places an order), information (customer browses information, but this does not culminate in action such as placing an order), and communication (eg, a bank customer checks an account balance online, calls about a missing deposit, then visits a branch to follow up). That latter example also illustrates the need to combine information captured by different systems.
Four types of tools are needed to measure scenario completion. "Usage analysis provides a foundation, usability studies answer 'Why?', performance monitors ensure reliable service, and opinion polls track satisfaction-but not behaviour."
These scenarios do not exist in a vacuum, and should be related to corporate goals such as revenue, cost savings, site ROI, and cross-channel ROI.
Usability studies, performance monitors, and usage analysis are equally applicable to action scenarios, but opinion polls aren't such a good fit because people's opinions are not necessarily congruent with their actions.
Information scenarios are well measured by useability studies and opinion polls, but usage analysis is not relevant because such scenarios rarely end with a measurable action. Performance monitors may provide some relevant information.
Communication scenarios are the worst served, because usage and performance tools must integrate data from non-Web systems. Usability testing is of little relevance because of the differences between online and in-person scenarios. Opinion polls may provide some information in this area.
An important issue is that Web analytics require unique user identification. One way of achieving this is to persuade important customers to log in to the site. Organisations must relate all customer activity-marketing automation, customer service, Web--to a single ID and use it to link disparate systems.
As for the question of doing the work in-house or outsourcing, Forrester's advice is "Firms just piloting data warehousing projects or struggling to get IT support should focus on outsourced analytics.
But companies with data warehousing and mining skills will duplicate effort and expense by outsourcing--they should install packaged solutions instead."
Either way, you should look for tools offering flexible reporting from OLAP (if you have data mining skills) or tools and services that provide highly customisable Web-based reports.
If you are doing the work internally, you'll need a way of archiving information from Web logs (the logs themselves are too big to keep), or of storing clickstreams, user IDs, etc in a database. A final tip from Forrester: "Design and build with measurement in mind."
META also believes in looking at the big picture. "Poor revenue numbers do not necessarily mean a Web commerce site is performing poorly. Companies must consider all the roles their Web presence plays to more accurately determine return on investment," says META's Carl Lehmann in his report "The Multidimensional ROI of Web Commerce."
"Other valuable business functions of a Web commerce site are often overlooked in ROI analysis. These include the Web site's role within existing advertising campaigns, as a direct marketing channel, and as an indirect sales support tool for sell-side business partners.
"We believe Web commerce sites should not stand alone as a distinct customer interaction channel. They must be aligned with existing customer interaction channels and governed within the context of a customer relationship management (CRM) strategy."
For example, organisations should look at cost of transactions (sales, support, etc) via the Web site, and compare the results with the costs for other channels.
If the Web really is cheaper--and it may be as little as 2 percent to 20 percent of the cost-they should seek to drive customers to online channels. Similarly, compare with other forms of advertising--use CPM (cost per thousand impressions). Also include contribution of Web to channel support functions such as lead management.
"Companies realise a greater return on investment when they consider the multidimensional role of a Web commerce site as also a mechanism for advertising, direct marketing, and channel relationship management."
The May 2002 issue of ZDNet Australia's Technology & Business Magazine contains information about about Web site analysis tools. For subscription information, visit Technology & Business.
