User interaction
There is a wide range of packaged software for log analysis, such as WebTrends and FunnelWeb Analyzer. These programs convert the mass of data held in server logs into useful reports such as the way traffic varies over time, most and least visited pages, visitors' countries, referring sites, and session durations.
Martin Aungle, online marketing manager at Dimension Data Australia, uses WebTrends to measure the usage of the company's various Web sites. (Dimension Data is also a reseller of this product.)
The reports he gets from WebTrends let him keep an eye on error rates to help identify and correct specific problems, and monitor the effect of changes to the site. For example, he can see the clickthrough rate for new items added to the home page.
Aungle also uses WebTrends to identify usage patterns and trends over time. Intranet-based business reporting tools are used heavily towards the end of the month, but the information he gets from WebTrends helps ensure everyone has access to these tools without degradation during busy times.
The software also identifies referring sites, top search terms (which aren't always the words you expected, providing clues for search engine optimisation), and platform statistics (ongoing development costs can be reduced by not testing against an obsolete browser once you know hardly any visitors are still using that version).
Keeping it in-house
You might be tempted to write your own log analyser, and most likely some of the commercial products started that way. As your operation grows, so does the complexity of the problem. Server logging generates a large amount of data, so efficient processing is necessary to turn that data into information in a timely manner.
If you have a fairly simple operation a free program such as FunnelWeb Analyzer StandardEdition is most likely adequate; in complex environments, you'd have to question the wisdom of trying to reinvent the wheel.
The appliance market also extends to traffic analysis. Evolvo System's AIM (Advanced Internet Monitor) appliance avoids the need for log file analysis or the use of cookies or JavaScript to collect information about users. According to the company, the device collects, interprets and consolidates the Internet traffic data.
A single AIM appliance can handle up to 100 Web servers, and uses a patented system to identify individual visitors despite the presence of corporate firewalls, ISP proxies, or dynamic IP allocation systems. Visitors' ISPs are identified to show their geographic location.
AIM provides a variety of reports including visitor sessions, page hits, clicks, bandwidth, countries of origin, and the most and least requested URLs. Custom reports can be defined by exploiting AIM's SQL 92 compatibility.
These reports are available through the browser interface, or they can be delivered by e-mail or to a mobile phone via SMS, which is especially useful for error reports.
Three models are available, catering for 100, 500, or 250,000 concurrent Web visitors. Web servers are automatically identified when the appliance is connected to a network, and when a new Web server is brought online it is automatically added to the analysis.
The collected data is stored within the unit for privacy and security reasons, though backup agents are available for use with Legato, HP, IBM, and Microsoft software.
