Business Intelligence software gives managers the tools to draw from many data sources and take a snapshot view of their company's performance. Why are business intelligence vendors defying the industry trend and continuing their stellar growth?

Virgin Mobile has built a loyal following within Australia by offering no-nonsense prepaid and contractual mobile phone plans designed to win and keep customers through sheer value alone.

But that value wasn't always self-evident. Finding out what customers want takes market research, a willingness to sacrifice margins to offer a better deal, continual monitoring of channel and business performance, and an effective way of collecting and analysing all manner of business data.

For Virgin Mobile, the solution came in the form of a business intelligence (BI) system that was implemented in late 2000 and quickly became critical to the company's operations.

Because it was administering its business using Oracle databases and financial applications, Virgin mobile decided the easiest way to implement its BI system was by using Oracle's own Business Intelligence add-on, with Cognos client-side software providing an easy-to-use interface for users.

A careful design and data integration strategy helped the company deploy the solution in just three months, delivering on Virgin Mobile's vision of managing each customer by their SIM card number--using a single identification number across various data sets simplifies the process of cross-referencing customer usage between a variety of products.

"We pull usage records from different platforms to try and understand customer usage and how we can better target products to them," says Nick Gatland, IT manager with Virgin Mobile, who helped expand usage of the 400GB database from the marketing organisation across the entire organisation.

"We analyse customer usage, average revenues, calling patterns, the types of products people buy, which [dealers] are performing well, and so on. It started out as a marketing database, and we now use the system for straight reporting of business key performance indicators [KPIs]. It's a multi-layered set of information."

BI has become central to Virgin Mobile's forward planning, but the company is hardly alone. Over the past few years--particularly as the rise of customer relationship management (CRM) forced companies to find new ways of monitoring and responding to customer desires--BI has become widely implemented and forms a critical part of most large businesses' long-term strategies.

This importance is reflected in analyst predictions, which give BI a rosy outlook despite a temporary slowdown in 2000 and 2001 due to delayed software purchases and a trend toward smaller contract sizes. According to research firm IDC, the worldwide BI market is expected to grow at 27 percent annually, from US$3.6 billion in 2000 to be worth US$11.9 billion globally by 2005.

Those healthy figures have been reflected in some quite healthy financial results from BI companies despite the market's overall lacklustre performance. Crystal Decisions' revenues have grown a reported 30 percent quarter on quarter for the past seven quarters, and SAS Institute reports Q1 revenues this year were 50 percent higher than for the same quarter in 2001. Clearly, customers are still willing to pay for the right BI solution even as tough economic times force them to trim back overall spending.

View from the top

"I don't think most companies really identify their most profitable customers," says Ron Swift, vice president of strategic customer relationships with the Teradata division of data warehousing giant NCR.

"They tend to model based on sales, and don't understand they can utilise detailed historical behaviours of transactions and events. But if you're selling the wrong product and don't have the analytics to know that, what good is it? Companies need an enterprise approach to understanding customers, and if you do it effectively, you can shorten sales cycles."

As marketing staff warm to the benefits of an appropriately structured BI solution, senior managers are becoming particularly interested in BI's ability to improve their view of the current state of the business.

Instead of waiting for monthly printed reports, they can log onto an online BI "dashboard" and get a top-down view of the way the business is performing--one department at a time. They're managing by wire, so to speak, and BI systems are being called upon to deliver that wire.

Often, these systems are led by initial deployments targeting specific departments of areas of the business, then expanded upwards throughout the business as usage and demand dictate.

At Virgin Mobile, for example, Gatland was careful to limit the Oracle BI environment to the marketing department first. Later he expanded the system to offer executives in other areas the ability to query more general business and financial data that was already being collected in the Oracle system.

This approach allows the BI tools to be progressively rolled out to areas of greatest need first. Kinks can be ironed out, business rules tweaked, and all the while the system provides a continuous and expanding indicator of the company's financial health by pulling relevant data from myriad back-end administrative and production systems.

"To have a good strategic solution, you need to be able to understand your suppliers, customers, and organisation," says Greg Wood, solutions manager for data warehousing with SAS Institute Australia.

"Data warehousing is very much about structuring data in a form that's good for answering specific business questions, and taking away the overhead for answering those questions."

By asking the right questions, some BI deployments have effectively become smaller, interactive versions of Kaplan and Norton's balanced scorecard philosophy (see www .balancedscorecard.org) This managing-by-the-facts approach is reflected in at least 28 BI tools from vendors such as SAS Institute, CorVu, Gentia, Accure, Hyperion, Peoplesoft, SAP, and Solvision. (Gartner and the UK-based Cranfield School of Management have published a comparative analysis, Balanced Scorecard Software Report, that's valuable reading for anyone considering a balanced scorecard approach.)

The balanced scorecard has yet to take off in Australia as it has overseas; it's been suggested that more than half of the largest US companies had adopted a business measurement framework such as the balanced scorecard by the end of 2000. Nonetheless, the principles underlying its success have been reflected in senior executives' growing demand for live business metrics through intelligent BI deployment.

"The Balanced Scorecard concept has created an awareness of how important it is to focus people," says Robert Zalums, managing director of Crystal Decisions Australia-New Zealand, "and how important it is for every person in an organisation to have a small set of performance indicators that focus their team on the things that are important right now. Those things might change as strategy evolves. If you manage the right things--things important to strategy and tactics--and you communicate those measures to people that can drive execution of those tactics, then you will achieve."

Changing architectures

That framework has evolved rapidly as BI tools--executive information systems, as they were called a decade ago--grew in sophistication and became accessible to all, with the rise of the standards-based intranet.

Yet data management remains a complex task: data warehouses were once seen as the best way of aggregating data from what was often dozens of disparate databases, but they also imposed a need for onerous data cleansing exercises and put a severe strain on server resources.

Data marts, the compromise developed to reduce the sheer volumes of data available for analysis by distributing relevant subsets of the data warehouse to individual departments, added additional complexity by adding data currency issues to the mix. But years later it is OLAP (online analytical processing), whose multidimensional "cubes" allow users to slice data subsets in a variety of ways, that has become the major driver for analysis of BI data.

With today's servers powerful enough to handle the massive data analysis projects their forebears could barely contemplate, OLAP provides an excellent way of letting users manipulate data as they see fit-as long as they know what they're trying to ask.

"It's always been a back-end issue as to whether the database could handle multiple queries," says Hummingbird Australia technical manager Paul Segal.

"Throwing around large volumes of data isn't so much an issue anymore. The biggest challenge is understanding the data in the database; if you don't know what the data is trying to tell you, you can't form any business decisions based on that data. Designing OLAP cubes requires a different mindset to normal reporting, but once people see its power they don't want to go back."

Segal's comments are reflected in IDC's analysis of the BI market, which predicted OLAP will be its second fastest growing segment, with 22 percent annual growth indicating widespread acceptance of its approach to data analysis. Indeed, OLAP revenues are expected to account for 27 percent of the overall BI market by 2005.

Interestingly, data mining--using artificial intelligence to pick out previously unforeseen patterns in data--is nowhere near the giant it was once was; its extreme complexity is part of the reason companies now design BI with specific business questions in mind.

Not everyone finds OLAP to be a breeze, points out Alphablox founder and chairman Michael Skok. "The average [OLAP] cube has six or seven dimensions. How many users can think beyond three? If your job is to slice and dice data all day, these are fantastic products; the question is how to get those people into the business."

There's little surprise, then, that IDC named the fastest-growing segment of the BI market as end-user query and reporting tools, which will grow at 30 percent annually to account for 52 percent of the BI market by 2005.

Such tools are critical in delivering the benefits of data analysis to users across the enterprise. Ease of use is a key factor these days: if normal users can't access the technology, it's simply of no use to them. "We see the Web as the only way to deliver BI to everyone in the organisation," says Brio Software managing director Paul Beks, who notes that many companies are now consolidating the interfaces of multiple BI platforms that have come bundled with various key enterprise applications.

"A lot of businesses have done a lot for the deployment of data management, improving data quality, and have put BI tools in place," he explains. "The biggest challenge is delivering that efficiently to the Web, so they can get as high as possible into the organisation and talk about the total organisational view rather than getting stuck in a departmental view. The portal does that."

The ability to aggregate multiple data sources was a key driver for the Royal Australian Navy's decision in May to purchase up to 1000 seats worth of Hummingbird's BI/Broker business intelligence portal solution, to improve employee access to the systems used for administering its Material Systems Branch. The OLAP-based system will be used to generate over 100 reports a day on everything from product inventories to maintenance schedules.

Ubiquitous access to BI can present its own problems, however, and it's important to manage OLAP proliferation carefully, points out SAS Institute solutions manager James Redman. "Because the newer tools fit on anyone's desktop and everyone can build their own cubes, the proliferation of independent views of things has grown rather than receded," he explains.

"We advocate having a central warehouse with a single version of the truth; the problem is that warehouses take a long time to build and a lot of users don't want to wait that long. Six months down the road everyone's got their own little islands of information."

That can lead to many different perspectives of the same information, which in turn can cause friction between business units if managers and other users aren't well trained about the contextual meaning behind their numbers.

Building bridges

"Business intelligence is a line of business project; it is not an IT project," says Ursula Paddon, market manager for IBM data management. "It has to be a business led project run in co-operation with IT. It's an iterative process: you develop business rules, manage the data and get it coming, then do a gap analysis and build on your original implementation. It is one hell of a learning curve for every organisation, but it's absolutely phenomenal what comes out of it."

Many companies are just starting to see what that means. One of the increasingly frequent results of such lateral thinking has been the extension of BI systems out to business partners who have traditionally been excluded because they lie outside the four walls.

Such BI extranets are proving to be a tremendously successful way of bringing partners closer, improving communication throughout the supply chain, and keeping partners and customers longer by creating expectations that your competitors can't match.

The UK arm of mobile carrier Vodafone, for example, uses Business Objects technology to offer its customers self-service access to detailed usage statistics. In the US, Zurich Insurance is using the same technology to let customers like Ford Motor Company dig through mountains of information relating to claim statistics, risk calculations, and the like.

"Many businesses have spent a lot of money on CRM and ERP implementations, and they've got this great data that would be very useful to their customers and suppliers," explains Joel Weingarten, vice president and general manager of Business Objects' extranet business unit.

"BI extranets represent a way that they can go that last mile and get that full utility out of this valuable BI information. You're really helping customers help you, and every time they log on they're dynamically connected to the freshest data they can access based on their security profile."

Extranets are already well-understood beasts, and extending the paradigm to business intelligence has become much easier as vendors combine the two approaches.

They can be used to engender closer teamwork between individual departments that may have built their own BI systems but lack a broader context for their systems. They can also provide excellent value-add, requiring relatively little additional effort, for companies seeking to augment the range of services they offer their customers.

For SMS marketing company 5th Finger, building a BI extranet has been an excellent way of adding value to its product offering, says director Steen Andersson.

Earlier this year, the company used Crystal Reports for Visual Studio .NET to build a system that lets its customers--which include companies like Channel V, Guinness, and McDonalds--log in at any point during a marketing campaign to see exactly how it's progressing and learn more about the customers that have responded so far.

"We always saw SMS as great, but what's exciting about it as a marketing tool is the information you can gather about customers," explains Andersson.

"We can design standard reports and let clients log in to see live updates. It's really getting quite practical, down to helping [Channel V] make more money because they can tell their advertisers that this particular show got x thousand entries."

Equally dramatic change came after a BI extranet set up by GrainCorp, a NSW grain handler that's used Business Objects technology to provide more than 3000 farmers with direct online access to a range of regularly updated reports. For example, GrainCorp tracks and can publish KPIs such as the number of tonnes it's processing per man-hour, the cost in dollars per tonne, and truck turnaround time.

This information is invaluable for helping farmers plan where to bring their grain, and GrainCorp information services manager Brian Dickinson thanked the system's rapid adoption for a winter harvest of 11 million tonnes-higher than the expected 10.5 million tonnes. Next year, the system will be available to up to 5000 growers and 200 buyers.

By extending BI reporting outside the conceptual limits of a company's four walls, extranets will also serve another purpose: paving the philosophical way for a new generation of more distributed analytics Web services.

US company Epistemic, for example, this month launches its Java-based Epistemic Analytics Toolkit, which bundles a modular BI suite that's exposed to applications as an online Web service.

This approach, which is also possible through Business Objects' recently launched Web Services SDK, effectively brings the analytics to the data instead of sending the data to the analytics. That will allow heavy BI users to provide a common analytics engine to multiple departments, eliminating redundancy and potentially facilitating ad hoc queries from users anywhere in the world.

Companies are constantly looking for new ways to improve communication up and down the supply chain, and the flexibility provided by extranets and Web services now provide an excellent option for making that happen.

While it's still in early days in Australia, increased intimacy between corporate BI systems and online customer should increase rapidly as appropriate technologies and business cases become more evident. That will handily push BI into its next major phase--the latest in a long line of improvements that are helping businesses learn things they never knew about themselves.

Case study: Ergon Energy

Its goal was to develop new marketing strategies by drawing on information in its customer data warehouse in new ways.

After months of planning and testing, Ergon selected Brio's Brio Intelligence platform, and purchased seven licenses for a pilot test that began in June 2001.

The company's first test of the technology came as it sought to identify those of its 570,000 customers who were using relatively expensive methods of account payment, such as credit cards, and encourage them to use cheaper methods such as direct debit.

By pulling data from the Oracle data warehouse and overlaying it on top of geo-demographic information sourced from specialist marketing firm Pacific Micromarketing, Ergon was able to get a detailed picture of which customers were using which payment methods, where they lived, what life stages they were in, and so on.

The system provided enough information that Ergon was able to successfully target its customers with an individualised campaign that took different approaches for different types of customers.

-We wanted to find out what our customers looked like, and understand them a lot better in order to market some non-energy products [such as water heaters and green energy] to them," says Ben Verdon, retail business analyst with Ergon.

-By looking at the way customers were paying, we were able to suggest better payment methods that would suit their needs and benefit Ergon. Nobody knew what our customer base looked like before, but with Brio within 20 minutes we were able to generate lists of who was doing what."

That one campaign delivered an immediate ROI of more than 6000 percent, and Ergon has since expanded its use of Brio as it works to build out a larger corporate data warehouse due in production by November on a two-CPU Sun Microsystems Enterprise 450 Server. The data warehouse is growing at 500MB a week and is expected to top 100GB soon.

Even in its developing form, the system is already being used to compare metrics between salespeople, which was never possible and has -opened a few eyes", Verdon says.

-We wanted quick wins and the system has provided them. The best thing is that it will provide the ability to have a holistic view of the customer, and to be able to find similar customers who fit the same demographic profiles but haven't bought anything else from Ergon. There's a constant driver to understand our customers better."

Case study: Queensland Health

That point was demonstrated vividly by Queensland Health, which last year began a massive project to re-engineer its four-year-old Clinical Benchmarking Project (CBP).

Initially based on Seagate Software's (now Crystal Decisions') Seagate Info business intelligence platform, CBP had grown to include around 1.5TB of data incorporating minute details of around AU$3 billion worth of annual expenditure on everything from doctors' hours and diagnostic patterns to Band-Aids and lightbulbs.

It had proved tremendously successful in identifying purchasing anomalies, offering dramatic savings such as the identification of a common but unnecessary diagnostic test that was costing one healthcare district over AU$200,000 a year.

The system's cost and resource efficiency were brought into question, however, after it was realised that the BI infrastructureâ€"which moved data back and forth from the Brisbane database across the WAN to client applications at 22 hospitals across the stateâ€"was degrading network performance and introducing unnecessary WAN costs. Creating a single 20MB report generated an average of 500MB of network traffic, since all the back-and-forth communications had to run over the remote link.

-Even when we were utilising the product the way it was meant to be, its impact on the WAN was significant," says CBP principal project officer Glenn Dalton. -We were not succeeding using the current environment because of technical reasons. Even though in one sense the decentralised architecture is relatively easy to use, we needed people at each site who knew how to run the system, roll it out, and run a very large database. We wanted to be able to delegate control of each environment down to individual hospitals."

After hearing that a recent application upgrade would enable a full thin-client interface, Dalton's team drew up a new business case and began restructuring the BI system early last year.

In its new configuration, the Windows 2000-based database and analysis serversâ€"-running Crystal Decisions' Web-based Crystal Enterprise 8.5â€"-are on the same network, eliminating their reliance on costly WAN links.

Reports are generated locally, with the results sent to the user's desktop over the WAN as a tiny HTML file. End users building queries interact with the reporting application via Citrix Metaframe, which uses far less WAN bandwidth than running the native Windows app from the remote server.

The efficiency of the new architecture alone has given Queensland Health net savings of AU$70,000 in the first year alone, as well as saving the organisation AU$250,000 worth of bandwidth capacity that was previously being used to prepare reports.

Despite centralising the database and reporting tools, the new project has maintained administrative independence for the individual hospitals by using Windows 2000's Active Directory to restrict access to BI resources by group.

Local administrators can delegate access to control to their own users according to internal policies, while collaborative interfaces between sites mean hospitals can also compare their performance with Queensland Health best practice by accessing reports from other locations.

Copyright © 2009 CBS Interactive, a CBS Company. All Rights Reserved.
ZDNET is a registered service mark of CBS Interactive. ZDNET Logo is a service mark of CBS Interactive.