How ATO found an antidote to the BI poison chalice

When Philip Hind joined the Australian Taxation Office as chief knowledge officer, the task of managing the organisation's data warehouse was widely viewed as a "poison chalice". How did the ATO evolve to make its warehouse-dependent BI applications a critical enterprise tool?

"About three years ago when I applied to work with the Taxation Office in this role, I was informed that part of the role of the chief knowledge officer was that I got to be the designer of the data warehouse," Hinds said during a presentation at the Gartner business intelligence seminar in Sydney.

"I took it from the way that information was conveyed to me that that was some of a poison chalice."

Although the ATO had operated data warehouses, largely powered by Teradata, since 1994, the analyses produced were "often unreliable and contradictory," Hinds said.

"It was seen as somewhat expensive, although that was largely driven by the fact that capacity upgrades kept coming along without being foreseen."

"From the user perspective it was seen largely as a playground," Hinds noted. "I took it from that that I had quite a challenge."

A major turning point for the ATO's data warehouse approach came with major changes to legislation in 2000.

"Our information architecture has matured and gone through various phases over that period. With the introduction of GST and the growth of super product, the data warehouse suddenly became a very complex system. It wasn't well architected from an information perspective."

"In recent years we've been assiduously re-architecting it [to become] a truly relational enterprise data warehouse," Hinds said. "We're well on the way to making that transformation. A major change was a move to consolidate dozens of transactional and case management systems into a smaller group of core applications, with information accurately staged between different systems.

"In our old data warehouse, we didn't have this data staging infrastructure properly set out."

The new system has a subject-based data mart structure, and most users draw on information from this area. Only highly trained specialist users can directly access the central relational data store. "You can't afford to have people doing data discovery that they haven't been trained for, don't have the right security clearance for, and that you can't afford to support."

The need to maintain continuity of service has meant a gradual change process rather than a major switch. "We had to work with the hybrid of a legacy environment and a new environment within the same platform. Effectively, what we're doing is incrementing and growing the new enterprise data warehouse structure and gradually decreasing the volume of data in our legacy data store."

"Around 2004 we decided we needed to move into the analytics data mining area." The ATO purchased SAS for analytic processing, and standardised on the use of Cognos as an enterprise reporting suite. Five core report types were identified and standardised.

While Hinds is pleased with the outcome, he doesn't pretend such a task is easy. "It has required discipline in trying to implement properly. You never get it right the first time and you have to keep going back to the core principles."

Having a solid infrastructure is vital, he said. "It's about the BI infrastructure, not the analysis, even though the analysis is where the value is."

"Controlling the BI infrastructure as an end to end process is an ongoing task -- I really don't hear people emphasising this enough. It's really the hardest thing from an IT perspective, because most IT departments don't think like this."

"BI has to be about how you link human intelligence capabilities with your BI infrastructure. You have to spend a lot of time thinking about your business."