Secrets of starting a data warehouse from scratch

Being able to build a data warehouse right from the beginning of a company's life can eliminate some of the pitfalls typically associated with the project, but doesn't necessarily eliminate the most obvious one: uncontrolled data from multiple sources. And it adds a whole new challenge: do you have enough data to start planning for intelligent analysis?

I was reminded of this during a recent interview with Martin Fletcher, the chief operating officer for the Australian operations of Mortgage Guaranty Insurance Corporation (MGIC), which set up shop down under in 2006 and provides insurance for loan companies against possible credit losses.

Right from the beginning, Fletcher was determined to set up a data warehouse to allow proper tracking of the business' performance.

"I've set up companies previously and it's very clear that to get to the stage where you can analyse the data, you need it all in one place," he said.

MGIC didn't want to simply import the technology used by the American parent company. "We're a subsidiary of a very successful US business, but as we looked at systems for our home office team, they've got something that suits a business of 3,000 people and we've got 200. The scale of it was overkill for what we needed."

Effectively starting a new company meant that no existing systems had to be dismantled. "From day one, we've hired people in the team here who are very keen to design their own processes from scratch," Fletcher said.

However, that didn't mean that the familiar tangle of spreadsheets and individual documents was entirely absent. Because the company had to be in operation from the word go, individual monthly reports had to be generated until the data warehouse was up and running.

That scenario was probably unavoidable, according to Fletcher. "Realistically, we couldn't have started it any earlier. Everyone's delivered the pieces of the project on time."

Fletcher originally thought that defining the long-term requirements could be carried out in-house, but MGIC IT leader Joe Gullotta convinced him that some external advice would be useful.

"My initial thought when I talked to Joe was we would be able to define our requirements, but early on Joe was advocating using consulting experts. When you look at your business, you may not understand the best way to organise your data to analyse it." MGIC eventually engaged Altis Consulting to help plan the system.

Accessibility was a key requirement. "As we started thinking about the future capabilities of the BI tool, I wanted it to be very user interactive. I didn't want a team of reporting analysts who were the only ones who could use the system."

Even with the basic architecture in place, planning for future needs was tricky. "The biggest challenge was we didn't have a lot of raw information in our portfolio because at first, we didn't have a lot of loans insured. Often we were describing qualities about the portfolio which we didn't have any data on. That was one of the problems we had to overcome."

With the MGIC project now well-established and able to deliver daily analysis on a wide range of metrics, what advice would Fletcher offer to other companies facing similar challenges?

"Definitely spend time upfront thinking about what you're trying to achieve, not just today's requirements. If you work closely with the vendor you can get a vision for what's possible in the future. Distinguish bells and whistles from essential things to drive the business forward. We spent a lot of time thinking about what we wanted as minimum. Then once we met with the vendors we had a much more realistic discussion."

Talkback

Add your opinion

In order to post a comment, you need to be registered. (Sign In or register below)

Post your comment

Terms of Service - As a ZDNet registrant, and by using this service, you indicate that you agree to our Terms and Conditions and have read and understand our Privacy Policy.

ZDNet Australia Live

Before accusing me of fudging the figures, that was the percentage in April, the latest available. It seems that as the advantage of the ...

58 minutes ago by GregoryB1 on NBN FUD: will Abbott ever learn?

Currently about 50% of connections are at the 100Mb/s rate.
As a consequence, ARPU is significantly higher than the projected figures.

1 hour ago by GregoryB1 on NBN FUD: will Abbott ever learn?

Currently about 50% of connections are at the 100Mb/s rate.
As a consequence, ARPU is significantly higher than the projected figures.

1 hour ago by GregoryB1 on NBN FUD: will Abbott ever learn?

Wireless currently carries less than 2% of total internet data traffic. Simply to carry the existing traffic, we would need 50 times the ...

1 hour ago by GregoryB1 on Blowing the digital dividend on wireless NBN

The stupidest part about a wireless solution for the burbs is that it will actually cost more to put an antenna on the roof to get the si...

1 hour ago by GregoryB1 on Blowing the digital dividend on wireless NBN

The problem is not range of the cell in the urban areas where Turnbull wants LTE instead of fibre, it is the number of users. In urban ar...

1 hour ago by GregoryB1 on Blowing the digital dividend on wireless NBN

After the Second World War, the pursuit of pleasure domains the entire world atmosphere, Lancel (Lancel) to adapt rapidly into the demand...

2 hours ago by PokArrackpask on Spam sees Westnet blocked by BigPond

RT @DellEnterprise: Dell Secureworks talks with ZDNet about Android's biggest #security flaws - http://t.co/LSFLQVFq #infosec

NBN users opt for 100Mbps: Customers are picking the top fibre plan that is available on the National Broadband ... http://t.co/sjtFSU3g

"Customers are picking the top fibre plan that is available on the National Broadband Network (NBN), more than a... http://t.co/M3P24Htn

Another thing I found so misleading here is the step on how you assume to make the USB bootable . (The NTLDR needs to be renamed to USBNT...

4 hours ago by WindowsAnalyzer on Boot Windows XP from a USB flash drive

You can also use the help of these links, just incase your stuff failed, I probably got Windows build by using the Pebuilder as per the i...

4 hours ago by WindowsAnalyzer on Boot Windows XP from a USB flash drive

RT @CorrieB: An iPad for every child: Inevitable or impossible? http://t.co/I7uS8l9s Thx to @timbuckteeth for this; http://t.co/jxkqIRIp

RT @MADinMelbourne: roxon "will enable more families to access credit" @MLolderandwiser: Privacy Act amendments http://t.co/Mv4c7PC2 via @zdnetaustralia

NBN users opt for 100Mbps - ZDNet Australia http://t.co/fLfHMzPn #australia #technews

RT @konradski: Whaddayaknow - turns out Wi-Fi CAN interfere with a plane's navigation systems http://t.co/ospQCU2S

This story has been voted 5 times in the last 24 hours!

7 hours ago, NBN's Tassie upgrade to cost $1.3 million

Sorry no deal Cinders, I'd rather send my money to someone and watch them desperately try to stop the NBN as this has much better enterta...

8 hours ago by Hubert Cumberdale on NBN users opt for 100Mbps

What else can you expect from a Dodo customer?

8 hours ago by Hubert Cumberdale on NBN users opt for 100Mbps

NBN users opt for 100Mbps - Communications - News - ZDNet Australia: NBN users opt for 100Mbps - Communications ... http://t.co/btB9gKWg

NBN users opt for 100Mbps http://t.co/xKqEb4bE via @zdnetaustralia

Biometric bugs too dangerous for public? http://t.co/8JLz5tdF via @zdnetaustralia

Oh please dont be unkind, I gotta have some fan's. btw I agree I dont set the standard, but who does I wonder?

10 hours ago by Doubt on NBN users opt for 100Mbps

You agree but give him thumbs down... I think you'd better take the medication before one of your alter ego's Fred/Frank/Frergers appear...

10 hours ago by Beta on NBN users opt for 100Mbps

Exploring: http://t.co/rT7RPZLA

+1

10 hours ago by Beta on NBN users opt for 100Mbps

War talk dominates #AusCERT 2012 - http://t.co/SlBpMj0c - #security #cyber

So we agree it was a stupid idea and even stupider comment then ;-)

10 hours ago by Beta on NBN users opt for 100Mbps

Not you obviously ;-)

And stop giving yourself thumbs up FFS.

10 hours ago by Beta on NBN users opt for 100Mbps

Ok Beta, understand now, just one point who sets the standard?

10 hours ago by Doubt on NBN users opt for 100Mbps

Oh no Beta you misunderstand me. I like my waterfront home and deep water jetty, it's those "other" people who can move to Willunga.

10 hours ago by Doubt on NBN users opt for 100Mbps

I agree with you Magnus, but really most people like living on the coastal fringe.

10 hours ago by Doubt on NBN users opt for 100Mbps

Travel Tech Q&A: Skyscanner's Ewan Gray http://t.co/vYexrDwu #ipad

Exploring: http://t.co/YNVjdrct

Exploring: Travel Tech Q and A: Skyscanner's Ewan Gray: Ewan Gray, Skyscanner's director for Asia ... http://t.co/bNLCyobv #ICTChallenge

Exploring: Travel Tech Q and A: Skyscanner's Ewan Gray: Ewan Gray, Skyscanner's director for Asia ... http://t.co/HEPuJgyt #ICTChallenge

#NewSouthWales ditches registration stickers 4 light #vehicles in favour of #technology http://t.co/xX5N0Rp9

Another use is city based top surgeons using 8K resolution monitors to provide real-time assistance to country surgeons and doctors to op...

11 hours ago by Magnus on NBN users opt for 100Mbps

Anonymous hacks Reliance's Internet filtering server - ZDNet (blog) http://t.co/uObU1HBP http://t.co/0UBXxwX4

Which Windows will make for a better tablet? http://t.co/4mAHg850

Listening to @stilgherrian cover AusCERT and cyberwar, http://t.co/6lGUEz8H

Travel Tech Q and A: Skyscanner's Ewan Gray http://t.co/VN5tGJzC

#Westpac Board goes paperless with #Ipads with #Tabula #App http://t.co/duxuj2fd #Cybersecurity #Bank

Microsoft is serious about open source??? http://t.co/mqQGgta7

@joedamato just try varying caps randomly. Maybe they do this http://t.co/1FN5FwYv

NSW outlines datacentre migration plans - Hardware - News - ZDNet Australia http://t.co/OQfUl0D1

"on the new fast Internets everyone wants the fast plan" #orly #nareally #yarly http://t.co/kvfCa84A

Chrome overtakes IE: does it matter? http://t.co/e4SILk8a

A ZDNet study showed that British Facebook users are drunk in 76 percent of their photos.

The HDMI cable ripoff and why retail is really dying http://t.co/eFT7zEW7

Travel Tech Q and A: Skyscanner's Ewan Gray http://t.co/IUysbyKf

Travel Tech Q and A: Skyscanner's Ewan Gray http://t.co/V7vL5QB9

ZDNet reports Microsoft launches its own social service http://t.co/VJS5BkwF

by http://t.co/vmlLt4bh: Travel Tech Q and A: Skyscanner's Ewan Gray: Ewan Gray, Skyscanner's director for Asia P... http://t.co/4bfDRXo4

Travel Tech Q and A: Skyscanner's Ewan Gray http://t.co/CtNlVWN7

Travel Tech Q and A: Skyscanner's Ewan Gray: Ewan Gray, Skyscanner's director for Asia Pacific, shares some of h... http://t.co/ZxjpmqiM

This story has been voted 12000 times in the last 24 hours!

2 days ago, Is Bill Gates a great leader?

Facebook Activity

Keep up with ZDNet Australia

ZDNet Events Calendar

ZDNet Events Calendar