CommBank's systems fell over yesterday while Australia's largest bank was preparing its infrastructure for Rudd's $10.4 billion fiscal injection, which is set to hit bank accounts next week.
CommBank CIO Michael Harte
(Credit: Commonwealth Bank)
Systems maintenance work at CommBank, which had been ongoing since last week, was behind yesterday's outage of its online banking system, NetBank, according to the company's CIO Michael Harte.
"It's the largest transactional website of any corporation in the southern hemisphere so we typically take pride in how much processing capability we have. And regrettably today, we disrupted a few clients for about two hours and they were unable to use NetBank," Harte told ZDNet.com.au late yesterday.
CommBank would on a normal day process around one million transactions, however, Harte said the $10.4 billion stimulus package, which Prime Minister Kevin Rudd announced in October, was expected to triple demand on its systems as the funds hit bank accounts.
"Right now we are making preparations for what we expect to be two to three times the normal volume at a peak load because of this cash injection," he said. "We could get three million hits," he said.
"We are planning for a huge increase in volumes right across the networks — ATMs, branches and online will be frantic," he said. "So we are making resilience improvements to our payments network, to our merchant acquiring network, to our cards issuing capability, and to NetBank, CommSee and the ATM network."
He explained that the cash injection had forced the bank to take extra measures to bolster its systems at a time when it would normally be conducting maintenance in preparation for the Christmas period.
"Pre-Christmas we make a whole lot of changes to the environment ... because we have made a commitment to our shareholders and customers that we will not have any disruptions to transactions or processing during that seasonal peak because people are relying on getting paid and buying Christmas presents," he said.
Yesterday's outage was blamed on a miscalculation of processing capacity required for the day while the bank was shifting loads between datacentres.
"This was load balancing within large datacentres and we were moving the volume from one datacentre to another in preparation for next week," he said. "And we had large volumes today, and because we were in the midst of that change we miscalculated the volume that would be happening today."
This glitch was not related to last week's processing error which had doubled withdrawals conducted on NetBank from some bank accounts.
"It was absolutely nothing to do with what happened last week," said Harte.
Harte defended the reputation of CommBank's technology and processes, and pointed out that customers were provided with several alternatives if they needed to make transactions while NetBank was offline.
"The important thing to note is that they got a message that said you can call the call centre to do your transaction, you can go to the branch, or come back if you want to do the transaction later," said Harte.
Harte also praised his team's approach to handling mishaps when they occur, and reaffirmed the bank's commitment to shareholders and customers.
"Say someone was overcharged a fee, we will always make that good; if someone got a double billing — those transactions we disrupted last week — we made those good and we made those good within 24 hours," he said.
Harte explained that the bank's batch process means that "if you make an error one night you can only fix it the following night. And that's also the difficulty when you have batch systems at the back and real time presentation online, which means that if you make a mistake it's instantly detectable".
So maybe they should think about updating their backend to a live system instead of the 60's style batch processing