Securify This! by Munir Kotadia

A hard look at the latest developments in IT security with a real world perspective.

Westpac's online failures expose weak points

Posted by Munir Kotadia @ 13:04 47 comments

Westpac, one of the biggest banks in Australia, experienced multiple hardware failures on Monday and at the end of last week, which caused its online banking service to fall over for hours at a time.

In today's e-commerce society, a faulty online service is the equivalent of a retail premise being locked with the windows boarded up. So you would think that very large companies would ensure that redundancies kick in during times of emergency.

After a Westpac customer and ZDNet Australia reader tipped us off, I spoke to the bank's media relations team to find out what was going on. The head of department, David Lording, boasted that Westpac has "backup systems on backup systems". Unfortunately, the facts are that the online bank was crippled several times, with Lording solely pointing the finger at "hardware failures".

What was the nature of these "hardware failures"? Well, Lording wasn't saying. But he had other things to say about the downtime suffered by Westpac. Below is an excerpt of my conversation with him on Monday afternoon. By then, the site had been inaccessible periodically from last Thursday to Monday:

Kotadia: So why is the Internet banking down right now?

Lording: Mate, we told you, there was a hardware issue.

Kotadia: But I was told it was fixed.

Lording: But that was last week.

Kotadia: So this is a new hardware issue is it?

Lording: Yea, that's right. A new one mate.

Kotadia: So do you not have any backup? If you have a hardware issue and there is a problem there, don't you have any backup systems?

Lording: Yeah of course we do mate.

Kotadia: So why aren't they working right now?

Lording: Well. We are rectifying the problem.

At this point he goes on to ridicule ZDNet Australia's readership base.

Lording: You go out to your little tech readers and you try and beat up a security issue that is not there. Well mate, I tell ya. I will be monitoring what you write and you better get it correct.

Kotadia: OK. So what you are saying right now is, there was a hardware issue last week and that was fixed and there is another hardware issue that you are working on. And you don't have any backup systems -- or you do? I am trying to figure out because ...

Lording: Look mate, what do you mean you are trying to work it out? All I am telling you is that we have had a hardware issue. It has reoccurred today and we are fixing the problem.

Kotadia: So do you have a backup system?

Lording: Oh mate get real, we are one of the biggest f**king banks in the country.

Kotadia: OK, so you do have a backup system?

Lording: Look mate, we have backup systems on backup systems ... but we have a hardware issue we are trying to rectify.

Kotadia: So if you have backup systems upon backup systems, then why is it that your system is down right now?

Lording: What are you trying to suggest?

Kotadia: I am trying to figure out why my readers cannot access your online banking services.

Lording: Well I am telling you. All I am saying to you, on the record, is that we have had a hardware issue.

Kotadia: OK.

Lording: That we are currently rectifying.

Kotadia: Has this hardware issue been happening all weekend?

Lording: No.

Kotadia: When do you think your customers will be able to access -- [interrupted].

Lording: I don't have a timeline at the moment.

I certainly hope this isn't the response Lording gives his customers when met with a difficult query.

Talkback 47 comments

    Backup system Anonymous -- 01/11/06

    Look, it's quite simple - the have a backup system. It's just that it's locked up somewhere in some storage room. And it's a 486. And it currently has a hardware issue. OK?

    ;-)

    PS. You don't expect a bank that makes billions every year to actually *use* some of that money to serve their customers, now do you?

    Wow Anonymous -- 01/11/06

    What a professional individual from Westpac.

    Lack of Understanding Anonymous -- 01/11/06

    You clearly lack understanding of the complex issues which surround mainframe systems such as those no doubt in use by the Westpac Bank. These sorts of systems no doubt have failover and redundancy measures in place to cover most eventualities, however to slam them as you have ... to demand to know when the inernet banking systems will be back online shows a real lack of understanding of the technical issues involved.

    Not really Anonymous -- 01/11/06 (in reply to #320071025)

    > You clearly lack understanding of the complex issues which surround mainframe systems such as those no doubt in use by the Westpac Bank

    Complex issues? Like: when the primary system fails, the backup system kicks in automatically.

    If mainframes are too complex for Westpac, they should use something else. Otherwise, what's so good about mainframes if they can't provide a simple failover?

    Rich Anonymous -- 01/11/06 (in reply to #320071025)

    I believe ZDNet were asking for the technical problem or at the very least an overview of what is going on and an eta on when things would be back up and running. In this instance the Westpac spokesperson handled the situation poorly.

    Technical issues happen, everyone in IT knows that but when it affects a major part of the population then Westpac should be more forthcoming to explain themselves instead of threatening media.

    I'm not usually one to defend media organisations but the bank is clearly showing its arrogance and disrespect to its customers with this sort of response.

    Poor Response Anonymous -- 01/11/06 (in reply to #320071028)

    I do agree that the Westpac employee in question did handle this issue poorly, and with a fair amount of arrogance. He just responded badly to the prodding of the media I wouldn’t like to be in his shoes at the moment.
    I just wanted to draw attention to the fact that sometimes it’s not easy to explain to end users EXACTLY what has gone wrong with a system.

    Poor response Anonymous -- 21/12/08 (in reply to #320071033)

    I do not feel sorry for Lording. As the Head of media Relations he would be paid a considerable amount to handle the prodding of the media and use plenty of verbs or adjectives rather than the ones chosen.

    Comments Anonymous -- 01/11/06

    Firstly I agree the interview was handled poorly.
    However zdnet should get it's facts before it starts "phishing".
    Secondly , why should any organisation tell you about private/confidential operations? Maybe so you can have a bash at trying to break the site. Come on , get real. Zdnet isnt perfect either. But they love to stir up a storm in a tea cup.

    ivory tower syndrome Anonymous -- 01/11/06

    As an employee of a company the "worker" obviously feels he has no obligation to explain something which is probably complex and also embarrassing. He's right!
    But rudeness and a threatening behaviour is not justified. Neither is the failure to understand that simple honesty, dare I say, humility would go a lot further in assuring Westpac customers that they are with a reliable and trustworthy bank

    By By Westpac !! Anonymous -- 01/11/06

    Excellent news piece....I cannot believe the arrogance of Lording and how rude he was to ZDNET !

    These "little readers", are the ones paying his large paychecks !!! If they only new how many professionals actually read or subscribe to ZDNET on a daily basis, and the money that we, as a collective, have at their banks. But he is obviously too busy with hardware issues to worry about customers, isn't he...

    I for one am considering all options. As a business customer, I want a professional organisation looking after my money, not one with a raving head of department.

    Btw Lording, best do some research, as I believe that companies like SEEK, IBM & McAFEE pump advertising revenue into ZDNET for a reason !

    Well done to Munir for holding on as long as he did.

    joker Anonymous -- 02/11/06

    I can understand how some might get their backs up if media are probing but this guy from Westpac is PR and he should know better.

    It's his job to answer media questions to the best of his knowledge and I'm assuming use spin to calmly explain the issue with minimal impact to Westpac. Unfortunately this spat makes him sound like a novice and that Westpac has something to hide -- which Munir found.

    Westpac's backup systems on backup systems don't work or failed.

    Westpac's online failures expose weak points Risky Business -- 02/11/06

    So why is Westpac, Comm Bank etc paying these so called Business Continuity Executives in excess of 200K a year? Despite them throwing so much money at these people and to solve potential problems, these banks continue to have hardware/software failures.

    Is the whole business continuity profession just a fraud?

    Goodbye Westpac Anonymous -- 02/11/06

    I got tired of Westpac's online banking being offline. I also tired of their arrogance towards fee increases becuase they are "one of the country's f**king largest". I ditched them. I moved my mortgage and daily banking. I suggest everyone else that has a problem with these big banks also move. My mortgage is now with ING and I also have an attractive online savings account with ING as well. I pay $5 per month for an all you can eat type transaction account with one of the big 4 and that is all I will ever have again with these big banks. I even held a Leaving Westpac party earlier this year to celebrate.

    So what happens when ING does something wrong? Anonymous -- 06/12/06 (in reply to #320071058)

    Wow, lucky there are plenty of banks for you to oscillate between each time something doesn't go your way. Makes for lots of 'Leaving XYZ" parties though.

    In my family, we just celebrate stuff like x-mas, birthdays, that type of thing; oh well, any excuse for a party I guess.

    Little tech readers? Anonymous -- 03/11/06

    I'm a business IT reader who visits ZDNet and a customer of Westpac. He's managed to *iss me off twice in one article.

    Nice PR

    Scapegoats make me angry D -- 03/11/06

    Lording may become the scape-goat of this issue, when infact there are underlying problems with the people above him.

    It's readily obvious that Lording isn't at fault!!!

    That's true Anonymous -- 06/11/06 (in reply to #320071101)

    That's true. Just because the PR is a prat, doesn't mean that this kind of scenario can't happen.

    2-3 years ago when most of the american ISPs were doing a major upgrade for a security release on Cisco, one of them had a failure and their backup routers didn't function and caused major problems for days.

    Backup and redundant systems can fail. Shame the PR prat was being difficult.

    Everyday Banking ? Online maintenance - AGAIN !! Anonymous -- 04/11/06

    Online maintenance

    Internet Banking will be unavailable from 00:01 to 06:00 (AEST) on Sunday 5 November 2006 for system maintenance.

    "Everyday Banking Accounts"...yeh, when you can actually get on !!

    Westpac non performance Alan Wills -- 04/11/06

    Typical Westpac attitude to their customers. I ditched Westpac after receiving seven (7) different figures for a payout figure for a card account and their systems were up and running at the time.

    I am still waiting for some sort of explanation eight weeks after lodging my complaints on the 5 day guaranteed responce email.

    Alan Wills

    Westpac non-performance up-date. Alan Wills -- 18/11/06 (in reply to #320071116)

    I received a email from Westpac Complaints Manager Mr. Alan Fleming on 15/1106 trying to justify Westpac's non-performance a la David Lording.

    I will quote Mr. Fleming once and in my humble opinion I think it sums up the Westpac Corpprate attitude to customers and their money.

    Quote: "Credit Card payout figures are more art than science."

    So where do they get their payout figures from?

    These people are, for a very large profit, looking after billions of other peoples money, should they be allowed to? Lets hope their corporate accounts are more science than art!

    Are we missing something...??? Anonymous -- 06/12/06 (in reply to #320071552)

    This looks suspiciously like one of those sentences that are part of a bigger circumstance. Why not publish the whole letter/s between you and this guy and then we can make a fair assessment of the situation.

    What you reckon - are you prepared to show the full picture, or just the part that suits your blog?

    All banks SAM CROWTHER -- 06/11/06

    ok, so yes on this issue we can bag westpac, and i did on many other issues, for me they became part of 'connexafonepac' connex, vodafone and westpac could all fail impressivly to do what they state their cqapable of, i swithched top optus, moved to where i could walk to work, but am still a westpac customer for the sole reason that when (read 'if') i actually walk into a westpac branch, i still get good customer service. which is more than i can say for the other banks..

    Yeas they ALL 'rob you' and ripp ya off, but better the devil you know...

    What a typical Westpac Response- sack this clown Anonymous -- 06/11/06

    I was a Bank of Melb customer initially, then became a westpac customer when they took over.

    Not for long. The stupidity, arrogance, lack of customer service, incorrect information and lack of customer service did me in. I left.

    And Mr Lording, just let me say that you are (based on those comments) one of the worst PR Hacks I have ever read.

    Westpac sack this clown!

    Is this for real? Anonymous -- 06/11/06

    It is truly hard to believe that this conversation was as colloquial as described. Westpac surely have clear guidelines as to media engagements and as a spokesperson Lording would be briefed to answer questions in either a non-committal or not to comment at all.

    Failed Response for Shareholders & Customers Anonymous -- 07/11/06 (in reply to #320071149)

    Munir is a senior journalist...have you ever read his other pieces ?...if you read his previous pieces, you will see that he is no nonsense and will not beat around the bush to get answers....

    I'm sure that Westpac do indeed have media guidelines, that Lording has quite obviously failed to follow...the problem in this case, is that Lording believes that he shouldn;t have to explain to both customers and shareholders, why Westpacs online systems keep failing. In this respect, he has failed in his duties.

    Should he have taken the question on board, and given a clear and concise response - sure,...but he is obviously clueless in that regard. What concerns me more, is that he works in IT, and has no idea of the size of ZDNET, ZDNET worldwide and it's readership !!

    Was this comment written by Munir? Anonymous -- 08/11/06 (in reply to #320071178)

    Sounds like a promo piece for Munir.

    Security Dandasaurus -- 08/11/06

    Hmm interesting that he should mention a security issue when the issue you were pursuing was clearly different. And also interesting that this 'hardware failure' should occur just a couple of weeks after a major DDNS attack on the National Australia Bank. Maybe the spin doctors have decided that customers expect 'harware failure' but get edgy when you mention organised criminal attack.

    At least he didn't give the usual non answers Anonymous -- 08/11/06

    I think you have to hand it to the guy for showing a little human emotion instead of the usual robot PR spin. He was telling you what he had been told and that's probably not everything.
    I think the guy just got jack of stupid pointless questions.
    The bank has backup systems and they didn't work properly. I'm sure that the Westpac site doesn't promise 100% uptime, if it does sue em.

    The fact that these types of systems are designed with no single points of failure means that it probably wasn't a hardware failure in the true sense ie something just broke twice, but unless there is lost of personal or financial data is involved I don't think they are going to spill their guts.

    Full of it Anonymous -- 11/11/06

    I reckon that the "journo" was feeding his inner-tabloid soul and hamming up the story a notch. It's hard to believe a PR/Media hack for one of the countrys biggest banks (who love to apologise for the sake of it) would say f*** off to a journo.

    Readers - don't believe everything you see, if you do then please remove your leaking brain from the tv dinner tray your head fell in.

    Believe vs Understand Anonymous -- 13/11/06 (in reply to #320071312)

    You may not want to believe everything you read, but perhaps you should try a little harder to *understand* what you read. The PR guy didn't tell the journo to F off. So, you were saying?

    Response from from Westpac Al -- 14/11/06

    I was wondering if ZDNet have had any formal response from Westpac regarding this interview. If Westpac remain silent then readers of ZDNet can and will assume that Westpac stands behind this transcript.

    Come on Westpac when you have stuffed up be brave enough and admit it - you may keep some respect...

    LOL RESPECT? Anonymous -- 15/11/06 (in reply to #320071423)

    Westpac are one of the most arrogant banks I have ever dealt with. They respect no-one, and I don't think they give a damn. Why on earth would they bother to read or answer any critique?

    Like I posted earlier the (Westpac) clown should be sacked!

    Why would they respond... Anonymous -- 06/12/06 (in reply to #320071423)

    Why would anyone respond to second hand information. Because it's a ZDNET journo? It MUST be a 100% faithful, reliable account of the actual conversation.

    Funny, when the journo's are tearing strips off a small business or an individual, everyone's shouting at us not to believe what we hear in the media.

    It's the fairweather syndrome and you got sucked in.

    He probably didn't know what was wrong Anonymous -- 16/11/06

    The PR guy probably did not know exactly what was wrong. He may have been told a sanitised story to put out. However, he is a PR pro so he should have been able to handle being mushroomed. Possibly even Westpac did not know what exactly happened, at that time, as a major part of their system has been outsourced to IBM.

    OTOH given that he was unable to handle pressure at that time he should review his long term career path as he may (like many of us) be in the wrong job.

    As already said, the problem was obviously quite complex else it would have been fixed much more quickly than has happened.

    I have worked in mainframe IT for a long time and I would not like to have been anywhere near this problem. The heat must have been intense and it can get to you. I don't think I even would like to be their for the post problem meetings. The knives would be out.

    I am happy I dont bank Westpac but, with a bit of misfortune, this could happen to almost any bank or big organisation.

    Glad I moved banks.. Anonymous -- 17/11/06

    Interesting, if the head of media relations handles the media with expletives (despite the pressure obviously in effect at that time of failure) it sure re-iterates my glee at having moved all my business to ANZ.

    Luckily, they haven't let me down yet (I was regularly subjected to Westpac online banking being unavailable) and as a net savvy user, I like being able to do what I want, when I want.

    What I want, when I want... Anonymous -- 06/12/06 (in reply to #320071527)

    So, as a 'web savvy' individual, you don't tolerate ANY downtimes, upgrades, faults, etc to ANY companys' systems.

    Oh yeah, that's web savvy.

    Outsourcing Ian McDermid -- 18/11/06

    How much of this has been caused by Wespac outsourcing to IBM GS. They must be the worst organisation with their "Press Ganged" workforce. Westpac only offered half redundancy o those who did not want to join IBM GSA as it was at the time. Those with mortgages had no choice but to sign up.

    This is really an non security issue! Anonymous -- 21/11/06

    The on-line or Internet access to the Banks Systems are only a front end to a Secure Main Frame Installation where secure comms are used. If you think that when you go into the branch they utilise something like the internet to access the system think again. Its not TCP/IP performing the comms, thank GOD. The issue is that even if the bank got hit by a DDOS attack it cannot go anywhere except just to deny internet access. The banks core business runs on secure SNA( I think) and site on multiple Mainframe's. You cannot damage the banks core operating system via the Internet.

    If you have the thought that the bank runs on hugs PC type servers connected via TCP/IP then run away overseas in a smart hurry.

    At least in this country we have kept our Main Frames and secure comms links and you cannot take them down by the internet interface. Thank GOD for common sense secure Australian thinking!

    Typical techo answer to a customer service problem Anonymous -- 23/11/06 (in reply to #320071618)

    Man have you missed the point. Customer service starts at the top and clearly this is what wetpac staff learn. Their customers are idiots and need to be treated that way. What the hell does yoiu useless answer have to do with anything, Surely the bank has proper systems to ensure these problems do not occur - Well ZDnet for exposing the frauds

    little tech reader... Not known -- 26/11/06

    Sounds like Lording doesn't like tech and doesn't want to know anything about tech at all. So why not find a job in a company in which there are no computer at all. He probably will not be bothered by "little tech readers" then.

    Westpac runs Windows Servers! Frank Daley -- 27/11/06

    How could anyone trust an organisation that runs its critical infrastructure on Microsoft Windows!!

    Down again 19-FEB-2007 Morgan -- 19/02/07

    Looks like their site was down again today.

    I am also a St. George customer and have never experienced any outages with their online services.

    The technical people at Westpac seem to have no clue and should all be fired immediately.

    Down Again 24-03-2007 Anonymous -- 24/03/07

    I work late many nights, and use WestPac's Internet banking frequently. I may be mistaken here, but they seem to be down nightly at least once a week. Compare that to my overseas account which is virtually never down. I know WestPac is using complex systems, but the fact that my overseas bank gets it right shows me that is can be done.

    The ever degrading westpac Anonymous -- 04/09/07

    Westpac is just terrible, thanks to their ongoing cutbacks their branches are declining in number, the amount of tellers in each bank is declining and customers have to wait sometimes up to 20 minutes in long lines just to bank a cheque (like i did in their strathfield office), their cut backs are diluting their services to the point they are completely degraded.

    In the branches they encourage customers to use their online banking but just like the branches that is terrible too - its often down for many hours at a time, every week!!

    It seems like the profit hungry westpac can justify their ongoing cut backs and resulting poor service becuase they feel that their customer are ignorant fools who are more than happy to endure 'their cattle class treatment'.

    I am going to a credit union !

    Ex Westpac Business Customer Anonymous -- 04/09/07

    I was a business customer with Westpac a few weeks back before pulling most of what i had into a Bank West business account. I found them to be very arrogant as mentioned by other members especially when I had called in a query, the customer services representative sounded on the phone as if he didn’t care, he asked me to go into a branch to close one account and this was after being transferred to 2 other representatives so they could verify who I was before proceeding. At one point the guy even sighed on the phone. There all nice when there setting your account up, but when your in there system, your just a number, no matter what you have. Sorry to get off the outage topic.

    the Fabulous WestPac Service Anonymous -- 09/10/08

    My experience with the "fabulous" WestPac Service is that they are not even able to fax a document from one branch to another branch. And that they are unable to give you a 10% loan for a deposit on a home that costs $440,000,- after you've sold your house for the price of $532,000.- because you did not apply BEFORE you've sold your home!
    It seems all rather logical, doesn't it.
    As a result I'm closing all my accounts with WestPac and I'll go somewhere else. I sugggest you do the same.

    westpac online banking Allan Bellista -- 05/01/09

    Yet again, online internet banking is down again.

Add your opinion

Munir Kotadia

Munir Kotadia

Producer

[+] Read bio

Latest Videos

Sponsored content

Power Centre - Content from our premier sponsors

Tags

Back to top

Featured