Breaking News: Internode: leave us, but it's not our fault

ZDNet / Outage / Story

Internode resorts to disaster recovery

By Suzanne Tindal, ZDNet.com.au on September 22nd, 2008

The email accounts of Internode users were stranded over the weekend as the internet service provider battled a major storage infrastructure failure and was forced to fall back to its disaster recovery centre to restore lost services.

Simon Hackett
(Credit: Internode)

Customer email, corporate Web hosting, personal Web hosting, Web mail and customer Web tools took a hit on Friday when some of the company's systems were taken down by a major hardware failure affecting multiple servers.

Corporate Web hosting was restored early Friday afternoon, although other services remained down until later. Email was the last to be restored, with the percentage of customers whose services had been fixed creeping up until Sunday afternoon, when the company announced all services were up and running. The company insisted no email was lost.

Internode managing director Simon Hackett wrote on broadband forum Whirlpool that the outage could have been over four hours after it started, but once the recovery process had been completed, the system began crashing within minutes of being fed production traffic. This required the entire file system, holding 22 million plus files, to be rebuilt.

Complications aside, the outage never should have happened, Hackett said in another post.

"We have a very large investment in a very high end dual-site/fully redundant storage area network system that just isn't supposed to do this — ever. Clearly, it has — and yes, the vendor of that system has been involved (from 20min after the initial failure) in being a part of 'the solution' here, too," he said.

The failure also involved the server operating system. When things were working again, there would be an investigation into exactly what caused the disaster, Hackett said.

"The wrap-up here is going to involve two separate vendors (SAN and server OS) debugging some failure modes neither of them has seen before, some changes in approach in handling the mail cluster to avoid the restoration process (in the unlikely event its ever needed again) from taking so long, and a variety of other related measures," he said in another post.

"This is a rare and extremely annoying thing for us, as well as for you — and we're absolutely determined to avoid it becoming a habit," he said.

Internode did not respond this morning to an emailed request for comment.

Talkback

Internode

So, S. Hackett does not issue an apology to his customers or discuss financial compensation for business loss attributed fully or partially to an issue S Hackett has taken full responsibility for an online public forum? Arrogant. If this was Telstra, the media would be sinking in their collective boot. Hackett is too busy providing an excessively technical explanation (excuse) on a blog, rather than offering an appology or reassuring it's growing customer base. Disappointing

DCDC September 23rd, 2008
Report offensive content Reply (0) (0)

Arrogance? Non!

Actually I found Simon's posts a refreshing change of honesty in these times of corporate spin and misinformation. Given that Simon is probably aware that a good percentage of his customer base uses Whirlpool (Whingepool, call it what you will); posting updates there was a master stroke of information dissemination.
BTW - I'm not an Internode customer, but the attention to their customers (rather than the media) they've shown here is the sort of act that attracts.
Cheers.

DRDR September 23rd, 2008
Report offensive content Reply (0) (0)

'The sort of act that attracts'??

What they are saying is that they relied on hardware redundancy. When that let them down, they could NOT perform a cold restore.

Redundancy does not replace Backups and Restores.

AnonymousAnonymous September 24th, 2008
Report offensive content Reply (0) (0)

What do they use?

I'm very interested in what redundancy and DR solutions they tried to employ (without success). Note that I'm not blaming the solution itself, as most IT problems are caused my mis-configuration, lack of training, and other "people"
issues, however some DR solutions keep that in mind by being able to perform virtual "fire drills" to test whether or not the DR solution has consistent and usable data (e.g. with Veritas Cluster Server and some other state of the art DR softwares).

And yes, I agree, part of a DR strategy *needs* to be traditional backup/restore methodology.

AnonymousAnonymous September 24th, 2008
Report offensive content Reply (0) (0)

RTFA

They did restore, then the restored system immediately started experiencing problems.

If you're relying on an ISP-provided e-mail account (@internode.on.net in this case) for your business, you can't value your e-mail very highly. Stop grandstanding.
ISP-provided e-mail is a no-extra charge service aimed at home users. If e-mail is important to your business get your own domain, host it on a small business server or a reputable outsourced e-mail provider.

E-mail is not an instant communication medium. If you're in a hurry, use the phone. The outage was over the weekend, replying to e-mail on Monday is (in most cases) going to be perfectly acceptable.

My Internet access was uninterrupted during this outage, and I'm very happy with Internode (so far, switched a few months back), and my e-mail was available (and obviously I'm not using Internode's mail).
Don't overstate this problem's impact.

I have to applaud Internode for their customer service and Simon for his openness and honesty. More ISPs (and other companies) should do the same, instead of bullshitting us.
Comments like some of the earlier posters here have made only encourage the corporate PR nonsense instead of honesty.

AnonymousAnonymous September 24th, 2008
Report offensive content Reply (0) (0)

Refreshingly Honest

As anyone who views/contributes to whirlpool forums knows, Simon responds to most Internode posts and is open, honest and displays respect for his customers by always providing a clear, non corporate view from his side on issues. eg ABCiView. You will not get such openness from any other ISP.
Of course Simon is sorry for what happened but it was out of his control so no apology is wanted by me or most of his reasonable customers.

AnonymousAnonymous September 24th, 2008
Report offensive content Reply (0) (0)

No apology?

Simon (or one of his clones) may respond to many posts but they are not open or honest hy any means. They are defensive of the company, pull on emotions (poor little ISP), offensive towards regulatuion and the competitors and political most of the time.

As to an opology not being needed, is that because it happened on a Friday night and impacted businesses and individual users less then if it happened on a Monday morning?

Also AnonymousAlso Anonymous September 24th, 2008
Report offensive content Reply (0) (0)
Add your opinion

In order to post a comment, you need to be registered. (Sign In or register below)

Post your comment

Resource Centre Useful content from our premier sponsors

ZDNet Australia Live

A user from Canberra measured 15468kbps @ Broadband Speedtest.

5 minutes ago, Click here to find out how fast your internet speed is.

Telstra completes the majority of a $280m fibre-optic broadband roll-out for NSW DET http://bit.ly/aBmMVm

@zdnetaustralia yeah not mentioned much before election even though I knew there would have to be some

http://twitter.com/HushCryBaby - Back to school with Adobe's Education Exchange http://bit.ly/c4hiVq http://bit.ly/cupitC

Twitter, Facebook and the tornado - ZDNet (blog) http://bit.ly/brru5s

Office for the iPad? Or are Office Web Apps good enough?: But if and until the Softies decide to deliver Office on... http://bit.ly/bxaBqh

@icanseeit 7 per cent wireless and satellite...

RT @icanseeit: And I though the NBN was all fibre and therefore better than coalition plan! “@zdnetaustralia: now we hear bout NBN Co's wireless component

Salesforce takes Chatter to smartphones: Salesforce has released its collaborative networking platform, Chatter, i... http://bit.ly/aeJhWa

Let My Smartphone Go!: The majority of popular mobile phones run on one of five operating systems: Symbian, Blackb... http://bit.ly/cfx5ex

And I though the NBN was all fibre and therefore better than coalition plan! “@zdnetaustralia: now we hear bout NBN Co's wireless component

HulloMail 1.2 for BlackBerry, ZDNet latest review http://bit.ly/d8jeMZ

RT @zdnetaustralia: Telstra completes the majority of a $280m fibre-optic broadband roll-out for NSW DET http://bit.ly/aBmMVm

Let My Smartphone Go! http://bit.ly/9gADuO

RT @zdnetaustralia: With regional Aus getting NBN priority, the cost will not increase http://bit.ly/aAiLXD

Google Instant: search as you type http://dlvr.it/4xtrs (ZDNet)

ZDNET UK: Google Instant: search as you type http://bit.ly/9DhDPV

RT @phonedevs: Qualcomm looking to Android for initial future developments - ZDNet UK (blog) http://bit.ly/ceYFYk

Well id rather see the 40 billion put into making my trip to work 20 minutes faster and reducing the price of houses so that i can actual...

39 minutes ago by rbosward on NBN roll-out rejig adds no cost: Conroy

Internode's Simon Hackett says his company will lose customers to BigPond because of wholesale port costs. http://bit.ly/cDosCu

NBN roll-out ramps up post election and rejig adds no cost: Conroy http://bit.ly/cnNdET

Oh you're back again old faithful, LOL! So you're no longer claiming "binding agreement" and now asking about a "for...

49 minutes ago by RS on Telstra walks while telco D-team squawks

Even all the spectrum in the world added together is hundreds of times slower than a single hairline strand of Fiber. 100,000,000,000Mbp...

58 minutes ago by Duideka on Much cheaper NBN wins it by a whisker

The nature of wireless is that any frequency band is inherently "capped" - there is a maximum ability to carry data regardless of...

1 hour ago by Marshall2 on Much cheaper NBN wins it by a whisker

I too am a Telstra customer. I even bought a few TLS shares when they went ex-div a few weeks back (after doing likewise previously and s...

1 hour ago by RS on Telstra 'price squeeze' claims go to ACCC

NBN roll-out rejig adds no cost: Conroy http://bit.ly/d4LIDI via @zdnetaustralia #NBN #openinternet

brando1... If WiMAX2 does actually do as they claim in 2012, great. I'm sure all the NBN supporters (like me) who welcome technology ...

1 hour ago by RS on Exetel chief: 'God help us all' on NBN

Australia has the strictest laws in the world on Telstra controlling it. If you want to see what happens without control go to America. O...

1 hour ago by Brumby on Telstra 'price squeeze' claims go to ACCC

There is some evidence the US Tea Party are bankrolled by The Koch Brothers, as reported in the New Yorker http://www.newyorker.com/repor...

1 hour ago by Blackbobs on Govt, use tech or have tech use you

It's easy to throw comments like that out there. It seems sort of reasonable on the face of it doesn't it? Things are always impr...

1 hour ago by neil_mc on Much cheaper NBN wins it by a whisker

Speaking of political leanings... It was said to be $26B!

1 hour ago by RS on Much cheaper NBN wins it by a whisker

Oh look, I've gained a new troll, LOL... I have NBN leanings big difference what yourr excuse? Don't see you bagging Scott, hmmm...

1 hour ago by RS on Abbott pledges vigilance on NBN stuff-ups

And am I wrong? Is Australia listed on the ASX? Well... Anyway... it has been projected by McKinsey/KPMG that the NBN will be $26B... A...

1 hour ago by RS on Abbott pledges vigilance on NBN stuff-ups

Thanks TimC3, broken promises aside it seems that you either agree or re-iterate everything I have said in this article and previous arti...

1 hour ago by Peter Carr on Open letter against taking Gershon funds

Does that mean that Telstra have been denied access to exchanges by, ah Telstra and Telstra were part of the case against Telstra, where ...

1 hour ago by RS on Telstra 'price squeeze' claims go to ACCC

And you hang other people for their obvious political leanings :)

2 hours ago by mwil19 on Abbott pledges vigilance on NBN stuff-ups

photos of the NBN rollout - http://bit.ly/dy7vrC

Telecoms giant BT is hiring 300 staff across the Asia Pacific region http://bit.ly/bKRHRD http://fb.me/GHfzPjUU

Check out Google Instant search. They say it's going to make searching easier, but is it distracting? Let us know... http://fb.me/DIPEBhCz

RT @fugazied: Fibre in Tasmania #NBN http://www.zdnet.com.au/houses-linked-up-in-tassie-nbn-photos-339304569.htm?omnRef=NULL

This story has been liked 5 times in the last 24 hours!

TechLines 6: email is a jack of all trades http://zdnet.com.au/339305877/ - so I have too much email because I have no process???

RT @zdnetaustralia: Telstra completes the majority of a $280m fibre-optic broadband roll-out for NSW DET http://bit.ly/aBmMVm

Realestate.com.au adopts cloud email http://zdnet.com.au/339305876/

Internode: leave us, but it's not our fault http://zdnet.com.au/339305865/

Internode: leave us, but it's not our fault http://zdnet.com.au/339305865/

really? i'll believe it when i see it ... "ACTA warms to ISPs?" http://j.mp/bcu7uh

Information security systems failed to prevent a Police analyst from leaking information on raids to bikie gang? http://ow.ly/2AYYA

Qld Uni start-up scores MRI grant: Magnetic resonance imaging (MRI) research is about to get another boost in Quee... http://bit.ly/bFw6fk

ISP: TPG announce unlimited Internet plans: http://bit.ly/damLGa

ACCC takes Optus to court (again) for misleading promos http://bit.ly/awhbMJ

Online hospitality tool nets venture capital: One of Australia's most prominent Web 2.0 investment groups, Future ... http://bit.ly/dt1hm0

Aussie software scores at Soccer World Cup: While Australia may not have finished well at the 2010 FIFA World Cup,... http://bit.ly/ctlKeu

@BigPondTeam - ok it's on http://bit.ly/b6Sl8G - they always tell employees last!

Looks like Windsor & Co will be keeping an eye on the roll out of the NBN http://bit.ly/c2kjKU

Keep up with ZDNet Australia

LinkedinLinkedin Join our network

RSS FeedsRSS Feeds Subscribe now

NewslettersNewsletters Subscribe today

TwitterTwitter Follow us

Compare Broadband Plans

Powered by WhistleOut

« 1 of 3»
1) Telstra BigPond47 plans 5%
2) Optus85 plans 1%
3) 39 plans 3%
4) Virgin Mobile7 plans 5%
5) iiNet31 plans 2%
1) Mobile Broadband Plans 10%
2) ADSL2+ Broadband Plans 12%
3) Cable Broadband Plans 7%
4) Naked DSL Broadband Plans 1%
5) ADSL Broadband Plans 2%

Mobile Phones | Broadband

CBS - ZDNET Australia Partner Services

Sponsored resources