exclusive EnergyAustralia is scrambling to restore business critical data "lost" as a result of back-up failures that went undetected for several months.
The fault was only discovered last month when a problem with a server caused data loss at a number of EnergyAustralia offices, including its Sydney headquarters.
In a recent internal e-mail to employees, Don Anderson, EnergyAustralia executive general manager for shared services, said: "The back-ups were affected by an underlying, undetected software problem that had been damaging the back-ups for a number of months."
An analysis has shown that "one quarter of the total files (about 1.2 million files) that were on the back-up tape of December 12 2006" were still missing, Anderson wrote in the e-mail seen by ZDNet Australia.
The energy utility and its contractor Fujitsu spent the Christmas/New Year period attempting to recover lost files from a number of shared drives. However, their efforts have been hampered by damaged back-up tapes. To minimise the impact on staff, EnergyAustralia is in the process of a second "major restoration attempt" -- which it expects to finish this month, according to Anderson.
New hardware has been purchased to aid the second restoration attempt. The additional hardware will help restore files without overwriting those already recovered, he added.
EnergyAustralia is more optimistic about this exercise, however, Anderson did not guarantee staff that all missing files would be restored.
"We expect this second major restoration will deliver a more comprehensive restoration of the back-up tapes of 12 December, but it is not guaranteed. Our aim is to restore the missing files.
"If the restoration is successful users should be aware that the recovered versions of some files may be weeks or even months old. Beyond this restoration there is little chance that we will be able to retrieve any additional files by other means," Anderson wrote.
What's on tape?
In his e-mail, Anderson said it was attempting to recover business critical files and directories. "There is a range of documents that may need to be recreated, including some regulatory documents," an EnergyAustralia spokesperson told ZDNet Australia.
As a New South Wales state-owned corporation, the utility company has to submit regulatory documents on its activities. It is ZDNet Australia's understanding that its submission is due early this year.
The EnergyAustralia spokesperson, however, said the back-up failure isn't expected to affect its provision of documents by deadline.
You might be interested in:
However, EnergyAustralia is investigating the matter and has committed to an independent audit of its IT systems, the spokesperson said.
When asked if it is considering legal action against Fujitsu, the spokesperson said: "EnergyAustralia is still investigating this matter so that is not something we are looking at, at this stage".
A Fujitsu spokesperson said it would co-operate fully with EnergyAustralia in any investigation into the matter. "As one of a number of suppliers to EnergyAustralia and in the spirit of our partnership, Fujitsu is working closely with EnergyAustralia to restore data lost as a result of a SAN incident on Friday, 22 December," she added.










Backup test
Now you would assume that the backups were automatically checked at the very least.
Isn't it a simple matter of ticking the box labelled "verify backup" ??
Verify Backup = fool's paradise
"Damaged tapes" could be spin. The verify process only checks that the data written is correctly on the media.
Most backup issues occur because the backup set is incomplete, either because major parts of the file system are not included in the backup set, or because incremental backups are being done and they are mishandled or the backup software has a bug.
None of these scenarios are picked up by "verify backup", which is a fool's paradise... IMHO