Parks Victoria puts 6TB backup on-demand

Realising it could take three months to restore critical servers after a disaster prompted Parks Victoria to become one of the first large organisations in Australia to adopt an on-demand model for its backup and disaster recovery.

Parks Victoria administers almost 20 percent of Victoria's land, covering 200 parks, marine sanctuaries, over 2,500 reserves and 10,000 cultural and indigenous heritage sites. It has 1,000 employees and 100 sites across the state -- yet its systems are managed by just four people, headed up by its infrastructure manager, Gavin Russell.

The agency is responsible for accommodation bookings for each of the parks it oversees and the system that supports it needs to be operational 24 hours a day, seven days a week, said Russell. Equally important are its HR and payroll system and financial systems.

The catalyst for the agency's decision to outsource its DR and backup came when Russell evaluated the agency's disaster recovery capabilities. His network administrators were too busy managing servers, desktops and laptops across the agency's 110 sites, leaving them no time to handle a disaster if one occurred. Its source files were locked up on tape backups and with few skilled staff, he knew it would be difficult to recover from disaster.

Parks Victoria's management had no idea know how long it would take to recovery from a disaster: the DR plan was literally "a folder full of paper on how we would rebuild our servers", said Russell.

"So I sat down and figured out how long that would take and found it would take between three weeks and three months to do, depending on the availability of hardware," he said.

"I said to the business this was unacceptable and that we needed a proper DR plan."

Satisfied that Russell's evaluation warranted a better DR capability, Parks Victoria issued an initial tender for an outsourced DR service, however the responses it received were initially not encouraging.

"Eight responses came back from the tender but some were excluded on the basis of location. We needed them to be outside Melbourne's CBD, and that comes down to insurance -- if the service is just round the corner it shares the same risk profile we do," he said.

Most of the respondents to the tender came back with "immature solutions".

"They hadn't quite thought it through. For example, one had a decent solution for DR, but they hadn't thought how you would rebuild a site and how to get anything back to the production site. In the end, they came back and said they would copy virtual machines over a link, but that could take two weeks to get the production site back," he said.

Parks Victoria eventually selected Global Storage, a relatively young company that uses AAPT's datacentre in Richmond, Victoria and Global Switch, in Ultimo, Sydney to host and protect its customers' data. But even Global Storage was a risk, said Russell, since it lacked customers of a comparable scale to Parks Victoria.

"[Deciding which vendor] really comes down to a customer reference but because we were the first to go on this system, I couldn't get one," he said.

Nonetheless, Russell was impressed that the company had thought through the recovery process. Although the initial tender was solely for a DR service, Parks Victoria opted to include backup in the agreement -- a decision that would move its critical servers away from the CommVault tape solution it relied on.

To date, outsourced backup has remained a pie in the sky for most large organisations, however de-duplication technologies are changing this. The ability for Global Storage to offer an enterprise scale backup service has been made possible by its adoption of EMC's de-duplication technology, Avamar.

Both Quantum and EMC last year acquired companies that gave them de-duplication technology, and have been touting the benefits of it for distributed offices in order to minimise network traffic and reduce backup windows. However besides Global Storage's use of the technology, it remains largely untested.

The two services Global Storage proposed for Parks Victoria were Data Ready -- the remote backup service -- and Environment Ready -- a replication service, which supports disaster recovery and business continuity objectives.

Both services are charged on a monthly basis. The backup service is charged according to the number of servers required to be backed up, while the recovery service is charged according to the required RPO or "recovery point objective", which can be anything from one hour to 24 hours.

Russell declined to comment on how much the service costs, however Global Storage's CEO, David Duncan, said the backup service (Data Ready) costs between AU$3 to AU$5 per gigabyte, plus a local-storage gateway charge of AU$300 per month. Environment Ready is charged per server, which typically costs AU$100 per server per month, depending on the RPO.

Parks Victoria implemented the service just two months ago so, according to Russell, it has not yet had the chance to test the system's capabilities in a real disaster.

The implementation has meant that Parks Victoria could easily prioritise its servers according to the desired RPO level for each aspect of its business. Russell identified 25 out of 110 servers that he wanted on Environment Ready, including Parks Victoria's Oracle e-business financial and HR applications, its SQL database used for Park Victoria's point of sale and accommodation bookings, as well as its Citrix, Exchange, file and Internet servers.

"We've chosen a six-hour RPO for six of our servers and 24 hours for the balance," he said.

The end result for Parks Victoria, according to Russell, has been great. The agency has backed up six terabytes of data with Global Storage and has halved its backup windows thanks to the new service.

"Our backup window on Friday night used to be 24 hours. That's down to 12 hours. We used to do file backups once per week, and then incremental backups the rest of week. With Avamar you're also not doing whole file changes," he said.

But perhaps the most significant benefit from Russell's and the network administrators' perspective is the ability to offload risk and effort to Global Storage.

"To us, it's been hands off and a complete transference of risk. Off-site storage is also fantastic and there's peace of mind -- should we have a disaster at 2am I can call someone else," he said.