It's an axiom of e-business: If you're running a Web site, you can't afford downtime, particularly unexpected downtime that knocks customers off line and threatens their transactions and data.

With demanding customers expecting 24-by-7 uptime from Web sites and the competition only a click away, every second offline means potential customers, and dollars, are lost.

But, realistically, just how much downtime are you willing to tolerate when disaster strikes, and how much are you willing to pay for the disaster recovery and business continuity services that can help you stay in business or recover quickly no matter what? As the ongoing power crisis causes lights to flicker all over California, e-businesses everywhere have begun to ask themselves those questions and to put updated disaster recovery plans in place. Increasingly, e-businesses are turning to disaster recovery providers for backup and co-location, which allow them to recover from a blackout or other breakdown in minutes rather than hours. For that, however, e-business managers say they're paying up to 25 percent of their overall IT budgets.

Why are they willing to pay so much? At PrimeLearning, Chief Technology Officer Derek Humphreys said it would be extremely difficult to sustain business if customer confidence were lost due to disaster-caused downtime or data loss. That's why he has put in place a disaster recovery plan that takes into account any type of incident.

PrimeLearning provides online training to corporations. A loss of data, including test scores, would be almost impossible to recover from, Humphreys said. "If there was a disaster and we didn't have an appropriate plan in place, it is something that would greatly affect our customers and shatter all confidence in our ability to deliver and maintain their data," he said. "That's not just a dent on the resume. We would go out of business."

To ensure that PrimeLearning can continue operations in the event of a disaster, Humphreys chose to co-locate his Web servers at a New Jersey facility run by Comdisco. Live-production boxes and staging boxes are all located at Comdisco's facilities in New Jersey. To further ensure availability, Comdisco mirrors all of PrimeLearning's data at its facility in Wooddale, Ill. The setup will enable Humphreys to bring his site back up in less than 8 hours.

E-business managers have a wide variety of disaster recovery and business continuity services from which to choose. The high-availability services of the type Humphreys currently uses allow for recovery in hours, and businesses can also select services that mirror transactions as well as data and promise zero unplanned downtime.

While the lag is acceptable for his business model right now, Humphreys said that as PrimeLearning expands internationally in the future, it will not be acceptable.

"Murphy's law pretty much dictates that when it comes to e-business, there is no safe window for a disaster to occur," Humphreys said. "So when it comes to a disaster recovery plan, the obvious return on investment is my ability to sleep at night."

Not that a good night's rest comes inexpensively. Humphreys estimates that he spends about a quarter of his IT budget on disaster recovery. Still, to many IT managers, it's a small price to pay for staying in business and preserving customer confidence.

That's pretty much the way officials at Damian Services see it, too. On two occasions in the past six months, after being hit by power outages on critical business days, the provider of online payroll and accounting services to independently owned staffing firms has asked its hosting provider, SunGard Data Systems to be on emergency alert. Damian hosts its servers at SunGard's data centre. Damian also uses software from Veritas Software to mirror its production environment. The total cost: about 15 percent to 20 percent of the overall IT budget, said Robert Kulla, CIO at Damian.

Damian's deal with SunGard is only part of the company's overall disaster recovery plan. Kulla also backs everything up himself on a Quantum digital linear tape system. And Tom Strenk, chief of staff at Damian, said his strategy encompasses everything from how to recover data to how to restore power in the company's offices using a backup generator.

It means a lot of effort and money, Strenk said. But it's what every e-business must do to turn continuous availability from an axiom to a reality.

"It is necessary to have business resumption capabilities," Strenk said. "Disaster recovery is a costly endeavor, but there's a great sense of security we get from knowing we've taken every conceivable precaution to ensure the integrity of our data."

Outsourcing degrees of disaster recovery

Disaster recovery services are more than just insurance policies companies begrudgingly buy for their own peace of mind; eWEEK Labs believes these services should be seen as the cornerstone of business preservation.

With the evolution of the Internet and online businesses, downtime has become a potentially costly hazard that can quickly drive customers to competitors (just ask MSN and Amazon.com). Data loss in high-transaction environments has become inexcusable.

Recovery service providers offer the expertise required to toughen up mission-critical systems at companies of all sizes. With nationwide networks and arsenals of standby servers and mainframes at the ready, these companies can quickly revive businesses when disaster strikes.

Until recently, at most companies, disaster recovery began and ended with daily tape backups, a tried-and-true process that has been around since the dawn of business computing. However, in the event of a hardware failure, incremental data that hasn't been backed up is lost.

Worse still, in the event of a large-scale disaster such as a severe earthquake or hurricane, companies that don't have off-site tape storage centers and replacement servers could suffer complete data loss and the paralysis of key business services.

Types of services
Depending on a customer's acceptable thresholds for downtime and data loss, a wide variety of services are now available, ranging from basic system recovery to hosting services with guaranteed uptime.

System recovery is the most basic service that can be purchased. Costs vary widely depending on the amount of data being recovered and on the service-level agreementâ€"anywhere from several thousand dollars to more than US$1 million.

Most system recovery services are outsourced tape-backup and restoration arrangements in which the provider monitors all backup activities and stores redundant backup copies at geographically dispersed data centers. This service is often a good fit for small companies that don't have geographically dispersed offices.

System recovery services usually complete restoration within 48 hours (depending on the service-level agreement and the amount of data restored) but also have the largest potential for data loss.

High-availability services can get companies up and running in anywhere from a few minutes to a couple of hours (depending on the level of service purchased) and often guarantee zero data loss. These options can easily cost $1 million: Customers often have data that is worth substantially more than the machinery on which it resides.

An excellent example of a high-availability implementation would be the Philadelphia Stock Exchange, which relies on SunGard Data Systems to protect its stock exchange servers.

The core of this system is EMC Symmetrix Remote Data Facility, a data mirroring system that duplicates production data to physically separate target systems hosted at SunGard's Philadelphia data centre.

To transfer data from the Philadelphia Stock Exchange to SunGard's recovery site, SunGard set up a Synchronous Optical Network ring between the two sites. In the event of a system failure, a mainframe at SunGard's site is activated and the stock exchange service is re-established with zero data loss.

The third, most rock-solid option is continuity service, in which the production servers of a business are stored at the client site and at a recovery center. This arrangement is even more expensive than those mentioned above, because it often includes leasing servers and storage capacity.

In this scenario, processing is done at both locations. Load balancers dele gate tasks equally to both sites.

In the event of a network failure or a disaster at the customer site, the application will continue to run at the parallel recovery center.

Senior Analyst Henry Baltazar is at henry_baltazar@ziffdavis.com.

Adding 9s to uptime can add up

The technology used for disaster recovery and business continuity service plans differs dramatically depending on the percentage of uptime that must be guaranteed.

A two-9s guarantee (99 percent uptime), for example, would allow for a couple of days of downtime per year. Although this would be acceptable for low-priority corporate applications, few transaction-heavy applications can withstand this much downtime. To guarantee 99 percent uptime, an IT manager could get by using simple tape backup without having the added expense of warm standby servers.

Boosting uptime to the three-9s level (99.9 percent uptime) would reduce acceptable downtime to a couple of hours a year. Disaster recovery providers are typically vendor-neutral, which should eliminate worries about losing the initial investments in backup systems.

But depending on the amount of data being protected, a tape-based recovery system along with standby servers might not be enough. For three-9s systems, more likely than not, off-site redundant servers will be required; these servers should run data mirroring software to ensure that, in the event of a disaster, the backup servers will be able to restore services.

Moving up to the five-9s range and beyond, distributed processing with load balancing becomes almost mandatory to meet uptime requirements. In this case, it would be common to see high-speed network connections running through powerful routers from companies such as Juniper Networks.

Before deciding on a disaster recovery and business continuity plan, make sure you know how much uptime your business really needs, because adding 9s can be an expensive proposition.

Copyright © 2009 CBS Interactive, a CBS Company. All Rights Reserved.
ZDNET is a registered service mark of CBS Interactive. ZDNET Logo is a service mark of CBS Interactive.