No disaster recovery plan can ever be like clockwork, because human response is always a variable. And the way to improve the human factor is through accountability.
Last week, I was having dinner in a popular chain restaurant and noticed that our waiter was particularly agitated.
When we were ordering, he was frantically writing down everything we said twice, and we had to repeat ourselves. I watched as he passed by the terminal near the kitchen -- where orders were normally entered into the house computer system -- and he handed a slip to someone in the kitchen, and then frantically scribbled a third copy of our order.
Later, there were harsh words in the kitchen that we could hear in the dining room. Intrigued, I later asked the waiter what was going on. The computer system had crashed, it seemed, and management had given the table servers contradictory instructions about the order slips they needed to fill out.
Making matters worse, handfuls of specially numbered slips had been given to each table server, and only after an hour and a half of activity were they told that these numbered originals had to be accounted for -- the number itself was not sufficient, for some reason -- and they'd have their pay docked if they didn't account for them. They had to root through garbage to save their paychecks.
Service was slow; the waiters and waitresses undeservedly suffered for this (we diners tend to take it out on them, even if the slowness is the fault of management or kitchen staff), and by the time the smoke had cleared, four of them had either been fired or quit that night.
This example will certainly seem simplistic when set beside your own disaster recovery tales, but its simplicity speaks to the most often-neglected factor in disaster recovery planning: accountability for human failure in the chain.
The price of passing the buck
In the era of Enron, we no longer accept that a certain amount of politics and managerial malfeasance is bundled into the cost of doing business. It should be even less so in the area of disaster recovery, which is not simply a matter of moving assets from here to there, but can be survival-critical.
Put simply, your disaster recovery system must be scapegoat-proof . When an employee at any level, from the lowliest assistant all the way up to the CIO, obscures his own failure to act as part of a disaster recovery plan or (worse yet) redirects blame to an individual who acted properly, it does more than obscure incompetence and damage morale. Your company's functionality and the integrity of its most immediate asset are placed at risk, and for the most disingenuous of reasons.
Your disaster recovery plan, then, must be as impervious as possible to politics, scapegoating, and buck-passing. If a human failure occurs in the process, and it isn't caught because an individual was able to hide the failure, then your recovery plan is insufficient. You must have objective, reliable means of enforcing accountability among the human participants in the system, or your disaster recovery plan could be inadequate.
Account for accountability
Most disaster recovery plans assume that all participants in the process share a common desire to valiantly see the crisis through with fierce determination. Such plans focus on technical preparations and rapidly-executed procedures intended to restore operations. It never occurs to the authors of such plans that there may be people in the loop who are less than dedicated to the cause, or who will be entirely unscrupulous if they screw up their part. Looking back at the restaurant incident, it's a fair bet that whoever set up their recovery procedure didn't give much thought to the vulnerability of the food servers and the discretionary powers of floor managers.
10%
8%
