Disaster Recovery Test

On a number of occasions, I have witnessed IT personnel pull cheats during Disaster Recovery (DR) tests. The reasons vary between wanting to execute a flawless recovery, lack of commitment to the process or wanting to rush the test and be done with it due to an excessive workload.  Some of the most common cheats include copying data or partially staging or configuring IT components ahead of the test which pretty much renders the test meaningless.

Our value system puts a lot of emphasis on success, so it is difficult for some to come to grips with the fact that identifying a flaw in our ability to recover IT systems constitutes success which can lead to cheating in order to do well on the test.  But continuous improvement of DR procedures is a valuable measure of success and one of the main reasons we conduct DR tests.

When cheating is due to a desire to get the test over with as quickly as possible due to workload or conflicting priorities or the result of lack of IT staff commitment to the DR process, it is a different discussion. It may call for a self-examination of senior management’s commitment to DR or their employee’s perception of it. If DR is truly a priority for senior management, it must be clearly communicated to the entire organization.

However, what looks like a cheat is not always one. When a storage administrator stores configuration shortcuts or memory minders on a cell phone or thumb drives, it is not really a cheat and more likely meant to maintain a certain state of readiness. Nevertheless, it is a bad idea just the same because that information is not available to others should that person not be unavailable during a real event. The bottom line is that if something eases or speeds up recovery, it should not be a well-guarded secret, it should be part of the DR plan.

The primary objective of a DR test should always be to expose gaps and improve the recovery procedures, not to achieve flawless recovery. The reality is that you cannot plan for absolutely everything. A situation will always throw a wrench into your plans so the idea is not to achieve recovery perfection but to get better at it.