This guide is for the person who has been told to “do a DR test” and has to decide what that actually means: an IT manager, an infrastructure lead, or a backup engineer who owns the outcome but not the budget. It covers how to design one drill that produces evidence, not how to buy tooling.
A drill is an experiment with a written hypothesis
Most failed drills fail at the start, because nobody wrote down what they were trying to learn. “Test disaster recovery” is a mood, not an experiment. A drill needs a hypothesis in a form that can be wrong:
Using only the current runbook and the backups held in the secondary repository, a person who did not write the runbook can restore the orders database to an isolated host, with data no more than 24 hours old, and the order-lookup check passes, within four hours of the declaration.
Every clause in that sentence is a decision you must make before the day. Using only the current runbook means no tribal knowledge. A person who did not write it tests the document rather than the author. Isolated host keeps the exercise away from production. No more than 24 hours old is a recovery point objective being checked; within four hours is a recovery time objective being checked (see RTO and RPO for how to set those honestly). The order-lookup check passes defines “recovered” as something observable.
If you cannot write the hypothesis, you are not ready to schedule the drill. Write the hypothesis first and let it dictate everything else.
The four decisions
1. Scope: a service, not a server
Name the thing the business cares about (“customers can place and look up orders”), then list the systems that thing needs. A drill scoped to “the database server” tells you that a database server can be restored. It tells you nothing about whether orders can be placed afterwards, because the application, the identity provider, DNS, certificates and the message queue were all assumed.
For a first drill, keep the scope deliberately small: one service, at most six or eight systems. A drill you can finish teaches more than an ambitious one you abandon at lunchtime.
2. Scenario: what has been taken away
The scenario defines the loss, not the cause. Useful scenarios are specific about what is gone and what is still available:
- The primary storage array is gone; the hypervisors and network are fine.
- The whole site is unreachable; nothing stored only at that site can be used, including the runbook and any password vault hosted there.
- Everything reachable with production administrator credentials is untrusted; only an offline or immutable copy can be used.
Choose the scenario that tests your biggest unverified assumption. If you have never confirmed that backups survive a compromise of production credentials, that is the scenario to run, even if a storage failure feels more likely.
3. Exercise type: how much is real
| Type | What happens | What it finds | Typical risk |
|---|---|---|---|
| Tabletop | People talk through the scenario around a table or call | Missing owners, unclear authority, communication gaps, undocumented dependencies | None to systems |
| Functional | Real restores, into an isolated environment, following the runbook | Bad backups, wrong order, missing credentials, real durations | Low, if isolation is real |
| Full interruption | Production is actually failed over or taken down | Everything, including cutover and failback | High; needs a window and an abort plan |
The ladder matters. A full-interruption exercise on an organisation that has never done a functional restore is an outage waiting for a calendar date. Start with a tabletop exercise if you have no isolated environment, and move to functional restores as soon as you have somewhere safe to put them. IT KORR publishes a separate guide to disaster recovery testing methods, cadence and evidence that sets out five test levels in a similar order, if you want a second description of the same ladder.
4. Measurement: decide what you will write down
Before the drill, decide which timestamps and artefacts you will record. At minimum: the moment the scenario is declared (T0), the moment restore work starts, the moment each system is individually verified, and the moment the verifier signs off the whole service. Also record which backup was used, as an identifier and the time it represents. Without these, the drill produces a feeling rather than evidence. The after-action review article covers what to do with them.
A worked example
A 40-person firm runs an order system: a web front end, an application server, a PostgreSQL database, a file share for documents, and a directory service for logins. Backups run nightly to a repository in another location. Nobody has ever restored the database.
The drill lead writes the hypothesis from earlier and picks a functional exercise, because an isolated virtualisation host with spare capacity exists. The plan:
| Time (from T0) | Activity | Who |
|---|---|---|
| 00:00 | Lead reads the scenario aloud: primary storage is gone | Lead |
| 00:00 | Scribe starts the log; operator opens the runbook | Scribe, operator |
| 00:05 | Operator identifies the restore point to use and records its ID and timestamp | Operator |
| 00:10 | Restore of the database begins on the isolated host | Operator |
| until done | Operator narrates problems; scribe records each deviation | Operator, scribe |
| on completion | Verifier runs the order-lookup check and compares row counts with the baseline taken the day before | Verifier |
| end | Lead stops the clock and states the elapsed time | Lead |
The day before, the verifier records a baseline so there is something to compare against:
psql -h prod-db -U readonly -d orders -At -c "SELECT count(*), max(created_at) FROM orders;"That query only reads. After the restore, the same query runs against the restored copy. Do not expect the two counts to match: the restored copy holds data only up to the restore point, so it will contain fewer rows than production if the backup is older than the baseline, and at least as many if the backup was taken after the baseline was recorded. What the comparison gives you is a sanity check (is the order of magnitude right?) and, more usefully, the restored copy’s newest timestamp. The gap between that timestamp and the moment of the declared fault is your measured recovery point, which you can hold up against the RPO you claimed.
What actually happens in a first drill like this is rarely clean. A frequent finding is that the restore completes but the application cannot connect, because the database role passwords were never part of the backup and live in a secrets store nobody remembered. That is a fine outcome. It is the reason to run the drill at a quiet time, with the person who knows where the secret lives available but not allowed to volunteer it unless asked.
Dependency order and the circular trap
Recovery has an order. Something has to exist before the thing that depends on it can start. A workable default is: network and name resolution, identity, storage and platform, databases, applications, then the edge (load balancers, web front ends, anything customer-facing).
The trap is circularity. The password vault runs on a virtual machine that needs the directory to log in, the directory runs on the hypervisor, and the hypervisor management credentials are in the vault. Each dependency is reasonable; together they form a loop that no one can enter in a disaster. Drills expose these loops because a drill forces someone to actually start from nothing. The usual fix is a small set of break-glass credentials stored somewhere independent of the environment: a sealed printed copy, a separately hosted vault, or a hardware token held by named people.
Write the ordering down as a list with the reason for each position. “Directory before file share, because the file share authenticates against it” is a better line than “1. Directory”. The reasons are what let a stranger notice when the order is wrong.
Keeping the drill from becoming an incident
A drill that restores a copy of production onto the network can cause the very outage it is meant to rehearse. Several hazards are specific to restored copies:
- Duplicate identity. A restored server with the same hostname, IP address or machine identity as the live one can conflict with it. Restore into an isolated network with no route to production.
- Live integrations. A restored application may immediately send emails, call webhooks, process queued jobs or attempt payments, using the real credentials stored in its configuration. Disable outbound connectivity in the isolated environment or neutralise those settings before the application starts.
- Touching real backups. The drill reads from the backup repository. It must never delete from it, prune it, or run maintenance against it. Use read-only credentials for the restore where the tooling allows.
- Real data in the wrong place. A restored copy of production data is production data. Apply the same access controls, and delete the restored copy afterwards according to your own retention rules.
Write an abort condition before you start: for example, “if any drill activity produces a change on a production system, stop and treat it as an incident”. Authority to call that stop belongs to the lead alone.
How drills fail
- The author does the restore. The person who wrote the runbook fills in gaps from memory without noticing. Have someone else operate it.
- The clock is not real. If time is only recorded at the end, it is reconstructed from memory. Start the log at T0 and have the scribe note times as things happen.
- “Recovered” is declared at the first green light. The VM booted, so the drill is declared a success, though the application was never exercised. Define the verification before starting.
- Findings with no owner. The review produces a list, nobody is named, and the next drill rediscovers the same problems.
- Only the happy path. Restoring the latest backup from an intact repository is the easy case. At some point test an older restore point, or a restore with one component unavailable.
What to do next
If you do not yet know which exercise suits you, answer the questionnaire in the drill planner; it produces a scope, scenario, roles, success criteria and evidence list you can edit. It is a rule-based worksheet and does not test anything about your systems. If you are about to run the first restore, read restore testing for the verification ladder, and print the one-page drill checklist for the day itself.