This article is for the engineer who will be asked to write the recovery runbook, and the manager who will be asked whether one exists. It is about runbooks written to be followed under stress by someone who did not write them, which is the only kind that earns the name.
The reader you are writing for
Picture the person who will use the document. It is early morning. They were woken by a phone call. The wiki is down, because it was on the failed storage. They are competent but they did not build this system, and the colleague who did is unreachable. They have to make several decisions quickly and one wrong command could make things worse.
That reader needs three things from a runbook: to know whether to start, to know exactly what to do next at every moment, and to know how to tell that it worked. Everything else is optional. A runbook that begins with two pages on architecture history has already failed them.
Where the runbook lives
Before content, location. A recovery runbook stored only on the system it describes will be unavailable exactly when it is needed. Keep at least one current copy in a place that survives your scenario: a printed copy in a known location, a copy in a separately hosted document store, a copy on a laptop of each responder that is refreshed on a schedule. State where the copies are at the top of the document and make someone responsible for keeping them in step.
The same logic applies to what the runbook depends on: the credentials, licence keys, encryption keys and contact numbers needed to execute it. If the only copy of the backup encryption key is in the password vault, and the vault is on the failed platform, the runbook cannot be completed. Record in the runbook how each secret is obtained, without writing the secret in the runbook.
Anatomy of a recovery runbook
A workable structure, in order:
- Header. Service name, owner, last review date, last time the runbook was followed in an exercise and by whom, version.
- Purpose and trigger. One paragraph on what this runbook recovers and the conditions under which to start. Include the decision point: who may declare a disaster and how.
- Targets. The RTO and RPO for this service (see RTO and RPO), so the operator knows what pace is expected.
- Prerequisites. Access needed, tools needed, where the backup is, who to call if it is not there. Include the break-glass path if normal authentication may be down.
- Ordered procedure. The numbered steps. This is the body of the runbook.
- Verification. How to confirm the service really works, written as observable checks.
- Abort and rollback. What to do if a step fails, and the point after which you cannot go back.
- Communication. Who is told what and when, with contact paths that do not depend on the failed systems.
- Known issues and lessons. Things that have gone wrong before.
- Change history. What changed and why.
Writing the steps
Each step should be small enough that the operator can tell whether it succeeded, and should say how. Compare two versions of the same step.
Weak:
Restore the database and bring up the application.
Usable:
Step 4. Restore the orders database (expected: 20-40 min for the latest full backup)
a. On the restore host, confirm the target database does NOT exist:
psql -h restore-db -U postgres -lqt | cut -d '|' -f 1 | grep -w orders
Expected: no output. If "orders" appears, STOP and escalate:
you are on the wrong host.
b. Create the database and restore (path from Step 2):
createdb -h restore-db -U postgres orders
pg_restore -h restore-db -U postgres --no-owner -d orders /restore/orders.dump
c. Check the newest row:
psql -h restore-db -U postgres -d orders -At -c "SELECT max(created_at) FROM orders;"
Expected: a timestamp within the RPO of the incident time recorded in Step 1.
If older than that, go to Step 4-ALT (older restore point is the best available).The better version states the expected duration (so the operator knows when something is hanging), includes a guard against the single most dangerous mistake (restoring onto the wrong host), tells them what a good result looks like, and gives them a branch for the bad result. Note that every command in it is either read-only or acts on an empty target by design. When a step is destructive, say so in plain words on the step and require a check before it.
Some conventions that pay off:
- One action per step. “Restore and then start and then verify” in one line hides where it failed.
- Exact commands, with placeholders marked. Use a visible convention such as
<RESTORE_HOST>, and list the values in a table in the prerequisites. Do not leave the operator to guess which parts to edit. - Expected output. What should the operator see? A runbook with no expected output cannot be followed by someone unfamiliar with the system.
- Time boxes. If a step normally takes thirty minutes, say at what point to treat it as stuck and what to try.
- No hidden knowledge. “Use the usual service account” is a dependency on a person. Name the account and say how to get its credentials.
Ordering and dependencies
The procedure must start with whatever the rest depends on. If recovery has several services, give a separate runbook for each and a short master document that sets the order and says when each can start, in parallel where possible. For the example order system, the master sequence might be: network and name resolution, directory service, database platform, orders database, application servers, then the web front end and DNS changes.
State the reason for each dependency. “Directory before application, because the application authenticates users against it and will not start without it” lets a responder reason about a situation the document did not foresee, which, in a real incident, will happen.
Look for circular dependencies while writing, because writing exposes them. If step 3 needs a credential that is stored in a system restored in step 7, the document cannot be followed. These loops are common and are the most valuable thing a first drill finds. The typical resolution is a small, independent break-glass mechanism, held outside the environment and tested.
Testing the runbook: the stranger test
A runbook has been tested only when someone who did not write it has followed it, on a representative system, using only what is in the document. During that test:
- The author watches and says nothing unless asked for something the runbook should have contained, in which case they write down that gap rather than answering.
- The operator reads each step aloud before doing it and narrates any hesitation.
- Every place the operator guesses, backtracks or asks a question is a defect in the document. Count them.
The count is a decent measure of progress. A first test often finds a dozen. A runbook that has been through two or three rounds usually finds a few, and those tend to be about things that changed since (a renamed host, a rotated credential, an upgraded tool whose output changed).
Document edits resulting from the test must be made immediately, while the pain is fresh, and the updated document should be the one used for the next test. Designing a recovery drill describes how to fold this into a larger exercise, and the after-action review article covers how to track the fixes.
Keeping it alive
A runbook decays as the system changes. Tie its review to events that change what it says:
- Any change to the service’s architecture, hosting or backup mechanism triggers a review before the change is closed.
- Staff changes: when a person named in the runbook leaves or changes role, update it.
- A calendar review at least annually, even without a trigger, plus a fresh exercise at an interval you can sustain.
Show the “last followed in an exercise” date prominently. A runbook that was last followed eighteen months ago on a previous version of the system should be treated as unverified and labelled so.
Common mistakes
- Writing the runbook as documentation of the system. Architecture is useful reference, but belongs in a separate document. The runbook is a procedure.
- Screenshots as the only instruction. They go stale with every interface change and cannot be searched or pasted. Prefer text and commands, with screenshots as a supplement.
- Assuming a working network, DNS or login. Test the runbook with these absent.
- Credentials in the runbook. Do not write secrets into the document; point to where an authorised person retrieves them.
- A single point of authority. If only one named person may declare the disaster, name a deputy.
- Treating “reviewed” as “tested”. A review by the author’s colleague at a desk finds typos. Only following the procedure finds gaps.
What to do next
Choose the service you would least like to recover from memory and write only its first three steps in the format above, then have someone else try them on a scratch system. If you are not sure where to start, the drill planner suggests which system to restore first from the names you give it; it is a worksheet, not a test of anything. For the underlying verification steps your runbook should call, read restore testing.