This article is for backup engineers and infrastructure leads who get green “job succeeded” emails every morning and want to know what, exactly, those emails prove. The short answer is: less than you hope. Restore testing is how you replace hope with a measurement.
What a successful backup job actually tells you
A backup job reporting success means the software believes it read the source, wrote something to the destination and did not hit an error it was designed to notice. That is a useful signal, but it is silent on every question that matters during a recovery:
- Can the data be read back from the destination, today, by someone with the credentials you will have during an incident?
- Is it complete, or did the job skip locked files, a mounted volume, or a database that was mid-transaction?
- Is it consistent, so that an application started from it will run, rather than a set of files captured at slightly different moments?
- Will it restore fast enough, onto hardware you actually have?
Restore testing answers these by doing the thing the backup exists for. Saying “we have backups” about systems that have never been restored is a common false assurance.
A ladder of assurance
Not every check is equal, and you do not need the top rung for every system every week. Think of it as a ladder where each rung proves something the one below cannot.
| Rung | Check | What it proves | What it still cannot prove |
|---|---|---|---|
| 1 | Job reports success | The tool saw no error it recognised | That any data is readable |
| 2 | Catalogue or index lists the expected backup, with plausible size and date | The backup exists and is the right order of magnitude | That the content is intact |
| 3 | Integrity verification (checksums, or the tool’s own check) | Stored data matches what was written | That the right things were captured, or that it is usable |
| 4 | Restore of sample files or objects, compared with the source | Selected data is readable and correct | That a whole system comes back |
| 5 | Restore of a whole system or database into isolation, started and queried | The system is recoverable and consistent | That the service works, or how long it takes end to end |
| 6 | Timed restore of a multi-system service in dependency order, verified by a business check | The service is recoverable and by how long it takes | Nothing about scenarios you did not test |
Many organisations sit at rungs 1 and 2 while believing they are higher. A sensible target is rung 3 continuously and automatically, rung 4 monthly, rung 5 on a rotation across your important systems, and rung 6 at least annually. Those frequencies are starting suggestions, not a standard; adjust to how quickly your systems change and how much downtime the business can bear.
Rung 3 in practice: integrity checks
Many tools have a built-in verification. With restic, for example, the repository structure can be checked, and the stored data can be read back and verified:
restic check
restic check --read-data-subset=5%The first command checks the repository’s structure. The second reads and verifies a random 5% of the stored data, which keeps runtime bounded while rotating coverage over successive runs. Reading all of the data (--read-data) is the strongest form and can be slow and, on cloud storage, can incur retrieval charges, so check your provider’s pricing before scheduling it. Borg has a comparable borg check command. If your product has no verification feature, record SHA-256 checksums when files are written and compare them later:
sha256sum /backups/orders-2026-10-01.dump > /backups/orders-2026-10-01.dump.sha256
sha256sum -c /backups/orders-2026-10-01.dump.sha256On Windows the equivalent is Get-FileHash -Algorithm SHA256 <path> in PowerShell. A checksum proves that the bytes have not changed since you computed it. It does not prove the bytes were a good backup to begin with: a checksum of a corrupt dump is perfectly stable.
Rung 5 in practice: a database restore into isolation
The point of this rung is to start the thing that was restored and ask it a question. For PostgreSQL, a custom-format dump can be restored into a scratch database on an isolated host:
createdb drill_restore
pg_restore --no-owner --dbname=drill_restore /restore/orders-2026-10-01.dump
psql -d drill_restore -At -c "SELECT count(*), max(created_at) FROM orders;"Run those only on the isolated drill host. Naming the target database something obviously temporary matters, because pg_restore with the --clean option drops objects before recreating them, and pointing it at the wrong server is the classic way a test becomes an incident. If you add --clean, double-check the connection target first.
Then check things the restore itself does not check: do foreign-key relationships hold, does the newest row have a timestamp near what you expect for the restore point, and can the application, started against the copy with its outbound integrations disabled, answer one real request? A database that restores with zero errors can still be missing last night’s data if the dump ran against a replica that was behind.
For virtual machine backups, “restored” means the machine boots, the operating system reaches a login, the application starts, and the application health check passes. Booting is rung 5 minus the part that matters. Start the service and exercise it.
Measuring time, honestly
Restore duration is the number that RTO claims depend on, and it is nearly always longer than the estimate. Measure three things separately, because they scale differently:
- Transfer time. How long the data takes to get from backup storage to the restore target. As rough arithmetic, 2 TB (2,000,000 MB) at an effective 200 MB/s is 10,000 seconds, about 2 hours 47 minutes. Link speed is not effective speed: a 1 Gbit/s link carries about 125 MB/s at the theoretical maximum, and real transfers are lower once the protocol, the storage and the backup software have their say. Measure the real figure on a representative restore.
- Rebuild time. Provisioning the target, installing or booting the platform, replaying transaction logs, rebuilding indexes, and starting services. For databases, log replay and index rebuilds can dominate.
- Verification time. Running the checks that decide the service is usable. People forget to budget for this and then discover that it takes an hour.
Record start and end times for each. If your restore scales with data volume, extrapolate carefully: a restore tested at 10% of production size does not take 10% of the time for sure, because some phases (a single-threaded index build, a rate-limited cloud retrieval) do not scale linearly.
Choosing what to test
You cannot restore everything every month, so choose deliberately.
- Rotate. Make a list of in-scope systems and work through it, so every system is restored on a known cycle. Write down which system is next.
- Weight toward the risky. Systems whose backups use a different mechanism, systems that were recently migrated, and systems with databases deserve earlier and more frequent restores than a static file share.
- Vary the restore point. The most recent backup is the easy case. Occasionally restore one from several weeks ago, because that is what a deletion discovered late requires. Also try restoring from the secondary copy rather than the primary repository, since that is the copy you will need when the primary is gone.
- Vary the person. If only one engineer has ever performed a restore, the capability belongs to a person, not the organisation.
Common mistakes
- Restoring in place. A test restore written over live data is a data-loss event. Always restore to a separate location, and double-check the destination path or hostname before pressing enter.
- Testing only files. File-level restores are valuable, but they say nothing about whether the database or the machine image is usable.
- Using privileges you won’t have. The test succeeds because the engineer is logged in as a domain admin. In an incident, the directory may not exist yet. Test with the access path you would actually have.
- Not testing the decryption. Encrypted backups are useless if the key lives only in the system you are recovering. Confirm where the key is held and that you can retrieve it without the thing you are restoring.
- Declaring victory on a partial result. “Restored with warnings” is a finding, not a pass. Investigate each warning until you can explain it.
- No record. If the test left no evidence, it will be repeated from scratch next time and cannot support an RTO claim.
What to do next
Turn the ladder into a schedule: pick the next system in your rotation and decide the rung you will reach. The drill planner can help you decide scope and a first system based on your answers; it is a worksheet and does not run or verify any restore. For the checks that sit below the restore itself (verification, copies, immutability), read backup validation, and for how to fold restore results into a larger exercise, see designing a recovery drill.