Carlos A. Rojas

Lab / Educational

Learning objectives

By the end of this lab you will be able to take an existing backup, restore it to an isolated scratch target, and prove the restored data is complete and usable — not merely that the backup job reported success. The goal is to turn “we have backups” into “we have verified, restorable backups with a known recovery time.”

Required knowledge & prerequisites

• Comfort with a Linux shell and whatever tool produced your backups.
• A scratch target: a spare VM, a throwaway container, or an empty directory/database.
• An existing backup to test. The examples below use a filesystem archive and a PostgreSQL dump; adapt them to your own stack.

What you are restoring, and where

Pick a target that is isolated from production. The whole point is to exercise the restore path without risking live data, so a fresh VM or an empty database you can drop afterwards is ideal. Record the start time — recovery time is a deliverable, not an afterthought.

The drill, step by step

1. Verify the backup artifact exists and is intact before you trust it. A checksum recorded at backup time lets you detect silent corruption:

# Confirm the archive integrity against a stored checksum:
sha256sum -c backup-2024-01-28.tar.gz.sha256

# For a tar archive, list contents without extracting (quick sanity check):
tar -tzf backup-2024-01-28.tar.gz | head

2. Restore a filesystem archive to a scratch directory:

mkdir -p /srv/restore-test
tar -xzf backup-2024-01-28.tar.gz -C /srv/restore-test
# Compare against expectations (file count, key paths present):
find /srv/restore-test -type f | wc -l

3. Restore a database dump to a scratch database:

# PostgreSQL example — restore into a NEW, empty database:
createdb restore_test
psql restore_test < db-dump-2024-01-28.sql
# or, for a custom-format dump:
pg_restore -d restore_test db-dump-2024-01-28.dump

Expected observations

Illustrative, not captured from a live run. The checksum check prints OK. The archive extracts without errors. The database restore completes and the row counts of key tables match what you expect from production (or from a figure you recorded at backup time).

Why this matters

Backup software reports success when it finished writing bytes — not when those bytes are restorable. Partial backups, excluded paths, corrupted archives, un-dumped tables, and incompatible versions all pass the “job succeeded” test and fail the only test that counts: a real restore under pressure. Rehearsing the restore turns an unknown into a measured, documented capability.

Verifying the restore is actually good

• Completeness: compare file counts / table row counts against a known-good baseline.
• Correctness: spot-check a few records or files you can independently verify.
• Usability: actually start the application against the restored data, or run a query that the app depends on.
• Recovery time: record elapsed time from “disaster declared” to “service usable.” That number is your real RTO.

Integrity checking is a theme worth tooling: the same mindset behind the VCF Integrity Checker — don't assume a file is well-formed, verify it — applies directly to backup artifacts.

Troubleshooting

• Checksum mismatch: the artifact is corrupt or truncated. Do not proceed; pull a different copy. This is exactly the failure a restore drill is meant to catch.
• Restore fails on version mismatch: record which tool/version produced the backup; restores are not always forward/backward compatible. Keep the matching tooling available.
• Restored app won't start: missing config or secrets that live outside the backup scope. Note the gap — your backup definition needs to include them.

Cleanup & reset

# Remove the scratch filesystem restore:
rm -rf /srv/restore-test

# Drop the scratch database:
dropdb restore_test

If you used a dedicated VM, snapshot “clean” before the drill and roll back afterwards, or delete the VM entirely.

Limitations & further reading

One successful drill proves one backup is restorable today. Backups drift: scopes change, databases grow, and tooling updates. Schedule drills on a cadence, and treat a restore failure as a real finding. The honest deliverable is not “we have backups” but “we restored backup X on date Y in Z minutes and verified completeness.”