Mega2580.Solutions
November 2018

The restore test nobody runs

By A. Tanaka · 5 min read

Every infrastructure review we've done this year has turned up backups that were configured correctly, ran on schedule, and had never once been restored. That's not a knock on the teams — restoring a production database onto a scratch environment is exactly the kind of task that's easy to defer indefinitely.

The problem is that backup systems fail in ways that don't show up as a failed job. Permissions drift, a schema change breaks the restore path, a credential rotates and nobody updates the backup service. All of these look identical to a healthy backup right up until the day you need it.

Our baseline recommendation: an automated monthly restore into an isolated environment, with a script that checks row counts against production and pages someone if they're off by more than a small tolerance. It's a few hours of setup that turns a hypothesis into an actual, tested guarantee.