Mega2580.Solutions
January 2024

Terraform state files: the mistakes we've made

By M. Kowalski · 5 min read

We don't usually write about our own mistakes this directly, but state file incidents are common enough across the industry that it seemed worth being specific. Three from our own history.

Two engineers ran an apply within minutes of each other against unlocked state, and the second run silently reverted the first's changes. We now run state locking as a hard requirement, checked in CI, not just a suggestion in the README. A state file got corrupted during a migration between backends, with no recent backup; we now snapshot state before every structural change, automatically. And a renamed resource got recreated instead of moved, taking down a production database for four minutes; we now require a plan review specifically for any destroy-and-recreate before apply.

None of these were exotic failures. They were all things a stricter process would have caught, which is most of what changed — not the tooling, the discipline around it.