Every piece of automation I ship comes with a document nobody asked for: how to do the same thing by hand when the automation fails.
Not if. When.
People tease me about this. Why write a manual runbook for a deploy script that works? Why document the by-hand steps for a migration that has run cleanly fifty times? I used to answer in theory. Now I answer with three stories, all mine, all real.
The backup that saved nothing
This one still stings. During a repo cleanup, the workflow was textbook: stash the file first as a safety net, then hard-reset the working tree. Backup, then destructive step. The correct order. The professional order.
Except the stash silently created no entry. The output that would have said so was piped away, nobody read it, and the chain moved on because each command exited without complaint. The hard reset ran. Uncommitted work was gone, unrecoverable, no reflog entry, nothing. The safety step had executed and safeguarded nothing.
The lesson was not “be more careful”. Careful was present that day. The lesson was mechanical: never gate a destructive step on the backup command running. Gate it on the backup artifact existing. Check the stash list has a new entry. Check the tar file is non-empty. Touch the thing that is supposed to save you before you need saving. A backup you have not verified is a mood, not a backup.
The installer that verified the wrong thing
One of my macOS apps ships with an installer script. An early version did what most installers do: copied the files, checked the copy succeeded, printed success. Solid green output. Meanwhile, two lines further down its own log, the process it had just installed was dead. Never started. The “success” left behind a broken app that had to be removed and redownloaded.
The script had verified its step instead of the outcome. Files copied is a step. App running is the outcome. Nobody installs software to have files copied.
The rewrite checks what the user actually cares about: is the process alive, is it responding, and it re-checks eight seconds later, because a process that starts and immediately dies passes the first look. When the outcome check fails, the installer rolls back to the previous version on its own. The failure path got designed first, and the success path was built inside it.
The failure that nobody noticed
Just this week, a deploy of this very website failed. Mid-build, a font fetch died inside Docker for no reason that was anyone’s fault. Transient network, bad minute, the usual entropy.
Here is what visitors experienced: nothing. The deploy script is written to verify the new container actually starts and answers before it replaces anything, and when the build failed, it simply kept the old version serving. I retried later, the build passed, done. Total production impact: zero. That boring outcome was purchased years earlier, on the day the script was written to assume its own failure.
The pattern under all three
Automation fails on the abnormal day. Not on the calm Tuesday: it breaks when the disk is full, the token expired over the weekend, the network is flaky because half the region is having a bad day. Which means it breaks exactly when you have the least spare capacity to improvise, and an untested manual fallback invented under pressure at midnight is where the real damage happens. Not from the original failure. From the improvisation after it.
So the habit is: be optimistic about what you build, pessimistic about what will happen to it. Destructive operations state their rollback before they run. Backups prove themselves before anything trusts them. Deploys assume the new version is guilty until it answers on the right port. And every script carries the by-hand steps at the top, because boring instructions are what a stressed brain can follow, and stress will be present at the exact moment those instructions matter.
Planning for my own fix to fail is not self-doubt. It is the opposite: enough confidence in the whole system that no single piece of it, including me, ever needs to be perfect on the first try at 2am.
The runbook is a love letter to whoever is on the other end of the failure. Three stories in, I can tell you exactly who that is. It is future me. He has always been grateful.