Small, reversible, predicted: three habits for safer changes

Most outages I read about aren't a missing tool. They're a change nobody could undo quickly, a guess nobody wrote down, and a review that ended at "human error".

Three habits that cost nothing and catch a lot. These started as three short LinkedIn posts; here they are together.

1. Predict the outcome before you change anything

My view, as opinion: the most useful reliability habit costs nothing. Before a deploy, a config change or a failover test, write one sentence on what you expect to happen. Then compare it with what did.

• If you were right, you understand the system a little better.
• If you were wrong, you found a gap in your mental model while it was cheap.
• If you can't write the sentence, you're not ready to make the change.

2. Can't undo it in a minute? Shrink it.

The bigger the change, the longer the undo, and the longer customers sit in the blast radius. Make changes easy to reverse:

• Ship small. One change at a time, so a problem has one suspect.
• Put it behind a flag, so "undo" is a switch, not a deploy.
• Release to a slice of traffic first, and watch before widening.
• Rehearse the rollback. An untested rollback is a hope.

You don't need a perfect change. You need one you can take back quickly.

3. "Human error" is where the analysis starts

Your post-incident review ended with "human error". Then it happens again next quarter. People will make mistakes; the system should stop a mistake from becoming an outage. A review that prevents the next one:

• Asks "why was this possible?" five times, not "who did it?"
• Looks for the missing guardrail: a confirmation prompt, a permission boundary, a pre-flight check.
• Writes action items with an owner and a date, and fixes the top three before anything else.
• Updates the runbook and the alerts, so the lesson lives in the system, not in someone's memory.
• Stays blameless. People only tell you the full story when they're safe to.

Google's SRE book makes the same case for blameless postmortems: focus on contributing causes, not on individuals.

Sources

• Google SRE book: Postmortem Culture

Views my own.