Your system will break. The only question is whether you're watching when it does.

You've done the reviews and the dashboards are green. Then the outage arrives and nothing behaves like the runbook said: alerts fire late, the failover nobody tested stalls, and everyone is learning the system live.

A game day is how you find that out on a Tuesday afternoon, on purpose, with everyone in the room, instead of at 3 AM. Here's the checklist I use.

1. Before: pick a small, safe fight

• One question, written down. Example: "If our cache dies, does checkout slow down or fall over?"
• A hypothesis with a number. "p95 latency stays under 800 ms and error rate under 1%."
• Smallest blast radius that still teaches you something. Staging first, then 1% of prod traffic, one zone, or one service.
• Abort criteria agreed upfront. For example: error rate above 2%, any data loss, or a customer-facing alert. Anyone can call the stop.
• A rollback that you have actually tested. If you can't undo the fault in under a minute, shrink the experiment.
• The right people and a heads-up. On-call, the service owner and support know the time window, and no deploys are running.
• A steady-state baseline. Capture 15–30 minutes of normal metrics so you know what "normal" looks like.

2. During: break one thing, watch everything

• Inject one fault at a time. Kill an instance, add latency, block a dependency, or fill a disk.
• Assign a timekeeper and a scribe. Log what happened, when, and who noticed.
• Did alerts fire, and for the right reason? Note the time to detect.
• Did the runbook work as written? Mark every step someone had to improvise.
• What did users see? Look at the actual user path, not only server health.
• For AI agents: fail the model or a tool call. Does the agent time out, fall back, and say "I can't verify that" instead of guessing?
• Stop the moment an abort criterion is hit. Stopping early is a result, not a failure.

3. After: turn surprises into fixes

• Run a blameless review within 48 hours. What did we expect? What happened? What surprised us?
• Fix the top 3 findings, each with an owner and a date. Ignore the long tail for now.
• Update the runbook and the alerts with what you learned.
• Re-run the same experiment after the fixes, then grow the blast radius.
• Put the next game day on the calendar. Quarterly at minimum, and after any big architecture or model change.

If you run one this month, tell me what broke first. That's my favourite part.

All opinions are my own.