Failure rehearsal and pre-mortems

Every team I have worked with can tell you how the current plan might break. Ask an engineer over lunch and they will name the dependency that always slips and the migration nobody wants to own. Everyone already knows where the risk is, and the team still rarely acts on what it knows until the thing has gone wrong.

That is where a pre-mortem or a practice run is supposed to help, and where it tends to disappoint. People have no trouble picturing the plan breaking. They picture it too late to change anything, nobody is asked to own what they find, and the list is forgotten by the time the meeting ends. A pre-mortem run the afternoon before launch, when the plan is already locked, changes nothing, and neither does a practice run that produces a page of notes and no owner. The late version survives because it lets everyone feel careful without putting the date at risk, and nobody in the room has an incentive to point that out.

So I run one only when two things are true: it can still change the decision, and every risk it turns up leaves with someone's name on it. If I cannot say yes to both, I do not run it, and the rest of this is how I run the ones that pass.

Pick the kind that fits the risk

There are two kinds, and they answer different questions. The first happens on paper: you gather the people close to the plan, tell them to assume it has already failed, and ask them to say why. That is the pre-mortem, and it suits a decision you have not committed to yet, a quarter-long bet or an architecture you are about to pick. The second kind is live. You break the real system on purpose and watch what the team does, in a game day or a disaster-recovery test, and that suits a system already running, where the question is not what might go wrong but whether the response works.

Use either only when the decision is hard to undo or reaches a lot of the system. A small, reversible change does not need one, and running one anyway is how people learn to roll their eyes at the whole idea, and the eye-rolling will still be there on the decision that deserved the exercise.

Make it safe to disagree

Telling people to assume the plan failed does one useful thing a plain "what are the risks" does not: it makes speaking up cheap. When you ask a team to find holes in a plan you clearly want, an honest answer costs something, because nobody wants to be the person slowing everyone down. Once the failure is stated as a fact, saying why it happened stops being criticism of your plan and becomes answering the question you asked.

Then protect that cheapness. Have people write their reasons down on their own before anyone talks, so the loudest person does not set the tone, and collect the notes before the discussion starts. Speak last yourself, because the moment you open with your own theory you have told the room which answers you want to hear. All of this ceremony exists for the one worry someone has been sitting on that you never considered, and it takes very little to convince them it is safer kept to themselves.

Give every risk an owner

A list of ways the plan could fail is worth nothing by itself, so each item has to leave the room decided: either someone owns it and has a fix, or the team looks at it and agrees, out loud, to accept it. Nothing gets to stay a worry everyone shares and nobody holds, because those are the ones that come back.

The version of this I have seen work is unglamorous. A team a week out from a data migration runs the pre-mortem, and someone writes down that the rollback has never been tested. Everyone knew that, and nobody had said it, because it was always going to be fine. Written down, it got an owner and became a two-day task before launch instead of a 2 a.m. problem during it. The exercise predicted nothing clever, but it got the team to say the thing it already half-knew and to put a name next to it.

Run it before the plan is locked

Run it too late and it is only for show. If the plan is set, the date is fixed, and all anyone can do is nod, then everyone leaves feeling diligent and nothing about the plan is different. Run it while the findings can still move the date or cut the scope, and leave enough room in the plan to act on them, because a risk you surface with no room to respond to it becomes a note that says you saw it coming.

For live systems, break it on purpose

On paper is enough for a plan, but not for a system already running, because there you are testing the response rather than your foresight, and you cannot rehearse a response by talking about it. So you break the thing on purpose: kill the instance, cut the dependency, or walk the team through a fake outage and have them run the real recovery steps as if it were happening. You do it to build the habit, because the first time the team handles this failure should not be the real one, and the person who wrote the service and the person who gets paged at 3 a.m. should have stood in front of the same broken thing at least once.

Put one person in charge of the drill and make them make the calls under something close to real pressure, because a drill where everyone knows the script and nobody has to decide anything only teaches people to follow a script. The gap you are looking for sits between the runbook as written and the runbook as run by a tired person at 3 a.m., and it only shows up when someone in the room genuinely has to decide.

Keep only what changed

The exercise is worth only what it leaves behind:

• a kill criterion written down
• a runbook edited
• an alert added for the failure you just pictured
• capacity set aside for the surprise you now expect
• a decision written down so the next person knows why the plan looks the way it does

If none of those exist afterward, you held a meeting.

This is the same habit as a postmortem, pointed the other way: a postmortem turns one real failure into a change, and this turns an imagined failure into the same change, earlier, so both get judged by what is different afterward.

The list gets longer with agents

Everything above predates the systems many teams are now building, and those fail in ways a normal pre-mortem never had to picture. An agent does not crash the way a server does. It keeps going and does the wrong thing, quietly, at a scale nobody is watching, so the list of things to check for grows, and the new entries are worth saying out loud when you run the exercise:

• output that reads as right and is wrong
• a long task that drifts off course over many steps
• an instruction slipped into its input that turns a helpful tool against you
• one wrong action taken with no person in the loop, and how far it reaches

For agents I run the pre-mortem as if it already happened: it is two years from now, the rollout failed, and the team writes down how. The failures people write down are rarely the model being dumb. They are an agent given more access than anyone meant to give it, an input nobody thought to distrust, a wrong action that set off no alarm because it looked like a right one. None of it is new in kind. The old risks have moved into a new place, and the list you read out just has to keep up.

Know when to skip it

None of this means running one on everything. A change you can undo in a minute is not worth the trouble, and a team that pre-mortems every ticket will have stopped taking it seriously by the time it counts. Use it where being wrong is expensive and hard to undo, and skip it everywhere else without guilt.

When you do run one, look at what came out of it. If the date moved, or somebody walked out owning a risk that had no owner that morning, it did its job. If the conversation was good and the plan is exactly what it was before, the plan was never open to change in the first place, which is an argument for running the next one earlier rather than for giving up on the idea.