How to run a root cause analysis

Set the triggers before you need them

Agree the trigger list in advance and apply it without debate: anything that reached users, any rollback called during live monitoring, any severity-one defect, any near miss caught by a person rather than by a control, and the third occurrence of the same failure shape. The delivery lead for that team makes the call within one working day and records which trigger fired. Everything else gets a bug ticket on the Bug Budget epic and no ceremony. Deciding case by case, in the mood immediately after an outage, is how the embarrassing incidents get investigated and the boring recurring ones never do.

Freeze the evidence within 24 hours

Name one owner for the incident record and give them a day.

Copy logs, dashboard screenshots, alert payloads, deploy records, feature-flag changes, support tickets and the incident chat into the record itself, rather than linking out to systems that roll at seven or thirty days. Then ask everyone involved to write down separately, before any group meeting, what they saw and when they saw it, in their own words. Separate accounts captured first are your only defence against the room converging on the loudest person's version inside the first ten minutes, and once that has happened you cannot recover the original recollections.

Build one agreed timeline in UTC

Book ninety minutes within five working days.

Invite the people who were there, plus a facilitator who was not. Open with the timeline and nothing else. Mark four moments: when the condition entered the system, when the failure began, when a human or a control first noticed, and when customers stopped being affected. Timestamp to the minute in UTC and name the source of every entry. Failure to notice is your time to detect. Failure to recovery is your time to restore. Circulate the timeline for correction before anyone discusses causes, so the argument about causes is not a disguised argument about the times.

Separate the trigger from the conditions

Draw two columns.

The trigger is the change that lit the fuse: a deploy, a traffic spike, an expiring certificate, a third party altering a response shape. The conditions are what made the system flammable: no timeout, no fallback, an alert routed to a channel with no rota, a test suite that never covered that journey. For each condition, ask what should have contained this and write the answer next to it. A rollback removes the trigger and leaves every condition standing, which is why the same incident returns wearing different clothes. Nearly every action you raise should come out of the conditions column, not the trigger.

Ask why until you reach a control

Take each condition and ask why it was there, stopping when you reach something the team can change: a standard, a test, an alert, a gate, a default, a runbook. If an answer names a person, you are not finished, ask why the system let one person's attention be the control. If an answer names something outside your control, a supplier's API or a cloud region, then the control you own is whatever should have contained it: a timeout, a circuit breaker, a cached fallback, a contract test. Three or four whys usually gets there. Chains of seven tend to be creative writing.

Test every cause against a counterfactual

For each candidate cause ask one question: if this alone had been different and nothing else had changed, would the incident still have happened? If yes, it is context and belongs in a background paragraph rather than in the causes list. If no, it is a contributing cause and it earns exactly one action. Run the test out loud so the room hears which items fail it. This is the step that stops an RCA becoming a list of everything anyone dislikes about the codebase. Expect two to four contributing causes. If you have eleven, you have written a wish list, and nobody funds a wish list.

Produce at least one detection action

Every RCA raises a detection action, no exceptions.

Write three things down: the current time to detect taken from the timeline, the target you are committing to, and the specific signal that would have fired. Name it precisely, an error-rate threshold on a stated endpoint, a synthetic journey run every five minutes, a support-contact spike, a queue depth. Then say who is rostered to answer it, because an alert routed to a channel nobody owns is not detection. If a customer told you before your systems did, state that plainly. Fixing this defect without shortening detection leaves you equally blind to the next one.

Convert findings into linked, budgeted tickets

An action that is not a ticket is a sentence.

Raise every one during the session, on screen, before people leave. Defects go to this quarter's Bug Budget epic. Missing tests, alerting and hardening go to the Tech Debt epic. Anything large enough to need product prioritisation becomes an epic under an outcome, with a quarter and a planned currency share, competing for capacity like any other epic. Give each ticket a named owner, not a team. If an action cannot get an owner and a quarter in the room, record it as deliberately declined, with the reason and who declined it, instead of listing it and quietly dropping it.

Publish it, then close actions in the open

Publish within five working days, in the same place every time, readable by anyone in the organisation rather than only the people on the call. Put the actions on the agendas of the next two retrospectives and close them out loud there. Where the failure exposed a standing risk you are choosing to live with, add it to the RAID log as an accepted risk with a named owner and a review date, so it stays visible rather than being resolved by silence. Before you open the next quarter's roadmap, reread the quarter's RCAs together: a failure shape that appears twice is the tech debt you should be funding.