How to run a root cause analysis

Live monitoring is the scheduled watch over a change after it ships: customer impact, support volume, the FAQ and the support macro, and the continue, watch or rollback call. It runs on every release, whether or not anything is wrong. An RCA is triggered by what live monitoring finds, or by an incident that arrives with no warning at all. Live monitoring asks whether this release is behaving. An RCA asks why the system allowed it not to, and it ends in funded tickets rather than a call.

Someone who did not build the thing that failed and does not manage the people who did. Their job is the method rather than the investigation: holding the timeline until it is agreed, applying the counterfactual test to every candidate cause, and stopping the room whenever an answer names a person instead of a control. Tenhaw does this as an outside facilitator, brought in because it did not build the system under discussion. Keep the room to the people who were there plus that facilitator. Once it becomes a stakeholder audience, people start performing rather than remembering, and you lose the detail you came for.

Then you have found the trigger and you still owe the conditions. You cannot action a third party's deploy schedule, but you can action the timeout you did not set, the fallback you did not build, the contract test you did not write, and the alert that would have told you their response had changed shape. Raise the commercial conversation separately, and log the dependency in the RAID log as an accepted risk with a review date, but do not let a supplier's name become the reason no ticket was raised.

Blameless means the cause is never a person, not that names vanish from the timeline. Write plainly that an engineer ran the deploy at 09:12, because the timeline is worthless without it. What you never write is that the engineer was the cause. If the honest finding is that one person's memory was the only thing standing between a change and production, then the missing control is a gate, and the action is to build it.

Agree the trigger list in advance and apply it without debate: anything that reached users, any rollback called during live monitoring, any severity-one defect, any near miss caught by a person rather than by a control firing, and the third occurrence of the same failure shape. The delivery lead makes the call within one working day and records which trigger fired; everything else becomes a bug ticket with no ceremony. Deciding case by case, in the mood straight after an outage, means the embarrassing incidents get investigated and the boring recurring ones, which cost more in aggregate, never do.

Freeze the evidence within 24 hours and hold the session within five working days. One named owner copies logs, alert payloads, deploy records and the incident chat into a record that will not rotate away, and everyone involved writes down what they saw before any group meeting. The ninety-minute session then builds one agreed timeline before causes are discussed, and the write-up is published within five working days. Wait two weeks and logs have rotated and everyone has rehearsed a version of events, so what you get is a coherent document that is partly fiction.

The trigger is the change that lit the fuse: a deploy, a traffic spike, an expiring certificate, a third party altering a response shape. The root causes almost always sit in the conditions that made the system flammable: no timeout, no fallback, an alert routed to a channel with no rota, a test suite that never covered that journey. A rollback removes the trigger and leaves every condition standing, which is why the same incident returns wearing different clothes. Nearly every action you raise should come out of the conditions column, not the trigger.

Run a counterfactual test on every candidate. If this alone had been different and nothing else had changed, would the incident still have happened? If yes, it is context and belongs in a background paragraph. If no, it is a contributing cause and it earns exactly one action. Run the test out loud so the room hears which items fail it. Expect two to four contributing causes from a typical incident; a list of eleven is a wish list of everything anyone dislikes about the codebase, and nobody funds a wish list.

Usually three or four. Stop when you reach a control the team can change: a standard, a test, an alert, a gate, a default, a runbook. If an answer names a person, you are not finished; ask why the system let one person's attention be the control. If it names something outside your control, the control you own is whatever should have contained it, such as a timeout, a circuit breaker or a contract test. Chains of seven whys tend to be creative writing rather than analysis.

Raise every action as a ticket during the session, on screen, before anyone leaves; an action that is not a ticket is a sentence. Each one gets a named owner, not a team, and a home where it competes for capacity, which in The Tenhaw Way means the quarter's Bug Budget epic for defects, the Tech Debt epic for missing tests and alerting, or a new epic under an outcome for anything larger. If an action cannot get an owner and a quarter in the room, record it as deliberately declined, with the reason. Then put the actions on the agendas of the next two retrospectives and close them out loud.

The names get used interchangeably, and the label matters far less than the bar you hold the session to. Four things decide whether it was worth running, whatever you call it: evidence frozen inside the first 24 hours, one timeline in UTC that names the source of every entry and states both time to detect and time to restore, every candidate cause put through a counterfactual so you finish with two to four rather than eleven, and at least one action that shortens detection. Every action leaves the room as a ticket with a named owner. If it ends as a document, it did not happen.

Set the mechanics up before the next incident rather than during it. Tenhaw publishes this method in full, and a team can adopt every step of it without hiring anyone. Write the trigger list down and get it agreed, and name the delivery lead for that team as the person who applies it within one working day. Decide where the incident record lives and who owns freezing it inside 24 hours, because logs and dashboards roll at seven or thirty days. Agree who facilitates, which has to be someone who did not build the thing that failed. Then book ninety minutes within five working days, open with the timeline and nothing else, and raise every action as a ticket on screen before anyone leaves.

Collect separate accounts before the room ever meets. Ask everyone involved to write down what they saw and when they saw it, in their own words, ahead of any group discussion, because ten minutes into a meeting the room has converged on the loudest person's version and the original recollections cannot be recovered. Then open the session with the timeline and nothing else. Timestamp to the minute in UTC, name the source of every entry, and mark four moments: when the condition entered the system, when the failure began, when a person or a control first noticed, and when customers stopped being affected. Circulate it for correction before anyone discusses causes.

Yes. Every RCA raises at least one detection action, because fixing the defect closes one hole and leaves you exactly as blind to the next one. Write three things down: the current time to detect taken from the timeline, the target you are committing to, and the specific signal that would have fired. Then name who is rostered to answer it, because an alert routed to a channel nobody owns is not detection. In the invented worked example on this page, card declines jump from 2% to 31% and a customer email is the first signal 41 minutes later, so the action is a decline-rate alert at 5% sustained over five minutes.

A timeline agreed by everyone who was there, with a named source on every entry and both time to detect and time to restore stated. Then two to four contributing causes, each having survived the counterfactual test and each naming a control the team can change rather than a person. Then the detection action, written as the current number, the target and the signal that would have fired. Then every other action as a linked ticket with a named owner and a quarter. Anything that failed the counterfactual sits in a background paragraph and gets no ticket. Publish within five working days, where anyone in the organisation can read it without asking.

On AI-native teams two conditions recur, and both are fixed in the outcome ticket rather than in the codebase. The first is a user journey the ticket never listed, so nothing tested it, and the fix is to add it to the standing list. The second is verification that rested on the model reporting it had finished, and the fix is to require a demonstrated run instead of a claimed one. On AI-augmented teams the failing control is usually a story's acceptance criteria or the human review that waved the change through, so the actions land in the definition of done and the test suite. Tenhaw's engineers build to a 72-rule engineering handbook, open source and enforced by an agent rather than by memory.

Reread the quarter's RCAs together before you open the next quarter's roadmap. A failure shape that appears twice is not bad luck, it is the tech debt you should be funding, and it will keep recurring until it is carried by something that competes for capacity. Missing tests and alerting go to the Tech Debt epic; anything large enough to need product prioritisation becomes an epic under an outcome, with a quarter and a planned currency share. Those epic types come from The Tenhaw Way, the operating model behind every Tenhaw engagement. Where an incident exposed a risk you are choosing to live with, put it in the RAID log as an accepted risk with a named owner and a review date.