What is root cause analysis? Methods, process, and limits

Root cause analysis (RCA) is a structured investigation that works backwards from a failure's visible symptoms to the underlying conditions that produced it, so the failure can be prevented rather than patched. The practice originated in manufacturing, with Sakichi Toyoda's 5 Whys at Toyota and W. Edwards Deming's quality movement, and is now a core discipline of software incident response.

TL;DR

Root cause analysis is a symptom-to-cause investigation aimed at stopping a class of failure from recurring.
Four methods cover most investigations: 5 Whys (Toyota), fishbone diagrams (Kaoru Ishikawa, 1960s), fault tree analysis (Bell Labs, 1962), and causal timelines.
A postmortem is the document and ritual; RCA is the analysis inside it. The Google SRE book dedicates chapter 15 to running both blamelessly.
Complex systems rarely fail from one cause. Richard Cook's 1998 treatise states that "post-accident attribution to a 'root cause' is fundamentally wrong"; practice has since shifted to contributing factors.
AI can automate evidence gathering across telemetry; forming and testing hypotheses about novel failures still needs humans.

What is root cause analysis?

Root cause analysis is a structured method for investigating failures that traces observable symptoms (an outage, a defect, a latency spike) back to the underlying causes that allowed them to happen. Its output is a set of corrective actions that change the system, so the same class of failure cannot recur.

The discipline predates software. Toyoda developed the 5 Whys inside what became the Toyota Production System, and Taiichi Ohno codified it in his 1988 book of the same name; Deming's quality-control work carried the same idea (fix the process, not the person) into Western manufacturing. Software teams adopted RCA as distributed systems made failures more frequent and less obvious, and it now sits inside the incident response process as the learning step after mitigation.

The goal is prevention, not blame. An RCA that ends with "an engineer made a mistake" has stopped too early: the useful question is why the system allowed one mistake to become an outage.

How do you do a root cause analysis?

A root cause analysis starts from a precise problem statement, reconstructs a timeline from telemetry and change records, identifies the causal chain with a structured method, and ends with corrective actions that are tracked to completion. The investigation examines conditions and processes, and the findings are shared openly.

In practice the investigation runs in six steps:

Define the problem. What happened, when, to whom, with what impact, stated in numbers ("checkout p99 latency exceeded 30 seconds for 29 minutes, affecting ~12% of sessions") rather than adjectives.
Reconstruct the timeline. Line up logs, traces, metrics, deploys, config changes, and feature-flag flips into one ordered sequence. Most software incidents correlate with a change.
Gather evidence. Pull the specific queries, dashboards, and diffs that support or refute each candidate cause. Evidence beats recollection.
Identify the causal chain. Apply a structured method (see the comparison below) to move from symptom to cause without anchoring on the first plausible explanation.
Broaden to contributing factors. Ask what detection, review, or safeguard would have stopped the chain earlier; these systemic gaps are usually the highest-value fixes.
Write it up and track actions. An RCA without owned, dated corrective actions is a document, not an analysis.

A worked example: the checkout latency incident

Consider a fictional incident at an online retailer. At 14:02, payment-service v2.41.0 deploys. At 14:05, checkout p99 latency climbs past 30 seconds; the alert fires at 14:09; rollback completes at 14:31. The 5 Whys chain: checkout was slow → the payment service's database connection pool was exhausted → the pool size had dropped from 100 to 10 → v2.41.0 migrated to a new config library whose default silently replaced the override → the review process had no config-diff step. Contributing factors: no saturation alert on the connection pool, and a 7-minute detection gap because the alert watched averages instead of p99. The corrective actions target the review process and the alerting.

How do you write a root cause analysis report?

A root cause analysis report states the impact in measurable terms, presents a timestamped timeline, walks the causal chain from symptom to underlying causes, lists contributing factors, and assigns owned corrective actions with due dates. It names systems and processes, and it is written to be read by teams who were not involved.

A minimal template:

1# RCA: <incident title> - <date>
2
3## Impact
4Duration, affected users/requests, SLO consumed, revenue effect if known.
5
6## Timeline (all times UTC)
714:02 - payment-service v2.41.0 deployed
814:05 - checkout p99 > 30s
914:09 - alert fired; incident declared SEV2
1014:31 - rollback complete; recovery confirmed
11
12## Causal chain
13Symptom → intermediate causes → underlying cause(s), with evidence links.
14
15## Contributing factors
16Missing safeguards, detection gaps, process gaps.
17
18## Corrective actions
19| Action | Owner | Due |
20|---|---|---|
21
22## What went well / what was luck

Which root cause analysis method should you use?

The four established RCA methods trade depth for effort. The 5 Whys is fastest but biases toward a single cause; fishbone diagrams force breadth across categories; fault tree analysis handles multi-condition failures rigorously; causal timelines fit distributed systems, where ordering is the evidence. Most software teams combine a timeline with the 5 Whys.

Method	Origin	How it works	Best for	Weakness
5 Whys	Sakichi Toyoda, Toyota Production System	Ask "why" repeatedly (typically ~5 times) down one causal chain	Simple, single-chain failures; fast triage of small incidents	Biases toward one cause; chain quality depends on who answers
Fishbone (Ishikawa) diagram	Kaoru Ishikawa, 1960s	Branch candidate causes into categories (people, process, tooling, environment) off a spine ending at the effect	Brainstorming breadth when the cause is genuinely unknown	Produces candidates, not conclusions; needs evidence to prune
Fault tree analysis	Bell Labs, 1962 (Minuteman missile program)	Top-down Boolean tree: which combinations of conditions (AND/OR) produce the failure	Safety-critical and multi-condition failures; quantifiable	Heavyweight; overkill for routine incidents
Causal timeline	Incident-analysis practice (aviation, then software)	Order every event and change, then test which earlier events were necessary for later ones	Distributed-systems incidents where interaction, not one defect, is the story	Depends on telemetry completeness and honest clocks

What are the 5 Whys in root cause analysis?

The 5 Whys is a questioning technique that starts from the observed problem and asks "why did that happen?" repeatedly (conventionally five times) until it reaches a cause that, if fixed, breaks the chain. It came out of Toyota's production system and works best on simple failures with one dominant causal path.

Its known failure mode is tunnel vision: five answers produce one tidy chain, while real incidents often have several. John Allspaw's essay The Infinite Hows (2014) argues for replacing "why", which invites attribution, with "how", which surfaces the conditions that made each step possible.

Root cause analysis vs postmortem: what's the difference?

A postmortem is the artifact and the ritual: the written record of an incident, reviewed with stakeholders after resolution. Root cause analysis is the investigative work inside it, the part that establishes what caused the incident. Every good postmortem contains an RCA; an RCA can exist without the surrounding ceremony.

The Google SRE book's postmortem chapter sets the tone standard: blameless. Blame matters here because it destroys signal: engineers who expect punishment stop volunteering the details the analysis depends on. Google's criteria for when a postmortem is mandatory (user-visible downtime, data loss, on-call intervention, resolution time above a threshold) are a reasonable default for any team formalizing the practice, and they connect the postmortem back to the incident response process that precedes it.

Why complex failures rarely have a single root cause

In complex systems, the phrase "root cause" is itself contested. Richard Cook's How Complex Systems Fail (1998) puts it bluntly: "post-accident attribution to a 'root cause' is fundamentally wrong," because such systems run with multiple latent flaws that only produce an outage in combination. The deploy, the silent config default, the missing alert, and the review gap in the worked example above were all necessary; none was sufficient alone.

This is why the incident-analysis practice that followed Cook's treatise (Allspaw and the resilience-engineering school behind site reliability engineering) prefers "contributing factors" to "root cause." The practical consequence is to treat RCA as plural. Ask which combination of conditions produced the failure and which safeguards were absent, and write corrective actions against several factors rather than crowning one. The method name survives; the single-cause assumption should not.

Can AI automate root cause analysis?

AI currently automates the evidence-gathering layer of root cause analysis, not the judgement layer. LLM-based agents can query logs, metrics, and traces, correlate anomalies with deploys and config changes, and draft a ranked list of hypotheses within minutes of an alert. Validating those hypotheses and choosing corrective actions remains human work.

An AI SRE agent is only as fast as the queries it can run over raw telemetry, and only as accurate as the instrumentation (typically OpenTelemetry signals) feeding it. Agents are strong on failures that resemble past ones and weak on novel failure modes, which is exactly where Cook's multiple-contributing-factors problem lives.

How ClickStack supports root cause analysis

Steps 2 and 3 of an RCA (reconstructing the timeline and gathering evidence) are query problems. ClickStack stores logs, metrics, traces, and session data as raw, high-cardinality events in ClickHouse, so investigators can ask arbitrary questions after the fact instead of being limited to whatever was pre-aggregated into dashboards. A typical evidence query is one aggregation:

1SELECT service, deploy_version, count() AS errors
2FROM traces
3WHERE status_code = 'ERROR'
4  AND timestamp > now() - INTERVAL 1 HOUR
5GROUP BY service, deploy_version
6ORDER BY errors DESC

HyperDX, the ClickStack UI, layers search, trace waterfalls, and event correlation on top, and the stack ingests OpenTelemetry natively, the same instrumentation an AI investigator would query. Teams whose RCAs stall at "we couldn't reconstruct what happened" usually need a better telemetry store rather than a better template, and the ClickStack getting-started guide covers standing one up.

FAQ

What is the goal of root cause analysis?

The goal of root cause analysis is prevention: identifying the underlying conditions that produced a failure so corrective actions can stop that class of failure from recurring. The measure of a good analysis is whether those corrective actions ship; an unactioned report prevents nothing.

When should you run a root cause analysis?

Run an RCA after any incident with user-visible impact, data loss, or on-call intervention, and after repeated near-misses of the same shape. Google's SRE practice makes postmortems mandatory above such thresholds. Small incidents deserve a lightweight version (a timeline plus a short causal chain) rather than no analysis at all.

What is the difference between a root cause and a contributing factor?

A root cause is presented as the single underlying reason a failure happened; a contributing factor is one of several conditions that were each necessary but not sufficient. Resilience-engineering practice favours contributing factors, because complex-systems failures almost always require multiple latent flaws to combine before an outage occurs.

Is root cause analysis the same as a postmortem?

No. Root cause analysis establishes what caused a failure; a postmortem wraps that analysis in a wider document covering impact, response quality, and follow-up actions, reviewed with stakeholders after the incident. An RCA can be run on any failure, formal review or not.

Running your own incident reviews? Try ClickStack, the open-source, OpenTelemetry-native observability stack on ClickHouse, and see how fast evidence gathering gets when raw telemetry is queryable.

TL;DR