The incident response process: from alert to postmortem

The incident response process is the structured sequence a team follows when production breaks: detect the problem, declare and triage it, mobilize responders, mitigate the impact, resolve the cause, and learn from it. The model most engineering teams use descends from the Incident Command System developed by US firefighters in the 1970s, adapted for software operations by Google and PagerDuty.

TL;DR #

• The lifecycle has six stages: detect → declare and triage → mobilize → mitigate → resolve → postmortem. Mitigation (stopping user impact) comes before resolution (fixing the cause), and the two are deliberately separate.
• Three roles cover most incidents: an incident commander who coordinates, a communications lead who handles updates, and subject-matter responders who do the technical work. Google's SRE book borrows the structure from the Incident Command System.
• Severity levels (SEV1–SEV4) decide how hard to escalate. The working rule from Google's incident management chapter: declare early, downgrade freely.
• Postmortems are blameless because blame destroys signal: engineers who fear punishment omit the details the organization needs to prevent recurrence.
• Detection and diagnosis both live in observability: alert quality sets time to detect, and query speed over telemetry sets diagnosis speed.

What is incident response? #

Incident response is the practice of restoring a degraded or failed service to normal operation as quickly as possible while keeping stakeholders informed. An incident, in ITIL's long-standing definition, is an unplanned interruption to a service or a reduction in its quality, which covers anything from a full outage to elevated error rates on one endpoint.

The term is shared with security: NIST SP 800-61 describes incident response for security breaches, with containment and forensics stages that operational incidents rarely need. This article covers the operational kind: the process site reliability engineering teams run when a deploy goes bad, a dependency fails, or latency spikes at 3 a.m.

The defining constraint is that incident response happens under time pressure with incomplete information. That is why mature teams script the stages, roles, and severity definitions in advance, so that nobody has to design a coordination structure during an outage.

What are the stages of the incident response process? #

The incident response process runs through six stages: detection surfaces the problem, declaration and triage assign it a severity, mobilization assembles responders under an incident commander, mitigation stops the user impact, resolution fixes the underlying cause, and the postmortem converts the incident into preventive work.

1detect → declare & triage → mobilize → mitigate → resolve → learn
2(alert)   (severity, SEV1-4)  (roles)    (stop the   (fix the   (blameless
3                                          bleeding)   cause)     postmortem)

• Detect. An alert fires, a customer reports a problem, or an engineer notices an anomaly. Alert-driven detection is the goal: it is faster and measurable as mean time to detect, covered in MTTR, MTTD, and the incident metrics that matter.
• Declare and triage. Someone states "this is an incident," assigns a severity level, and opens a dedicated channel. Declaration is cheap; late declaration is not.
• Mobilize. Responders assemble under an incident commander with clear roles (next section). Paging follows the on-call rotation, not personal address books.
• Mitigate. Stop the user-facing damage first: roll back the deploy, fail over, shed load, disable the feature flag. Google's emergency response chapter treats rollback-first as the default reflex.
• Resolve. With impact contained, fix the underlying cause and verify recovery against the same signals that detected the problem.
• Learn. Write the postmortem, extract action items, and track them to completion. Skipping this stage guarantees a rerun.

Mitigation before diagnosis is the counterintuitive part. Teams that insist on understanding the root cause before acting extend outages; the full causal analysis belongs in root cause analysis after service is restored.

Who does what during an incident? #

An incident response team splits into three roles. The incident commander owns coordination and decisions, the communications lead handles updates to stakeholders and customers, and subject-matter responders (sometimes called operations leads) do the hands-on technical work. One person can hold multiple roles in small incidents; the roles still exist.

The structure comes from the Incident Command System (ICS), created after the 1970 California wildfire season to coordinate multi-agency emergency response. Google adapted it for software, and its SRE book chapter on managing incidents is explicit about the lineage; PagerDuty published its own adaptation as the open PagerDuty Incident Response guide.

The critical rule is that the incident commander does not debug. The moment the coordinator dives into a terminal, nobody is tracking the overall state, delegating tasks, or deciding when to escalate. ICS calls this maintaining a "recognized command structure": every responder knows who is in charge and what they themselves own.

How do incident severity levels work? #

Severity levels are a pre-agreed scale, commonly SEV1 (critical) down to SEV4 (low), that maps an incident's impact to who gets paged, whether an incident commander is assigned, and how often stakeholders hear updates. Deciding severity from a table takes seconds; negotiating it during an outage wastes minutes that count.

The thresholds vary by business; the discipline does not. Two rules make severity schemes work in practice. First, declare early and downgrade freely: Google's SRE book notes it is better to declare an incident early and close it as a false alarm than to spin up incident management hours into a growing problem. Second, severity describes impact, not effort: a one-line config fix that takes the site down is still a SEV1.

How do you write an incident response plan and runbook? #

An incident response plan is the standing document that defines severity levels, roles, escalation paths, and communication channels before anything breaks; a runbook is the per-service or per-alert companion that tells the responder what to check and what to do for a specific failure mode.

A useful runbook fits on one screen and is written for a stressed engineer at 3 a.m. A minimal outline:

1# Runbook: <alert or failure mode>
2
3## Impact
4What users experience when this fires.
5
6## Severity guidance
7Conditions that make this SEV1 vs SEV2/3.
8
9## Dashboards and queries
10Links to the service dashboard; the 2-3 queries that
11confirm or rule out the usual suspects.
12
13## Mitigations (in order)
141. Roll back the last deploy (command/link).
152. Fail over / disable feature flag <name>.
163. Scale out <component>.
17
18## Escalation
19Owning team, on-call schedule link, upstream dependencies.
20
21## Known pitfalls
22Things that look related but aren't; past incidents.

Runbooks decay. Tie each one to an alert, review it after every incident that used it, and delete steps that automation has absorbed: a runbook whose first three steps are automated is three steps of noise.

How do you run a blameless postmortem? #

A blameless postmortem is a written analysis of an incident that documents the timeline, impact, contributing causes, and follow-up actions while assuming every person acted reasonably given what they knew at the time. The output is a set of action items with owners and deadlines rather than a culprit.

The mechanics, drawn from Google's postmortem culture chapter:

• Trigger criteria are pre-agreed. User-visible downtime beyond a threshold, any data loss, or on-call intervention typically require a postmortem; deciding case by case invites skipping the embarrassing ones.
• A timeline anchors the document. Timestamps for when the problem started, when it was detected, when mitigation landed, and when it fully resolved. The detect-to-mitigate gap feeds the metrics that matter.
• Causes are systemic, plural, and specific. "The deploy pipeline allowed an untested config change to reach all regions simultaneously" is actionable; "human error" is not.
• Action items get owners and tracking. A postmortem whose actions die in a backlog changed nothing.

Blamelessness is an engineering requirement before it is a courtesy. Blame destroys signal: an engineer who expects punishment describes what happened selectively, and the missing details are precisely the ones the next incident will exploit.

How does observability determine incident response speed? #

Observability sets both ends of incident response speed. Detection latency is alert quality: good alerts catch user-facing impact in minutes, while noisy ones train responders to ignore the pager. Diagnosis latency is query speed: responders spend most of an incident asking questions of logs, metrics, and traces, and every slow answer extends the outage.

Alerting on the four golden signals (latency, traffic, errors, saturation) is the standard starting point, because they track what users experience rather than what components report. Once responders mobilize, the questions become discriminating ones: which region, which endpoint, which customer segment, what changed since the last deploy.

This is why teams treat observability as part of their incident-response infrastructure. If confirming a hypothesis means a query over billions of raw events, the latency of that query is added, repeatedly, to the outage. High-cardinality questions ("is this one customer, one node, or one build?") discriminate between hypotheses fastest, and aggregated dashboards cannot answer them.

Investigating incidents with ClickStack #

The diagnosis loop (filter an alert's time window, group errors by arbitrary dimensions, pivot from a spike to the exact traces behind it) is an analytical query workload, and that is what ClickStack, the ClickHouse observability stack, is built for. It ingests logs, metrics, and traces via OpenTelemetry, stores them as wide, queryable events in ClickHouse, and exposes them through HyperDX, its UI, with full SQL for the high-cardinality questions that make up most of incident triage. Teams that want the "which customer, which build, which node" query to come back in seconds during a SEV1 can try ClickStack as their observability backend.