What is SRE? Site reliability engineering explained

Site reliability engineering (SRE) is a discipline that applies software engineering to operations problems so that systems stay reliable as they scale. It originated at Google in 2003, when Ben Treynor Sloss was asked to run a seven-engineer production team, and was codified in the 2016 book Site Reliability Engineering, which Google publishes free online.

TL;DR #

• SRE treats operations as a software problem: reliability work is engineered, measured, and automated.
• Three mechanisms do most of the work: service level objectives (SLOs) with error budgets, a 50% cap on operational work, and blameless postmortems.
• Error budgets gate release velocity: when the budget is spent, launches pause and reliability work takes priority, ending the developers-versus-operations standoff.
• SRE is a concrete implementation of DevOps ideas; the Google SRE workbook phrases it as "class SRE implements interface DevOps."
• Observability is SRE's primary instrument; none of the mechanisms work without telemetry to measure them against.

What is site reliability engineering? #

Site reliability engineering (SRE) is an approach to running production systems that assigns operations work to engineers who solve it with software. Instead of scaling headcount with traffic, SRE teams automate deployment, monitoring, incident response, and capacity management, and use quantitative targets (SLOs and error budgets) to decide how reliable a system needs to be.

The term comes from Google. Ben Treynor Sloss, who started the function in 2003, defines it in the Google SRE book: "SRE is what happens when you ask a software engineer to design an operations team." The model spread after Google published Site Reliability Engineering: How Google Runs Production Systems (O'Reilly, 2016), and today SRE teams exist across most large engineering organizations, from banks to streaming services.

The premise behind the discipline is that 100% reliability is the wrong target. Users can't distinguish 99.999% from 100%, but each extra nine costs dramatically more. SRE makes that trade-off explicit: pick a reliability target the business actually needs, then spend the slack on shipping faster.

Why does SRE matter? #

SRE matters because it resolves the structural conflict between development teams, who are rewarded for shipping changes, and operations teams, who are rewarded for stability. By agreeing on an explicit reliability target and an error budget, both sides share one number that decides when to ship and when to stop.

Without that shared number, the conflict plays out as process: release freezes, launch reviews, and gatekeeping that slow everyone down without making systems measurably more reliable. With it, the argument becomes arithmetic. A service with a 99.9% availability SLO has an error budget of 0.1%, roughly 43 minutes of downtime per 30 days. While budget remains, teams ship as fast as they like; when it is exhausted, releases pause and engineering effort shifts to reliability. The Google SRE book notes that this only works with management support: a product team won't stop launching because a dashboard turned red unless leadership backs the policy.

The second reason SRE matters is economic. Manual operational work scales linearly with service growth; software does not. A team that automates itself out of repetitive work can run far larger systems with the same headcount.

How does SRE work? #

SRE works through three reinforcing mechanisms: SLOs with error budgets set the reliability target and gate release velocity; a cap on operational work forces automation; and blameless postmortems turn each incident into system improvements. Together they convert reliability from a vague aspiration into a measured, budgeted engineering activity.

SLOs and error budgets #

A service level indicator (SLI) is a measurement: the fraction of requests served successfully in under 300 ms, for example. A service level objective (SLO) is the target for that measurement, such as 99.9% over 30 days. The error budget is what remains: the 0.1% of requests that are allowed to fail. Teams spend the budget on launches, experiments, and planned maintenance; exhausting it triggers a slowdown. The distinctions between SLIs, SLOs, and customer-facing SLAs, and how to pick targets, are covered in SLO vs SLI vs SLA, explained.

The 50% toil cap #

Google places a 50% cap on the aggregate "ops" work (tickets, on-call, manual interventions) that an SRE team performs, per the book's chapter on eliminating toil. The remaining time goes to engineering projects that reduce future toil. The cap is an upper bound, not a target: a healthy SRE team automates until operational load is a small fraction of its time.

Blameless postmortems #

After every significant incident, the team writes a postmortem that documents the timeline, contributing causes, and follow-up actions, without naming culprits. Blame destroys signal: engineers who fear punishment hide details, and the organization loses the information it needs to prevent recurrence. The postmortem culture chapter of the SRE book describes the practice; the broader lifecycle from alert to postmortem is covered in the incident response process.

What does a site reliability engineer do? #

A site reliability engineer spends part of their time on operational duties (on-call shifts, incident response, and release management) and the majority on engineering work: building automation, capacity planning, improving monitoring, and hardening systems against the failure modes that previous incidents exposed.

Day to day, that breaks down into a few recurring activities:

• On-call and incident response. Carrying the pager for owned services, triaging alerts, and acting as incident commander or responder when something breaks. Diagnosis speed depends heavily on the quality of telemetry, which is why SREs treat metrics like MTTD and MTTR as products of their own tooling choices.
• Postmortems and follow-up. Running root cause analysis after incidents and landing the fixes, not just filing them.
• Capacity planning and provisioning. SREs forecast organic and launch-driven demand, load-test to map raw capacity to serving capacity, and provision ahead of need.
• Reliability tooling. Building deployment pipelines, canary analysis, autoscaling, alerting rules, and dashboards: the software that replaces manual operations.
• Consulting with product teams. Reviewing designs for operability, defining SLOs with service owners, and onboarding new services to production standards.

The engineering share of the role is what distinguishes an SRE from a traditional system administrator, whose week goes to the manual work an SRE would automate away.

SRE vs DevOps: what's the difference? #

DevOps is a cultural philosophy: break down silos between development and operations, automate, measure, and share ownership. SRE is a concrete job function and set of practices that turns those ideas into enforceable mechanics. The Google SRE workbook summarizes the relationship as "class SRE implements interface DevOps."

The practical differences show up in scope and prescription. DevOps names the values; SRE prescribes the mechanics, down to the error-budget policy that decides whether this week's releases ship. Platform engineering, a third term often mixed in, builds the internal developer platform (golden paths, self-service infrastructure) that both depend on.

Teams don't have to choose one. A common arrangement is DevOps culture organization-wide, an SRE function for the highest-stakes services, and a platform team supplying the shared infrastructure.

What is observability in SRE? #

Observability in SRE is the ability to understand a production system's internal state from its telemetry (logs, metrics, and traces) well enough to detect SLO violations and diagnose novel failures. It is the instrument every SRE mechanism depends on: an SLO that cannot be measured cannot be enforced, and an error budget nobody can compute gates nothing.

The SRE book's monitoring chapter distills what to watch into four golden signals: latency, traffic, errors, and saturation. Frameworks like RED and USE turn them into per-service checklists. Instrumentation standards such as OpenTelemetry, a CNCF project, make the telemetry portable across tools.

At the data layer, SLIs are aggregations over raw telemetry. Stored as wide events, the availability SLI above is one query shape away:

1-- Availability SLI and error-budget burn over raw request events
2-- (illustrative schema; SLO target = 99.9%)
3SELECT
4    countIf(status_code < 500 AND duration_ms < 300) / count() AS sli,
5    (1 - sli) / (1 - 0.999) AS budget_burn  -- >1 means burning too fast
6FROM request_events
7WHERE timestamp >= now() - INTERVAL 30 DAY
8  AND service = 'checkout'

Because the SLI is computed from raw events rather than pre-aggregated series, the same data answers the follow-up questions (which region, which customer, which deploy) that turn detection into diagnosis.

What are the challenges of adopting SRE? #

The most common SRE failure mode is renaming an operations team without changing its mandate: same ticket queue, same manual work, new title. SRE only functions when the organization gives the mechanisms teeth by protecting engineering time under the toil cap and genuinely binding release decisions to the error budget.

Specific pitfalls recur across adoptions:

• SLOs without consequences. If exhausting the error budget never pauses a launch, the SLO is a dashboard decoration. The policy needs management backing in place before the first violation puts it to the test.
• Toil creep. Operational load grows quietly until it consumes the engineering time meant to eliminate it. Teams that do not measure toil cannot cap it.
• Hero culture and on-call burnout. Rewarding individuals for repeated firefighting hides systemic problems and burns out the people masking them. Sustainable on-call needs rotation depth, alert quality, and postmortem follow-through.
• Tool-first adoption. Buying an incident-management platform or an AIOps product doesn't create SRE; the practices and the telemetry foundation have to exist for the tooling to act on.

Teams that start small, with one critical service, one SLO, and one enforced error-budget policy, tend to succeed where big-bang reorganizations stall.

Measuring SLOs with ClickStack #

Computing SLIs over raw events, burning down error budgets, and drilling from an alert into the offending traces is an analytical query workload, and that is what ClickStack, the ClickHouse observability stack, is built for. It ingests logs, metrics, and traces natively via OpenTelemetry, stores them as wide, queryable events in ClickHouse, and exposes them through HyperDX, its UI, alongside full SQL for exactly the kind of SLI aggregations shown above. Teams that want to define an SLO and query the error budget directly from raw telemetry can try ClickStack as their observability backend.