What are the four golden signals of monitoring?

The four golden signals are latency, traffic, errors, and saturation: the minimum set of metrics that Google's Site Reliability Engineering book (2016) recommends for monitoring any user-facing system. If you can measure only four things about a service, measure these four: together they describe what users are experiencing and how close the system is to its limits.

The term comes from chapter 6 of the SRE book, "Monitoring Distributed Systems", written by Rob Ewaschuk. The chapter promises that if you "measure all four golden signals and page a human when one signal is problematic (or, in the case of saturation, nearly problematic), your service will be at least decently covered by monitoring." The signals are symptom-oriented: they detect that users are affected before you know why.

What does each golden signal measure? #

Latency tells you how slow the service is, traffic how busy it is, errors how often it fails, and saturation how full it is.

Latency must split successful requests from failed ones. A fail-fast HTTP 500 returns in milliseconds and drags the average down, masking a real problem; a slow error (a request that times out after 30 seconds) is worse than a fast one. Track percentiles, not averages: Dean and Barroso's "The Tail at Scale" (CACM, 2013) shows that with one server in 100 responding slowly, a request fanning out to 100 servers has a 63% chance of hitting the slow path.

Traffic is the demand measure that gives the other three context: an error spike at 10 requests per second means something different from the same spike at 10,000. Pick a high-level, system-specific measure such as requests per second for an API, concurrent sessions for streaming, or reads and writes per second for storage.

Errors come in three forms, per the SRE book: explicit (an HTTP 500), implicit (an HTTP 200 carrying the wrong content), and by policy (if you promised one-second responses, a 1.2-second success counts as an error). Counting only 5xx responses undercounts all three.

Saturation is how full the service is, measured against the resource that will run out first rather than average utilisation across the fleet. It also covers prediction: "the database will fill its disk in four hours" is a saturation alert.

Why do the four golden signals matter? #

The golden signals matter because they are symptom-based: they alert on user-visible problems (slow, failing, or overloaded services) regardless of the underlying cause. Cause-based alerts multiply with every architectural change; the four signals stay stable, which keeps paging rules small and makes them the natural starting set for any new service.

They also feed directly into reliability targets. Latency and error ratio are the two most common service level indicators, so a team that instruments the golden signals has already done most of the measurement work for SLOs, SLIs, and SLAs.

How do the golden signals differ from RED and USE? #

The golden signals are the umbrella. The RED method (Rate, Errors, Duration; Tom Wilkie, 2015) is the golden signals minus saturation, scoped to request-driven services. The USE method (Utilisation, Saturation, Errors; Brendan Gregg) applies to every hardware and software resource rather than to services.

In practice, RED tells you whether users are having a bad time, USE tells you whether a resource is the bottleneck, and the golden signals span both by pairing three request-side signals with one resource-side signal. Most teams run RED per service and USE per resource; RED method vs USE method walks the same outage through both lenses to show how they complement each other.

What does golden-signal data look like at rest? #

Three of the four signals fall out of a single wide request-event table: one row per request with a timestamp, duration, and status code. The SQL shape below computes latency (split by outcome), traffic, and errors in one pass; saturation comes separately, from resource gauges such as memory usage or queue depth.

1-- Illustrative shape: one wide event per request
2SELECT
3  time_bucket('1 minute', timestamp)                        AS minute,
4  count(*)                                                  AS traffic_rpm,
5  countIf(status_code >= 500) / count(*)                    AS error_ratio,
6  approx_percentile(0.99, duration_ms)
7    FILTER (WHERE status_code < 500)                        AS p99_success_ms,
8  approx_percentile(0.99, duration_ms)
9    FILTER (WHERE status_code >= 500)                       AS p99_error_ms
10FROM http_request_events
11GROUP BY minute
12ORDER BY minute;

At scale, this query pattern (percentile aggregations over billions of rows, grouped by time and sliced by service, route, or customer) decides whether a golden-signal dashboard renders in milliseconds or minutes, which makes the storage engine underneath the signals a first-order design choice.

Common misconceptions about the golden signals #

• Saturation means average CPU utilisation. Saturation is the fullness of the most constrained resource. A service at 30% average CPU can be saturated on memory, connection pools, or disk; the SRE book's guidance is to emphasise "the resources that are most constrained."
• Errors are just HTTP 5xx responses. Explicit failures are one of three error classes; wrong-content 200s and policy violations (responses slower than the committed latency) are errors too.
• Low average latency means users are fine. Averages hide the tail. When one request touches many servers, the p99 of individual servers determines the user experience, which is why golden-signal latency is tracked as percentiles.
• The golden signals replace SLOs. They are inputs, not targets. The signals tell you what to measure; an SLO says how good those measurements must be.

Monitoring the golden signals with ClickStack #

ClickStack, the ClickHouse observability stack, ingests golden-signal data through native OpenTelemetry pipelines and stores it as wide events in ClickHouse, whose columnar engine handles the query shape shown above: percentile and ratio aggregations over high-volume request data. HyperDX, the stack's UI, is where teams build per-service latency, traffic, and error dashboards and attach alerts. Because raw events are retained rather than pre-aggregated away, a golden-signal alert can be drilled down to the individual slow or failing requests behind it. ClickStack is open source and takes a few minutes to try.