48.9k
A latency percentile is the value below which a given percentage of requests complete: p99 is the response time that 99% of requests beat and the slowest 1% exceed. Google's Site Reliability Engineering book recommends percentile-based indicators over averages because latency distributions are right-skewed. In the worked example below, the mean is 80 ms while p99 is 2.4 seconds.
TL;DR
- Latency distributions are right-skewed: 12 slow requests out of 1,000 push the mean to 80 ms while the median stays at 52 ms and p99 hits 2,400 ms.
- Google's SRE book recommends percentile-based latency indicators because averages flatten the distribution's shape into a single misleading number.
- Percentiles cannot be averaged across hosts or time windows. In the two-host example below, averaging per-host p99s reports 550 ms where the true fleet p99 is 1,000 ms.
- At scale, percentiles come from mergeable approximations: histogram buckets (Prometheus, OpenTelemetry) or quantile sketches (HdrHistogram, t-digest, DDSketch), each with known error bounds.
What is a latency percentile?
A latency percentile marks a position in the sorted distribution of response times. p50, the median, is the time half of all requests beat; p95 is the time 95% of requests beat; p99 is the time 99% beat. The slowest 1% above p99 is the latency tail.
The mental model is a sorted list. Take 1,000 response times, order them fastest to slowest, and p99 is roughly the 990th value. Percentiles are order statistics: each one reports a real position in the data, which is why a set of them (p50, p90, p95, p99) sketches the whole distribution's shape. That slowest 1% matters more than its share suggests; tail latency disproportionately hits your busiest users and compounds across service calls. Percentile charts are the standard latency view in application performance monitoring for this reason.
Why use percentiles instead of averages for latency?
Because the mean of a right-skewed distribution sits nowhere near what any real user experienced. A handful of very slow requests drags the average up while the median stays put, and the same average simultaneously hides how slow those requests were. Percentiles report actual positions in the distribution instead of a single distorted summary.
Suppose a service handles 1,000 requests in a minute: 988 complete in 52 ms, and 12 stall for 2,400 ms behind a lock.
| Statistic | Value | What it says about the 1,000 requests |
|---|---|---|
| Mean (average) | 80 ms | Matches no request that actually happened: fast ones took 52 ms, slow ones 2,400 ms |
| p50 (median) | 52 ms | The typical request |
| p95 | 52 ms | Even the 95th percentile misses a 1.2%-sized tail |
| p99 | 2,400 ms | 12 requests, and the users behind them, waited 2.4 seconds |
The mean sails under a 100 ms alert threshold while p99 is 30 times higher. At one million requests per day, that "healthy" average conceals roughly 12,000 requests slow enough to feel broken. The SRE book's service level objectives chapter draws the same conclusion: use percentiles for indicators, because averages permit exactly this failure.
Why can't you average percentiles?
Percentiles are order statistics: computing one requires the full sorted distribution, and a finished percentile value cannot be recombined. Averaging p99s across hosts, windows, or shards produces a number that is not the p99 of anything. The correct method is to merge the underlying distributions, or mergeable sketches of them, then take the percentile.
Host A serves 1,000 requests, every one in 100 ms, so its p99 is 100 ms. Host B serves 1,000 requests, 900 at 100 ms and 100 at 1,000 ms, so its p99 is 1,000 ms. Averaging the two per-host values gives 550 ms. The true p99 over all 2,000 requests is 1,000 ms: the 100 slow responses occupy the top 5% of the combined distribution, so the 99th percentile lands squarely among them. The naive average understates reality by almost half, and request-count weighting doesn't fix it; the information needed was destroyed when each host reduced its distribution to one number.
The same trap applies over time: averaging 60 per-minute p99 values does not yield the hourly p99. Storage formats that stay mergeable exist precisely to avoid this.
How are percentiles computed at scale?
Exact percentiles require keeping every observation, so most telemetry systems approximate. The two dominant approaches are bucketed histograms (Prometheus classic histograms, OpenTelemetry explicit-bucket histograms) and quantile sketches (HdrHistogram, t-digest, DDSketch). Both are compact and mergeable across hosts and windows; the trade-off is a bounded approximation error at read time.
Bucketed histograms count observations falling into fixed ranges. Prometheus's histogram_quantile() walks the cumulative bucket counts to the requested quantile and interpolates linearly inside the target bucket, so accuracy depends entirely on how the buckets are laid out. Exponential histograms in the OpenTelemetry data model replace hand-picked bounds with exponentially scaled buckets and a fixed relative error.
Sketches adapt their structure to the data instead. HdrHistogram (Gil Tene) records values at configurable precision across a wide dynamic range; t-digest (Dunning & Ertl) clusters observations into centroids sized to be most accurate at the extremes; DDSketch (Masson, Rim & Lee, VLDB 2019) guarantees a fixed relative error and full mergeability.
One measurement pitfall survives all of these: coordinated omission, named by Gil Tene in "How NOT to Measure Latency". Load generators that wait for each response before sending the next stop sampling during stalls, silently deleting the worst observations before any percentile is computed.
p50 vs p95 vs p99: which should you alert on?
Alert on the percentile that maps to your service level objective: p99, or p95, for user-facing latency, because the tail is what users actually feel. Watch p50 for capacity trends and regressions in the typical path. At low traffic, p99 jumps with every slow request, so prefer p95 or longer evaluation windows.
Decision rules that hold up in practice:
- User-facing SLO: alert on p99 (or p95) against the objective, ideally via burn rates over multiple windows; how SLOs, SLIs, and error budgets fit together covers the mechanics.
- Fan-out services: tail percentiles set overall latency. Dean and Barroso's "The Tail at Scale" (CACM, 2013) shows that when one request touches 100 backends, a 1-in-100 slow response at each backend delays 63% of user requests.
- Capacity and trends: p50 shifts indicate systemic change (a slow deploy, a saturated pool) rather than tail noise.
- Low-traffic services: below a few hundred requests per window, p99 is a handful of samples; alert on p95 or widen the window.
When p99 does spike, the cause is usually systemic, and hedged requests and load shedding are the standard mitigations.
How ClickStack computes percentiles over raw events
ClickStack, the ClickHouse-based observability stack, sidesteps the pre-aggregation problem by keeping raw events. Spans, logs, and metrics land in ClickHouse as wide rows, and percentiles are computed at query time with ClickHouse's quantile functions, such as quantileTDigest(), which builds the t-digest sketch described above on the fly. Against a simplified schema, the query looks like this:
SELECT
ServiceName,
quantileTDigest(0.50)(Duration) AS p50,
quantileTDigest(0.95)(Duration) AS p95,
quantileTDigest(0.99)(Duration) AS p99
FROM otel_traces
WHERE Timestamp >= now() - INTERVAL 1 HOUR
GROUP BY ServiceName
ORDER BY p99 DESCBecause the sketch is built from raw rows per query, the can't-average-percentiles trap never appears: any grouping, filter, or time window yields a correct percentile for exactly that slice. For dashboards that pre-aggregate, ClickHouse's -State/-Merge combinators store the t-digest itself in a materialized view, keeping rollups mergeable instead of freezing finished percentile values. HyperDX, ClickStack's UI, renders p50/p95/p99 series from these queries by default. ClickStack is open source; the ClickStack documentation shows the full schema these queries run against.
FAQ
How do you calculate p99 from a histogram?
Walk the buckets in order, accumulating counts until you reach 99% of total observations, then interpolate within that bucket's bounds; Prometheus's histogram_quantile() interpolates linearly. Accuracy is bounded by bucket width: a bucket spanning 1 to 2.5 seconds can place p99 anywhere inside it. Exponential histograms shrink this to a fixed relative error.
Is p50 the same as the median?
Yes. p50, the 50th percentile, and the median are the same statistic: the value splitting the sorted distribution in half. In a right-skewed latency distribution the median sits well below the mean, which is why a service's average can run far above what a typical request actually takes.
What is a good p99 latency target?
There is no universal number: targets derive from user expectations and product requirements, which is why the Google SRE book frames them as service level objectives rather than constants. Interactive endpoints commonly carry budgets in the low hundreds of milliseconds; batch and reporting endpoints tolerate seconds. Set the target from user tolerance, then alert on it.
Should you still track average latency at all?
The mean keeps two legitimate jobs. It aggregates exactly (a request-weighted mean of means is correct, unlike any combination of percentiles), and total time spent serving requests maps directly to cost and capacity. Treat average latency as a resource metric rather than a user-experience metric, and never as an SLO indicator.
