What is tail latency? Why p99 spikes when averages look fine

Tail latency is the slow end of a service's response-time distribution: the requests at or beyond the 95th percentile (p95, p99, p99.9) that take far longer than the typical request. Dean and Barroso's "The Tail at Scale" (Communications of the ACM, 2013) showed why this sliver drives user experience: when a request fans out to 100 backends, a 1-in-100 slow response delays 63% of user requests.

The term describes a region of the distribution. p99 is the most common yardstick for it, but the tail is whatever your slowest few percent of requests are doing, and that slice has different causes and different fixes from the rest of the distribution. The measurement side, why percentiles rather than averages, is covered in percentiles vs averages; this page covers what makes the tail slow and what reduces it.

Why does tail latency matter? #

Tail latency matters because fan-out multiplies it. A single user request to a distributed system fans out to tens or hundreds of backends, and the whole request waits for the slowest response. Rare slowness at each backend becomes common slowness for users, so the tail sets the speed users perceive.

The paper quantified this fan-out amplification. Suppose each backend answers fast except for 1 request in 100. The chance that a user request hits at least one slow backend grows with fan-out:

In the paper's more extreme case, with only 1 request in 10,000 slow per server, a fan-out of 2,000 servers still leaves almost one in five user requests slow. Latency is one of the four golden signals precisely because of this compounding, and the same arithmetic applies within one service whose request touches many spans across a distributed trace.

What causes tail latency? #

Tail latency comes from transient resource contention and background activity inside otherwise healthy systems, not from broken code paths. The typical request misses all of these hazards; the unlucky 1% lands on one of them. Dean and Barroso call the resulting slowness "latency variability" and treat it as inherent to shared infrastructure.

The recurring causes:

• Garbage collection pauses. A stop-the-world or long GC phase in a JVM, Go, or .NET process freezes every in-flight request on that instance for tens to hundreds of milliseconds.
• Queueing. Bursty arrivals queue at every layer: thread pools, connection pools, NIC buffers, OS run queues. Wait time explodes as utilisation approaches saturation, and queueing delay compounds across hops.
• Cold caches. A request that misses an in-process or distributed cache pays the full backing-store cost, often 10-100x the hit path, as does the first traffic after a deploy or failover.
• Noisy neighbours. Co-tenant VMs, containers, or background jobs contend for CPU, disk, and network on shared hosts; the affected instance slows without any change in its own code.
• Background maintenance. Compaction, log flushes, daemon wakeups, and periodic jobs steal resources on a schedule unrelated to request traffic.
• Retries and timeouts. A retry after a multi-second timeout rescues the request but books its full wait into the tail, and retry storms feed the queueing problem above.

How do you reduce tail latency? #

You reduce tail latency by tolerating slowness rather than eliminating it. Variability cannot be removed from shared infrastructure, which is why Dean and Barroso call the effective fixes "tail-tolerant" techniques.

The techniques, roughly in order of leverage:

• Hedged requests. Send the same request to a second replica once the first has exceeded, say, the p95 latency, and take whichever answers first. In the paper's BigTable benchmark, hedging after a 10 ms delay cut p99.9 for a 100-server read from 1,800 ms to 74 ms while adding only 2% more requests.
• Tied requests. Enqueue the request on two servers at once, with each able to cancel the other's copy the moment one starts executing, removing the hedge delay entirely.
• Timeouts with retry budgets. Set timeouts near a realistic tail percentile rather than a generous constant, and cap retries as a fraction of traffic so recovery cannot become a retry storm.
• Load shedding and admission control. Reject or degrade excess work before queues saturate; a fast error beats a 30-second success that arrives after the user has gone.
• Isolating background work. Throttle compaction and batch jobs, reserve headroom for request traffic, and break large background operations into small interruptible pieces.

Fixes are workload-specific, so measurement comes first: find which endpoints, hosts, or tenants own the slow 1% before choosing a mitigation.

Common misconceptions about tail latency #

• Tail latency only affects 1% of users. It concentrates on your most active users. A session issuing 100 requests has a better-than-even chance of hitting a 1-in-100 tail at least once, and heavy users issue the most requests.
• A healthy average means a healthy service. The mean tracks the fast majority and can sit 30x below p99; averages hide exactly the requests that hurt.
• Faster hardware fixes it. Variability comes from sharing and background activity, which better hardware still exhibits; software techniques like hedging and load shedding are what actually shrink the tail.
• Tail latency and tail sampling are the same thing. Tail latency is a property of response times; tail sampling is an OpenTelemetry Collector technique that decides which traces to keep after they complete. The names collide because both concern the "tail", but one is a symptom and the other a telemetry policy.

Finding tail latency with ClickStack #

ClickStack, the ClickHouse-based observability stack, keeps raw OpenTelemetry events rather than pre-aggregated summaries, which changes what a p99 spike investigation looks like: the slow 1% are individual rows you can pull up. A query of this shape (illustrative, simplified schema) lists the actual tail requests for one endpoint:

Each row carries a TraceId, so every slow request links to its full distributed trace, which is where the causes above become visible: a GC pause shows as one service stalling, queueing as gaps between spans, a retry as a repeated child span. HyperDX, ClickStack's UI, does this pivot by default, from a p99 chart to the exemplar traces behind it. ClickStack is open source; the getting started guide covers the full schema.