LLM inference latency: TTFT, tokens per second, and what to measure

LLM inference latency is the time a large language model takes to produce output after receiving a request, and it is measured with four core metrics: time to first token (TTFT), time per output token (TPOT), tokens per second, and end-to-end latency. OpenTelemetry's GenAI semantic conventions standardize the first two as histogram metrics, gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token.

TL;DR #

• Time to first token (TTFT) measures how long a user waits before anything appears; time per output token (TPOT) measures how fast the response streams after that.
• End-to-end latency decomposes cleanly: TTFT + (output tokens − 1) × TPOT. Long answers are slow even when both component metrics look healthy.
• The prefill phase, which processes the whole prompt before the first token, sets TTFT; TTFT grows with prompt length, so a "good TTFT" depends on input size.
• The OpenTelemetry GenAI semantic conventions define gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token as histograms in seconds; vLLM exposes the equivalent vllm:time_to_first_token_seconds and vllm:time_per_output_token_seconds.
• Latency metrics only make sense as percentiles (p50/p95/p99) over raw request events, never as averages.

What is time to first token? #

Time to first token (TTFT) is the elapsed time between sending a request to an LLM and receiving the first token of its response. It includes network transit, any server-side queueing, and the prefill phase in which the model processes the entire prompt. TTFT determines how quickly a streaming interface appears to respond.

TTFT is the LLM equivalent of time to first byte in web performance: it measures responsiveness, not total speed. Because prefill must ingest every input token before generation starts, TTFT scales with prompt length: a 500-token question and a 100,000-token document produce very different first-token waits on the same model. Inference servers treat it as a first-class signal: vLLM records vllm:time_to_first_token_seconds as a Prometheus histogram with buckets from 1 millisecond up, precisely because the realistic range spans four orders of magnitude.

Which LLM inference latency metrics should you measure? #

The four to measure are TTFT, TPOT (also called inter-token latency), tokens per second, and end-to-end latency. Together they decompose any slow response into "slow to start," "slow to stream," or "simply long."

End-to-end latency is roughly TTFT + (output tokens − 1) × TPOT. A response with a 300 ms TTFT and 30 ms TPOT still takes over 15 seconds if the model writes 500 tokens, which is why per-request tokens per second and output length matter as much as either latency metric. Per-server aggregate throughput is a different quantity; batching trades it against per-request TPOT.

Why does TTFT dominate perceived responsiveness? #

TTFT dominates perceived responsiveness because streaming UIs hide generation time behind reading time. Once tokens flow faster than a person reads, the user never waits on the model again; the only wait they consciously experience is the silence before the first token appears. Optimizing TTFT therefore buys more perceived speed than raising throughput.

Adults silently read non-fiction at about 238 words per minute, roughly 4 words or 5–6 tokens per second, per Brysbaert's 2019 meta-analysis. A stream sustaining 20 tokens per second outpaces nearly every reader, so cutting TPOT further is imperceptible in a chat UI. Meanwhile the pre-stream silence is governed by classic response-time thresholds: Nielsen's limits put the boundary for an uninterrupted flow of thought at about 1 second.

The calculus inverts for non-interactive consumers. An AI agent doesn't read as it goes; each step blocks on the full completion, so end-to-end latency (and TPOT across long outputs) dominates, and TTFT barely matters.

What happens between the request and the first token? #

LLM inference runs in two phases. Prefill processes every prompt token in parallel and builds the KV cache, a stored set of attention keys and values for reuse; it is compute-bound and sets TTFT. Decode then generates one token per step, reading the whole cache each time; it is memory-bandwidth-bound and sets TPOT.

A streamed completion looks like this on a timeline:

1t=0        client sends request
2t=0–50ms   network + server queue (waiting for a batch slot)
3t=50ms     prefill: all 3,200 prompt tokens processed, KV cache built
4t=400ms    ── first token streamed  ← TTFT = 400 ms
5t=400ms+   decode: one token per step, ~30 ms each  ← TPOT
6t=12.7s    final token (410 tokens out)  ← end-to-end latency

Serving optimizations act on specific segments. Continuous batching, introduced by the Orca system at OSDI 2022, admits new requests at token granularity instead of waiting for a whole batch to finish, cutting queue time. PagedAttention (SOSP 2023), the technique behind vLLM, manages KV-cache memory in blocks so more requests fit per GPU. Quantization shrinks weights to fewer bits, easing the memory-bandwidth ceiling on decode and lowering TPOT. Prefix caching reuses the KV cache for shared prompt prefixes, collapsing prefill cost for repeated system prompts.

Why is my LLM response so slow? #

A slow LLM response usually comes down to prefill, a cold start, queueing, overhead outside the model, or simply a long output, and measuring TTFT and TPOT separately tells you which one you have.

Triage in this order:

• High TTFT, prompt is long → prefill. Trim context, summarize retrieval results, or use prefix caching for static system prompts.
• High TTFT on the first request only → cold start. Model weights loading onto the GPU can take tens of seconds; keep instances warm.
• High TTFT under load, fine when idle → queueing. Requests are waiting for batch capacity; add replicas or admission control.
• TTFT fine, total time high, TPOT normal → output length. The model is writing an essay; tighten max_tokens and prompts.
• Model-side timings fine, client-side slow → network, proxies, or an orchestration step (retrieval, guardrail checks) before the model call. Only a trace across the whole request reveals this.

How do you measure LLM inference latency? #

LLM inference latency is measured by wrapping each model call in an OpenTelemetry span, recording the first-token moment as a span event or attribute, and computing TTFT and TPOT percentiles over the resulting request events. The GenAI semantic conventions define the standard metric names; inference servers like vLLM export equivalent histograms.

The OpenTelemetry GenAI semantic conventions (Development status) specify gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token on the serving side, and gen_ai.client.operation.time_to_first_chunk on the caller side. Capturing at the span level preserves each request's full context for later slicing. A captured span looks like this:

1{
2  "name": "chat gpt-4o",
3  "attributes": {
4    "gen_ai.operation.name": "chat",
5    "gen_ai.request.model": "gpt-4o",
6    "gen_ai.usage.input_tokens": 3200,
7    "gen_ai.usage.output_tokens": 410,
8    "llm.ttft_ms": 400
9  }
10}

Averages are the wrong summary for these distributions: latency tails are where users suffer, as Dean and Barroso's The Tail at Scale (CACM, 2013) established for distributed systems generally. Dashboards should track p50, p95, and p99 rather than a mean, segmented by model, prompt version, and input-token bucket, since a TTFT regression often turns out to be a prompt that grew.

How Langfuse handles LLM latency telemetry #

Langfuse, the open-source LLM observability platform that ClickHouse acquired in January 2026, records each model call as a generation within a trace: model, prompt version, token counts, time to first token for streamed responses, and total duration in one record, rather than as pre-aggregated histograms. Its dashboards chart latency percentiles segmented by model and prompt version, which is exactly the slicing described above.

Storing raw events is what makes latency triage segmentable after the fact. A pre-bucketed histogram can answer "what is p95 TTFT?" but not "what is p95 TTFT for prompt version B on requests over 8,000 input tokens?" Because Langfuse keeps every call as a raw event in ClickHouse, that question is a filter over existing data, not a new instrumentation decision.

The same traces serve the surrounding LLM observability questions: cost attribution, prompt debugging, and agent runs. To see your own TTFT distribution, instrument the application with the Langfuse SDK and chart the percentiles in its dashboards; Langfuse is open source and free to self-host.