48.9k
Distributed tracing is an observability technique that follows a single request across service boundaries, recording each step as a timed span so the full request path can be reconstructed and analyzed. Every span shares one trace ID, which turns a request that touched a dozen microservices into one connected tree instead of a dozen disconnected log lines. Teams use it to find where latency and errors originate in systems no single log file can explain.
How does distributed tracing work?
Distributed tracing works by generating a unique trace ID when a request enters the system, then passing that ID to every downstream call the request makes. Each unit of work along the way emits a span, a timed record with a name, timestamps, and attributes. Every span records its parent, so a backend can reassemble the spans into one tree.
The mechanics break down into three steps:
- Instrumentation. Each service creates spans around meaningful work: handling an HTTP request, running a database query, calling another service. OpenTelemetry, the CNCF project formed in 2019 by merging OpenTracing and OpenCensus, is the standard way to do it.
- Context propagation. When service A calls service B, it forwards the trace ID and its own span ID in request headers, so B's spans attach to the right parent in the same trace.
- Export and storage. Spans ship over a wire protocol, OTLP in the OpenTelemetry world, to a backend that stores them and reconstructs traces on demand.
Picture one checkout request crossing three services:
Trace 4bf92f3577b34da6a3ce929d0e0e4736
└── POST /checkout [api-gateway] 412 ms
├── charge_payment [payment-svc] 318 ms
│ └── INSERT payments [postgres] 41 ms
└── reserve_stock [inventory-svc] 64 msThe tree shows immediately that charge_payment accounts for most of the 412 ms, an answer a flat log stream cannot give directly.
What is the difference between a trace and a span?
A span is one timed operation in one service: a single HTTP handler, database query, or function call, with a start time, end time, and attributes. A trace is the complete set of spans produced by one request, linked by a shared trace ID into a tree that shows the request's full path and timing.
Per the OpenTelemetry specification, every span carries:
- Name: what the operation is, such as
GET /api/ordersorSELECT orders. - Trace ID: a 128-bit identifier shared by every span in the trace.
- Span ID: a 64-bit identifier unique to this span.
- Parent span ID: the span that caused this one. The root span has none.
- Start and end timestamps: nanosecond-precision bounds that give the span its duration.
- Attributes: key-value pairs like
http.status_code=500ordb.system=postgresql. - Status: whether the operation succeeded or errored.
Strictly, a trace is a directed acyclic graph of spans (OpenTelemetry span links can connect across traces), but for a typical request it behaves as a tree rooted at the entry-point span.
Where did distributed tracing come from?
Distributed tracing descends from Dapper, the internal tracing system Google described in a 2010 research paper that established the trace-tree-of-spans model still in use. Zipkin, Jaeger, and OpenTelemetry are its direct descendants, each generalizing that model toward an open standard.
The Dapper paper remains the canonical read. Twitter open-sourced Zipkin, a Dapper-inspired implementation, in 2012; Uber followed with Jaeger, open-sourced in 2017 and now a CNCF-graduated project. OpenTelemetry consolidated the instrumentation layer in 2019, so the standard today is a vendor-neutral SDK and protocol rather than a per-backend agent. Traces also sit alongside logs and metrics in the classic three-pillars model of observability.
What is trace context propagation?
Trace context propagation is the mechanism that carries a trace's identity across process boundaries, usually as HTTP headers, so spans created in different services join the same trace. Without it, each service would start a new, disconnected trace, and the cross-service tree that makes tracing useful would never form.
The W3C Trace Context specification, a W3C Recommendation since February 2020, standardizes two headers. traceparent packs the version, trace ID, parent span ID, and sampling flags into one value:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01tracestate carries vendor-specific key-value pairs alongside it. Because the format is a W3C standard, a request can cross services instrumented by different teams and still produce one coherent trace.
Distributed tracing vs logging: which do you need?
Logging records discrete events inside a single service; distributed tracing records the path and timing of one request across all services. Most systems at scale need both, correlated by trace ID. If the question spans services, traces answer it; if it concerns one service's internals, logs do.
| Dimension | Logging | Distributed tracing |
|---|---|---|
| Unit of data | One event in one service | One request across all services |
| Structure | Free-form or structured lines | Spans in a tree with timing and causality |
| Best at answering | "What happened inside this service?" | "Where did this request spend its time, and which hop failed?" |
| Cross-service causality | Manual correlation (request IDs, timestamps) | Built in via trace and parent span IDs |
| Typical volume control | Log levels, retention policies | Sampling (head-based or tail-based) |
Stamping the trace ID onto every log line gets you both views of the same request: the trace locates the slow or failing hop, and the logs explain what happened inside it.
What are the challenges of distributed tracing?
The main challenges of distributed tracing are sampling, instrumentation coverage, and storage cost.
- Sampling trade-offs. Head sampling decides at the root span and is cheap but blind to outcomes, while tail sampling decides after the trace completes and can keep every error, at the price of buffering whole traces in memory.
- Instrumentation coverage. An uninstrumented service orphans its downstream spans into separate traces. Auto-instrumentation narrows the gap but rarely closes it.
- Storage and query cost. Retaining spans long enough to debug last week's incident is a database problem. Aggressive sampling also skews tail-latency percentiles, because the rare slow requests are exactly the ones sampled away.
What does trace data look like at rest?
At rest, a span is a wide row: IDs, timestamps, a duration, and key-value attributes. A simplified span in OTLP-style JSON:
{
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"spanId": "00f067aa0ba902b7",
"parentSpanId": "3ce929d0e0e47364",
"name": "charge_payment",
"kind": "SPAN_KIND_CLIENT",
"startTimeUnixNano": "1753660800000000000",
"endTimeUnixNano": "1753660800318000000",
"status": { "code": "STATUS_CODE_OK" },
"attributes": {
"service.name": "payment-svc",
"http.status_code": 200,
"user.tier": "enterprise"
}
}Because spans are uniform rows, trace analysis at scale is aggregation, not tree-walking. "Find the slowest operations by service over the last hour" is a SQL shape:
SELECT ServiceName, SpanName, quantile(0.99)(Duration) AS p99
FROM otel_traces
WHERE Timestamp > now() - INTERVAL 1 HOUR
GROUP BY ServiceName, SpanName
ORDER BY p99 DESC
LIMIT 10Single-trace lookup needs point-read speed; queries like the one above need columnar scan speed over billions of rows. A tracing backend has to do both.
Distributed tracing with ClickStack
ClickStack is the ClickHouse observability stack: the ClickHouse database for storage and query, HyperDX as the UI, and a bundled OpenTelemetry collector for ingestion. It is OTel-native, so the span model described above lands unchanged: spans arrive over OTLP and are stored as wide rows in an otel_traces table, with any attribute queryable without pre-declaring it as a dimension.
That design targets the challenges above: columnar compression makes retaining lightly sampled or unsampled traces cheaper, so deciding which traces matter can happen at query time instead of at ingest. HyperDX renders the same rows as waterfall views and correlates them with logs on the shared trace ID.
FAQ
How does distributed tracing work in microservices?
Each microservice creates spans around its work and forwards the trace context (trace ID plus parent span ID) in headers on every downstream call, typically using the W3C traceparent format. Spans are exported over OTLP to a backend, which joins them by trace ID and reconstructs the cross-service request tree.
How much overhead does distributed tracing add?
Overhead comes from creating spans in-process, propagating headers, and exporting data, and is controlled mainly through sampling. Google's Dapper paper reported keeping production impact negligible by sampling as little as 1 in 1,024 requests at high-throughput services; OpenTelemetry SDKs batch exports off the request path for the same reason.
Do I still need logs if I have distributed tracing?
Yes. Traces show where a request went and where time was spent; logs capture the detail inside each step: stack traces, payload context, decisions taken. The strongest setups correlate the two by writing the trace ID into every log line, so a slow span links directly to the logs it produced.
To see your own traces as queryable rows, try ClickStack, an open-source, OpenTelemetry-native observability stack built on ClickHouse.
