Trace sampling
High-throughput services can produce millions of spans per second. Storing every span is expensive, so teams commonly run the OpenTelemetry Collector's tail-sampling processor to keep only 1-in-N spans. Each kept span carries a SampleRate attribute recording N.
Once data is sampled, naive aggregations are wrong: count() returns N-times fewer events than actually occurred, sum() and avg() are biased, and percentiles shift. Dashboards show misleadingly low request counts, throughput, and error rates.
ClickStack solves this with a sampling-aware query engine. When you configure a sample rate expression on a trace source, the query builder rewrites SQL aggregations to weight each span by its sample rate — across dashboards, alerts, and ad-hoc searches.
How it works
When a trace source has a sampleRateExpression configured, ClickStack wraps it as:
Spans without a SampleRate attribute default to weight 1, so unsampled data produces identical results to the original queries.
The query builder then rewrites aggregations:
| Aggregation | Before | After (sample-corrected) |
|---|---|---|
| count | count() | sum(weight) |
| count + condition | countIf(cond) | sumIf(weight, cond) |
| avg | avg(col) | sum(col * weight) / sum(weight) |
| sum | sum(col) | sum(col * weight) |
| quantile(p) | quantile(p)(col) | quantileTDigestWeighted(p)(col, weight) |
| min / max | unchanged | unchanged |
| count_distinct | unchanged | unchanged |
Percentiles under sampling use quantileTDigestWeighted, an approximate T-Digest sketch. Results are close but not exact.
Configuring the sample rate expression
Open your trace source in Source Settings and enter the ClickHouse expression that evaluates to the per-span sample rate in the Sample Rate Expression field.
For example, if your OpenTelemetry tail-sampling processor writes the rate into SpanAttributes['SampleRate']:
Once configured, all charts, dashboards, alerts, and service dashboard panels automatically apply sample-weighted aggregations. No changes to individual queries are needed.
