Event Deltas with ClickStack

Event Deltas in ClickStack are a trace-focused feature that automatically analyzes the properties of traces to uncover what changed when performance regresses. By comparing the latency distributions of normal versus slow traces within a corpus, ClickStack highlights which attributes are most correlated with the difference - whether that's a new deployment version, a specific endpoint, or a particular user ID.

Instead of manually sifting through trace data, event deltas surface the key properties driving differences in latency between two subsets of data, making it far easier to diagnose regressions and pinpoint root causes. This feature allows you to visualize raw traces and immediately see the factors influencing performance shifts, accelerating incident response, and reducing mean time to resolution.

Using Event Deltas

Event Deltas are available directly through the Search panel in ClickStack when selecting a source of type Trace.

From the top-left Analysis Mode selector, choose Event Deltas (with a Trace source selected) to switch from the standard results table, which displays spans as rows.

This view renders the distribution of spans over time, showing how latency varies alongside volume. The vertical axis represents latency, while the coloring indicates the density of traces at a given point with brighter yellow areas corresponding to a higher concentration of traces. With this visualization, users can quickly see how spans are distributed across both latency and count, making it easier to identify shifts or anomalies in performance.

Users can then select an area of the visualization - ideally one with higher duration spans and sufficient density, followed by Filter by Selection. This designates the "outliers" for analysis. Event Deltas will then identify the columns and key values most associated with those spans in this outlier subset compared to the rest of the dataset. By focusing on regions with meaningful outliers, ClickStack highlights the unique values that distinguish this subset from the overall corpus, surfacing the attributes most correlated with the observed performance differences.

For each column, ClickStack identifies values that are heavily biased toward the selected outlier subset. In other words, when a value appears in a column, if it occurs predominantly within the outliers rather than the overall dataset (the inliers), it is highlighted as significant. Columns with the strongest bias are listed first, surfacing the attributes most strongly associated with anomalous spans and distinguishing them from baseline behavior.

Consider the example above where the SpanAttributes.app.payment.card_type column has been surfaced. Here, the Event Deltas analysis shows that 29% of the inliers use MasterCard, with 0% among the outliers, while 100% of the outliers use Visa, compared to 71% of the inliers. This suggests that the Visa card type is strongly associated with the anomalous, higher-latency traces, whereas MasterCard appears only in the normal subset.

Conversely, values exclusively associated with inliers can also be interesting. In the example above, the error Visa Cash Full appears exclusively in the inliers and is completely absent from the outlier spans. Where this occurs, latency is always less than approximately 50 milliseconds, suggesting this error is associated with low latencies.

How Event Deltas work

Event Deltas work by issuing two queries: one for the selected outlier area and one for the inlier area. Each query is limited to the appropriate duration and time window. A sample of events from both result sets is then inspected, and columns for which a high concentration of values appears predominantly in the outliers are identified. Columns for which 100% of a value occurs only in the outlier subset are shown first, highlighting the attributes most responsible for the observed differences.

Recommendations

Event Deltas work best when the analysis is focused on a specific service. Latency across multiple services can vary widely, making it harder to identify the columns and values most responsible for outliers. Before enabling Event Deltas, filter spans to a set where the distribution of latencies is expected to be similar. Target analyzing sets where wide latency variation is unexpected for the most useful insights, avoiding cases where it's the norm (e.g., two different services).

When selecting an area, users should aim for subsets where there is a clear distribution of slower versus faster durations, allowing the higher-latency spans to be cleanly isolated for analysis. For example, note the selected area below clearly captures a set of slower spans for analysis.

Conversely, the following dataset is hard to analyze in a useful way with Event Deltas.