Alert detail page with evaluation history
Demo by @wrn14897
The alerts page currently has a history strip and an error button. It looks useful until you try to answer anything with it.
The error lives on the alert document itself, so the page shows the latest state rather than a real history. You can’t tell whether the alert fired, whether it failed earlier or whether it’s keeping up with its evaluation schedule. This surfaced during work on alert performance, when the existing view turned out to be actively confusing rather than merely sparse.
The new detail page records every evaluation as an event. A timestamp range lets you inspect any window of history, with firing and resolving represented separately and clearly marked.
Grouping is broken out in the table. If an alert uses
GROUP BY, you can open an evaluation and see exactly which groups fired and which didn’t. That distinction matters when only part of the result crosses the threshold. Errors are stored against individual history entries as well, so you can see what failed and when, including the original ClickHouse query error.
The timing columns explain whether an alert is keeping up. Query duration records the time spent running the ClickHouse query. If an alert is scheduled every minute but its query takes three, delays are inevitable rather than mysterious. Webhook duration shows the time spent sending the result to its destination.
Skipped buckets make the resulting backlog visible. A value of seven means seven scheduled evaluation windows were missed and processed later.
Related PRs: #2833 alert evaluations read model and GET /alerts/:id/evaluations, #2834 persist alert evaluation errors and analytics in AlertHistory, #2835 alert detail page with evaluation history
Measuring metric tool adoption in the MCP with evals
Demo by @karl-power
This is the third, and probably final, metrics scenario for the eval framework. It sits deliberately between the other two.
The existing
metric-saturation scenario asks whether an agent can use the metric tools when forced to. The new deploy-regression scenario asks whether it chooses them on its own. A staged rollout of checkout-api pauses after three of six pods. The new build throws a TypeError on fixed-amount promo codes, causing around 7–8% of checkouts to return a 500, but only on the updated pods and only for those codes.
None of this is labelled for the agent. The strongest clue comes from cross-tabbing failed checkouts by pod name, then matching those pods against the rollout event logs. The planted metrics confirm when the failures begin, but they don’t expose the pod split or the underlying defect. An agent can ignore metrics entirely and still solve the scenario. That’s what makes any metric use organic rather than compelled.
There are a couple of traps to stop the path from being too neat. An unrelated rollout lands minutes before the failures begin, while a harmless flood of deprecation warnings rises at the same boundaries as the real errors.
Building the scenario also exposed places where the MCP could do a better job of showing agents which metric kinds and names are available. The resulting changes make both easier to discover.
The improvement is modest, and the comparison is honest about that. Agents already score well without the changes. With them, however, they reach the useful metrics quite a bit faster in several runs. The gap narrows with Fable, which is simply the more capable model here.
One result moved in the opposite direction. Opus scored slightly worse with the metric changes in a couple of runs. That needs more data before anyone explains it away.
For the first time, the eval framework has measured an improvement to how the MCP surfaces metrics. We no longer have to rely entirely on whether a change feels better.
Next comes enough runs to understand the Opus result, followed by a tidy-up of the code behind it.
Related PRs: #2730 add deploy-regression scenario (measure organic metric tool adoption), #2717 harden the metric-saturation scenario, #2694 grade and report metric-tool adoption, #2855 expose summary metrics through the MCP
Distributed tables, histograms and faster trace lookups
Demo by @pulpdrew
A batch of smaller fixes this time, several prompted by feedback from the ClickHouse team. The simplest was a missing clear button in one filter section when every other section already had one. A top-level “clear all” is still on the wish list.
The distributed table case was more involved. Some underlying target tables don’t declare every column exposed by the distributed table. ClickStack runs a
SELECT * when loading full row details, which fails outright against that setup. The row side panel already showed an error, but the expanded row didn’t, and neither explained why ClickStack was issuing a SELECT * in the first place. Both views now surface the error with enough context to make the guidance useful.
Metrics had two separate problems. First, the exponential histogram table was never persisted on metric sources. That didn’t matter until exponential histogram support landed recently. A user opening an existing, incomplete source would see the field filled by schema inference, reasonably assume there was nothing to change, and never save it.
Schema inference no longer runs simply because you opened an existing source. It now runs when you create a metric source or change its database, making it clear that the inferred table still needs to be saved.
The aggregation dropdown also offered average, min, max and other functions for histogram metrics, despite histograms supporting none of them. Choosing one would fail when the query ran or when the tile was saved. Those options are now hidden for histogram and exponential histogram metrics. The MCP’s query_tile path rejects them for persisted tiles as well, matching the UI rather than finding its own creative way to fail.
The series limit fix is subtler. When a GROUP BY produces multiple series, you can set a limit intended to keep the top N by maximum value. In ratio mode, the ranking used only the numerator. That favoured large numerators rather than genuinely high ratios, allowing a series with a large numerator and equally large denominator to displace one that was actually higher. Ranking now uses the ratio being plotted.
Source defaulting on the search page has changed too. It previously selected the first configured source, even if that source contained metrics or sessions. Users could land on an incompatible-source error for a choice they hadn’t made. Search now defaults to the first enabled source it can actually use.
Selecting a trace from a log side panel also had an expensive lookup hiding behind it. HyperDX searched by span and trace ID alone, ignoring the timestamp partition and primary keys. That gets slow on high-volume deployments. The lookup is now bounded by a date range inferred from the source, with a deliberate fallback to an unbounded query when the window misses. A log linked to a span that began several hours earlier is one example where that fallback matters.
Source-name deep links landed the previous week, followed by the entirely fair question of how users were supposed to discover them. The URL parameters accepted by each page were already treated as a contract, so they’re now documented as one. Source filters are the only omission because they remain ClickHouse-only for now. A separate documentation pass added source configuration fields such as span links, covering both recent additions and a few things that had simply been missed.
Related PRs: #2771 improve distributed table SELECT * error state and extend to expanded rows, #2817 only auto-detect metric tables when the database selection changes, #2794 don’t infer metric tables for sources that already have tables (open), #2793 hide unsupported aggregate functions for histogram metrics, #2796 reject persisted histogram tiles with unsupported aggFns in query_tile (open), #2759 use ratio value for series-limit ranking in ratio mode, #2769 prevent search page from defaulting to incompatible source kind, #2816 bound the side panel’s row lookup after View Trace to a time window, #2836 add filter variable configuration
Heatmap percentiles and Lucene search from contributors
Demo by @pulpdrew
Around ten pull requests came in from external contributors this week. Two deserve a special callout.
The first, from @niladrix719, adds percentile context to the heatmap hover tooltip. Instead of eyeballing one cell against the rest of the heatmap, you can now hover and see that, for example, the 26 millisecond bucket sits at the 85th percentile of the durations shown.
The second is a run of Lucene search improvements from @shuvamk.
Unbounded ranges now work properly.
Duration:[* TO 500] becomes a <= 500 predicate instead of asking ClickHouse to convert the string * to a UInt64, which ends about as well as you’d expect. Curly braces are now supported for exclusive range boundaries too.
The escaping fixes matter more because these bugs returned the wrong results rather than an error. Lucene field terms go directly into an ILIKE pattern, where an underscore means any single character and a percent sign means any sequence of characters. A search for ServiceName:user_service was therefore also matching values such as user-service and user.service. Those metacharacters are now escaped before the query reaches ClickHouse.
A separate fix stops map subscripts from being double-escaped in numeric and boolean searches. The generated predicate had been treating the entire expression as one identifier instead of performing a map lookup.
The in-app examples behind the Lucene language switcher have also been updated to cover the new range forms.
Related PRs: #2789 show percentile context in the heatmap hover tooltip, #2779 honor open, exclusive and non-numeric range bounds, #2774 escape LIKE metacharacters in search terms, #2841 escape Map subscripts once in numeric and Bool searches, #2837 add examples for new Lucene syntax
RED metrics on trace search
Demo by @alex-fedotyev
This is exploratory work, with no commitment to ship it.
Trace search currently uses the same single count histogram as log search, coloured by log level. That tells you how many traces you’re looking at, but almost nothing about how they’re performing.
The proposed results view replaces that histogram with RED metrics for trace sources. Throughput appears as bars counting spans. Errors can switch between a percentage rate shown as a line and raw volume shown as bars. Duration plots average, p95 and p99 directly from the source’s raw duration column.
The heatmap is the more interesting view. In the demo, it makes a service with steadily increasing duration obvious almost immediately. It also shows the shape of the latency distribution, which a percentile trend alone can hide.
Cost remains the unresolved part. Trace search already fires plenty of queries for each search, and adding several more aggregations on top will need performance work before this can go any further.
We’d love to get feedback on this behavior.
Related PRs: #2826 show RED metrics on the trace search results view (open, exploratory)
Custom log columns in the ClickHouse Grafana plugin
Demo by @alex-fedotyev
A handful of customers raised the same problem a few weeks ago: the Grafana plugin’s compact log view made additional columns and fields from their logs difficult to see.
This adds a Columns setting to the logs section of the data source configuration. It lives at the data source level rather than on individual queries, so the choice persists for everyone using that source instead of having to be reapplied each time.
You can select any table column. The plugin folds those columns into the log labels under their real names, making them available throughout Grafana. They appear in the Fields list on the left and in the log row details, where a new Fields group sits alongside Resource attributes and Log attributes with the same filter-for and filter-out actions. In the table view, they behave as proper column filters.
All of this is driven by the same query. Without the configuration, those fields appear in none of those places, which was exactly the frustration behind the request.
The reports came from both sides of the schema question. Some customers use OpenTelemetry but add their own columns. Others use entirely custom schemas and keep fields in real columns rather than resource or log attributes, for reasons of their own. Neither group could see those values in Grafana at all.
The change was still in review when this was demonstrated, with the hope of including it in the following week’s plugin build. There’s more context in the ClickHouse Grafana plugin 4.20 post.
Related PRs: grafana/clickhouse-datasource#2108 browse and filter by any log table column (open at the time of the demo)
Capping high cardinality series at the source
Demo by @brandon-pereira
A high-cardinality response can hand a chart hundreds of thousands of rows. Before drawing anything, the client has to convert every row into JSON. On a heavy dashboard, that transformation costs more than the query itself. An unbounded
GROUP BY can also exhaust server memory before the result reaches the browser.
The new approach stops most of those rows from leaving ClickHouse. Queries now include maximum row and maximum grouped-row settings, both currently capped at 5,000. That number is, candidly, a guess at a reasonable ceiling. When the response indicates that a limit was exceeded, the chart warns that the query returned too much data.
This sits on top of an earlier front-end optimisation that caps rendering at 250 series. Holding tens of thousands of series in memory while drawing roughly a hundred lines was pushing browser tabs into several gigabytes and making hovering and panning slow.
Both caps still apply. A pathological GROUP BY now fetches around 5,000 rows and renders 250 of them. Explicit “load all” escape hatches remain for the occasions when you genuinely need everything.
The proper fix for a chart that repeatedly hits either cap is still better SQL: add a limit or make the GROUP BY more selective.
Related PRs: #2802 cap high-cardinality time-chart series with load-all escape hatches, #2856 cap raw-SQL tile cost at the source with a server-side row/cardinality limit (open)
Exemplars, from a metric chart to the trace
Demo by @jordan-simonovski
Exemplars connect an aggregated metric to an individual event, usually a trace. If a latency histogram shows the 99th percentile jumping to 2.4 seconds, an exemplar can point to one real request at 2.4 seconds and open its trace.
When an application records a measurement inside an active span, OpenTelemetry includes it in the normal aggregation. An exemplar filter decides whether that measurement is eligible, then a small reservoir retains a handful of examples to export alongside the aggregated metric point. Each exemplar carries its original value and timestamp, the trace and span IDs, and any attributes dropped from the aggregated stream.
The small reservoir gives you concrete context without exporting every raw measurement or attaching
trace_id and other high-cardinality values to every metric series. An exemplar is still only an example. It isn’t necessarily the worst request or a statistically representative sample. Whether its link resolves also depends on the exporter, the backends involved and whether the referenced trace was retained.
The demo uses a Prometheus backend through the query_exemplars proxy endpoint merged the previous day. If the requested window is too long, the endpoint narrows it instead of rejecting the query.
Test data comes from an OpenTelemetry Collector emitting span metrics. As it processes spans, the collector turns them into metrics with exemplars linked to the trace ID. Those exemplars then appear as markers on the metric chart. Hovering shows the exemplar’s value and timestamp alongside its trace metadata, with a button that opens the trace directly. It works well and has been reasonably fast so far.
Configuration is per tile. Enable exemplars on the chart, then choose the trace source that should resolve the links. The feature currently supports only single-series metrics, and the whole thing remains behind a deployment-level flag in addition to the per-chart toggle.
Testing uncovered a few small bugs, some of which other people had already found independently, but the Prometheus path is essentially ready. ClickHouse comes next. That path will query exemplars directly from the metric table, giving teams that build metrics from trace data the same route back to an individual trace.
Related PRs: #2805 derive request metrics with trace exemplars from spans (open), #2806 add /v1/prometheus/query_exemplars and harden the proxy, #2807 split the two largest chart files into directories, #2808 exemplar overlay for metric and PromQL time charts (open), #2809 accept exemplar settings on API- and agent-authored tiles (open)
Terraform import helpers and batch export
Demo by @jordan-simonovski
Dashboards, saved searches and saved-search alerts now have an Export to Terraform button. It gives teams a route for bringing existing resources under Terraform management through the ClickHouse provider, without hand-writing import blocks or guessing resource type names and ID formats.
The choice that drew questions in the demo was the output itself. The button produces an
import block and leaves the full resource definition to Terraform. Simply adding a resource block doesn’t claim ownership of the existing resource. It can instead try to create another one or overwrite what’s already there. Terraform’s newer import workflow handles that distinction properly.
Paste the generated import block into your configuration, then run terraform plan with -generate-config-out pointing to a file such as generated.tf. Terraform inspects the existing resource and writes the corresponding resource block. Once imported, the resource belongs to Terraform state and future applies manage it rather than attempting to recreate it in ClickStack.
Team settings also includes a batch export that downloads one file covering every supported resource. In the demo, that meant 70 dashboards, around 40 alerts and 55 saved searches. Webhooks and sources are supported as well.
Secrets are deliberately excluded from the V2 API. A GET returns only what the UI already exposes, so a connection includes its host and username but not its password. You supply the missing secret when bringing the generated configuration into use.
Related PRs: #2741 add Terraform import helpers for ClickStack resources
Shared chart components, visual polish and a drafts proposal
Demo by @elizabetdev
Cards in custom dashboards and cards from presets had drifted apart visually, which raised the obvious question: why weren’t they using the same component in the first place? They are now. A shared ChartCard wraps standalone charts in the same primitives used by dashboard tiles, keeping the border, padding and full-bleed header divider in sync without us having to match them by hand. Getting both paths onto one component was more involved than expected, but they now render the same thing internally. There’s still some follow-up work for cards without controls on the right, which need to retain a consistent height.
There’s also been a run of smaller refinements, several aimed squarely at how ClickStack looks in screenshots and demo videos. The segmented control rendered its list line as a border around a zero-height box, causing the top and bottom edges to stack into what looked like a 2px line. That’s now a true 1px edge. The button colours have been reworked for the same reason.
The open source logo in light mode is fixed too. We previously switched between themes using a CSS filter that inverted every colour, including the logo, so light mode showed the wrong brand colour. The same logo works in both themes, as it turns out, so there’s no longer any need to branch on theme.
Clicking rows in search behaves properly again as well. The old behaviour was deliberate, but it didn’t feel deliberate. The drawer could open above the row you clicked, leaving you looking at an apparently unchanged part of the screen. Clicking within the drawer region now updates it, while clicking outside closes it.
We’ve also added semantic Alert components for warning and success states alongside danger. New agent skills steer anything reaching for raw red or warning text towards an Alert, rather than letting it hardcode a Mantine palette colour.
The final part is design exploration only. These are mockups, and feedback is very much wanted.
It began with a narrow problem in the tile editor: a modal opened a drawer, which opened another modal, and a single press of Escape closed the whole stack instead of taking you back one level. From there, the work grew into a broader look at dashboards.
The proposal replaces the saved-versus-temporary dashboard split with drafts. Clicking “New dashboard” creates a private draft that only you can see. When it’s ready, you can save it to the team. You can also move a team dashboard back to drafts or discard it altogether. This covers what temporary dashboards do today without forcing you to make that decision up front. One concept instead of two.
Favourites would get a list view alongside cards, since a page of large cards can push the thing you’re looking for surprisingly far down the screen. Templates would collapse into a single row. Tag filtering would support more than one tag, with sorting by name or last viewed, instead of treating one selected tag as the only available form of organisation.
The tile editor would dock its settings panel on the right instead of opening a drawer. It would also merge the two current routes into settings into a single, predictable path.
Related PRs: #2829 add shared ChartCard component and migrate ChartBox usages, #2814 refine Mantine theme (tabs, code bg, segmented control), #2704 AA-tuned semantic color tokens and Alert/Text variants, #2714 document semantic Alert/Text/danger variants, #2682 close search and session drawers on outside click, #2721 move tile editor to a drawer with docked settings panel (still open). The drafts and favourites redesign has no PR, it is mockups at this stage.