## Volume and cardinality became the hard limits

In 2025, it became clear that volume alone was no longer the primary challenge in observability. While data volumes continued to grow, high cardinality emerged as the more fundamental constraint. Applications now emit more telemetry than ever, but with far greater dimensionality, turning what was once an edge case into the default.

AI workloads accelerated this shift. Model versions, prompts, GPU IDs, shards, tenants, and execution graphs all introduced new labels, compounding cardinality across every signal. Large-scale training and inference systems, often spanning thousands of GPUs, multiplied telemetry across hardware, network, scheduling, and software layers. The result was an explosion in unique combinations that observability systems were expected to store and query interactively.

This level of cardinality exposed the limits of many existing platforms. Storage systems were no longer just required to ingest data, but to aggregate and query it efficiently at scale. In practice, high cardinality became a huge cost driver for observability users. It’s clear that existing observability data stores are well past their limits and the industry needs to rethink how to scale to current user demands, without requiring users to lose data through sampling and stripping out cardinality.

## Tracing became the dominant signal

Over the past year, tracing moved from an advanced feature to the primary way engineers reason about complex systems. Asynchronous workflows, queues, streaming pipelines, and agent-based execution have blurred traditional request boundaries. At the same time, AI workloads introduced non-determinism and fan-out patterns that metrics alone could no longer summarize in a meaningful way.

Traces provided the structure that logs and metrics could only hint at. Increasingly, we saw teams consolidate metrics onto traces, using spans as the organizing unit for understanding system behavior. This shift placed new demands on observability platforms, stressing assumptions that were never designed for trace-first workloads.

Span counts per trace continued to grow, often dramatically - we regularly engage users with 10k spans per trace. Many teams were forced to rely on aggressive sampling just to keep systems usable. The difference became clear between platforms built with tracing as a first-class concern and those where tracing was added later as an afterthought.

## OpenTelemetry fully crossed the chasm

By the time we started ClickStack in mid-2025, we made a deliberate decision to ensure that HyperDX, the UI behind ClickStack, could work with any schema. That flexibility still has real value. In practice, though, we increasingly see users standardizing on OpenTelemetry, with alternative schemas and competing specifications falling away.

New services are now instrumented with OpenTelemetry by default, and legacy systems are steadily being retrofitted. Value is emerging in areas like fast, opinionated instrumentation and data pipelines, with companies such as [Bindplane](https://bindplane.com/) and [Odigos](https://odigos.io/), along with newer projects like [Rotel](https://streamfold.com/), standing out as particularly compelling.

However, OpenTelemetry still has challenges around its adoption - with complex configurations required out of the box, in-depth concepts that are difficult to learn on the fly, and standards and best practices that continue to evolve with the project. This is an area our team will help contribute to in the coming year to continue advancing vendor-neutral standards in observability.

## Open Source adoption accelerated through real migrations

Since its launch last year, we’ve seen ClickStack adopted by teams migrating from existing observability platforms, rather than purely through greenfield experimentation. That was somewhat unexpected. Open source projects often see early traction from new workloads, with migrations following later. In observability, the pattern appears to have reversed.

The reasons were pragmatic rather than ideological. As tracing volumes and retention periods grew, SaaS pricing models have become harder to justify. Open source offered cost predictability, data ownership, and relief from opaque billing, making it an increasingly attractive option. For many teams, these factors outweighed the appeal of a perfectly polished, all-in-one user experience.

At higher data volumes, priorities have shifted. Flexibility in querying and the ability to scale cost-effectively mattered more than feature completeness. Teams were willing to accept trade-offs, focusing on strong coverage of core workflows rather than comprehensive but expensive solutions. Running open source observability systems does require work, but for many users in 2025, that effort proved preferable to the costs and constraints of proprietary platforms - especially for those at larger scale.

## AI SRE advanced, but is still pre-breakout

Throughout 2025, we engaged with a growing number of vendors promising some form of “agentic SRE.” Companies like [traversal.ai](https://traversal.com) and [resolve.ai](https://resolve.ai) are clearly pushing the space forward, joined by both established players such as [Wild Moose](https://www.wildmoose.ai/) and a wave of new entrants. The pace of experimentation has been impressive.

In our experience, what worked best were co-pilot-style workflows focused on summarization, correlation, and guided investigation, while a human stayed in the loop. These interactions felt meaningfully useful rather than gimmicky and robust in production (vs just a great demo) - [our own internal testing largely reinforced](https://clickhouse.com/blog/llm-observability-challenge) that impression. 

There is no clear winner yet. Most tools still fall well short of the reasoning depth and reliability seen in mature code-focused assistants. The bar to enter the market remains relatively low, but the gaps show quickly: shallow reasoning, fragile context windows, and limited actionability. These systems can think faster than humans, but they do not replace human judgment. The ceiling is clearly visible. While we saw encouraging progress in 2025, it has not been broken yet. 2026 promises to be an important year for this space.

## Data quality emerged as a first-class question

For years, many teams spent most of their energy simply getting observability pipelines to work end to end. By 2025, that work had become significantly easier. OpenTelemetry, along with a maturing ecosystem of agents and pipeline tools, standardized instrumentation and transport and made reliable delivery far more attainable.

That maturity didn’t mean data quality suddenly became important. Some teams had been wrestling with it for years. What changed in 2025 was the scale and urgency of the problem. As observability became easier to adopt, teams found themselves collecting far more telemetry than they could realistically use, while still paying to store and query most of it. Cost pressure made this impossible to ignore. As a result, the conversation shifted. Data quality became more nuanced and, in many ways, harder to answer. Teams stopped asking whether they were missing data and started asking different questions. Can we actually use this telemetry? Is it actionable? Does it help us reach the root cause faster and make better decisions? Most importantly, which signals are worth their cost?

This shift went well beyond schema validation or ingestion correctness. It forced teams to think about usefulness, redundancy, and the cognitive load required to interpret their data. While a few organizations made real progress on this in 2025, the same questions surfaced repeatedly in our conversations. This tension between signal, cost, and usability feels set to become the defining observability battleground of 2026.

## My final take

For me, 2025 was the year observability collided with scale, complexity, and AI. Growing volume and cardinality forced vendors toward architectural honesty. Tracing became unavoidable and emerged as the dominant signal. OpenTelemetry solidified the ecosystem, while open source options expanded well beyond the traditional ELK and LGTM stacks, becoming both more diverse and far more scalable.

AI-driven SRE took its first real steps forward, but it did not yet break through into a standard, production-ready approach. Deployments still required deep vendor involvement and remained difficult to adopt without significant effort. At the same time, data quality finally became a first-class conversation, shifting focus away from simply collecting telemetry toward understanding which signals are actually useful.


## Introduction

With the start of the new year, it feels like the right moment to pause and reflect on a year that was pivotal for observability at ClickHouse and for me personally. As Head of Product Management for Observability, my focus throughout 2025 was building ClickStack and working closely with customers, partners, and the broader community to rethink what scalable observability should look like.

In May 2025, we introduced ClickStack with a simple goal: make high-performance observability on ClickHouse accessible to everyone. Until then, teams either built their own user interfaces on top of ClickHouse or relied on more generic visualization tools like Grafana. By pairing ClickHouse with the HyperDX UI, ClickStack removes that friction and allows teams to immediately benefit from ClickHouse’s compression and fast query execution optimized for logs, traces, and metrics.

For a detailed reflection on what 2025 looked like for ClickStack itself, [see our earlier blog](https://clickhouse.com/blog/clickstack-a-year-in-review-2025). 

Here, I want to focus instead on a few broader observations about how the observability landscape shifted over the past year, shaped by my conversations with our users, time spent at conferences, and ongoing discussions across the observability community. 


Explore the ClickHouse-powered open source observability stack built for OpenTelemetry at scale.



Learn about ClickStack

Mike Shi, Head of Observability at ClickHouse, reflects on a pivotal year building ClickStack and learning from users. From cardinality and trace-first systems to OpenTelemetry and AI SRE, he shares what changed in 2025 and what matters next.

large_o11y-year-in-review.png

medium_o11y-year-in-review.png

small_o11y-year-in-review.png

thumbnail_o11y-year-in-review.png

o11y-year-in-review.png

Observability - a year in review

Welcome to the November edition of What’s New in ClickStack, the open-source observability stack built for ClickHouse. Each month, we highlight new ClickHouse features and HyperDX UI improvements that work together to make observability faster, easier to use, and more capable than ever.

This release introduces Service Maps, integration with incident.io, root span filtering, searching within traces, line chart comparisons, and new controls for highlighting specific attributes.

## New contributors

Building an open-source observability stack is a team sport - and our community makes it possible. A big thank you to this month's new contributors! Every contribution, big or small, helps make ClickStack better for everyone.

**[jwhitaker-gridcog](https://github.com/jwhitaker-gridcog)**
**[alok87 ](https://github.com/alok87)**
**[hiasr](https://github.com/hiasr)**
**[dhtclk](https://github.com/dhtclk)**
**[beefancohen](https://github.com/beefancohen)**


## Service Maps

Service Maps are now available in beta, bringing one of the most requested features in ClickStack to life. Service Maps give teams a high-level view of how their services interact, showing the flow of requests between components and surfacing traffic patterns and failures. They help turn raw traces into an intuitive picture of system behavior, making it easier to understand dependencies and spot issues across distributed architectures.

![image10.png](/uploads/image10_c88d9a0ffc.png)

In ClickStack, we always prefer to present features in context rather than just as isolated screens, so you’ll find Service Maps integrated throughout the ClickStack experience. Although on the left panel you can explore your full service graph to see how everything connects, you’ll also encounter Service Maps in other contexts - for example, when viewing an individual trace. Next to the columns in the trace waterfall, a focused map appears, showing how that specific request moved between services, giving you a visual representation of the path without breaking your investigative flow.

service_map.mp4

Since Service Maps are launching in early beta, the initial release focuses on traffic and error visualization. We also sample data to ensure performance at scale. For those interested in how this works under the hood, the query powering the map links server spans and their corresponding client spans within the same trace using a sampled dataset. This lets us infer directional edges between services, calculate request counts, and highlight failed calls while keeping computation manageable. For the SQL enthusiasts:

<pre><code  run='false'   type='click-ui'   language='sql'  runnable='true'   clickhouse_settings='{"enable_parallel_replicas": 0}'   play_link='https://sql.clickhouse.com?query=V0lUSAogICAgbm93NjQoMykgQVMgdHNfdG8sCiAgICB0c190byAtIElOVEVSVkFMIDkwMCBTRUNPTkQgQVMgdHNfZnJvbSwKICAgIFNlcnZlclNwYW5zIEFTCiAgICAoCiAgICAgICAgU0VMRUNUCiAgICAgICAgICAgIFRyYWNlSWQgQVMgdHJhY2VJZCwKICAgICAgICAgICAgU3BhbklkIEFTIHNwYW5JZCwKICAgICAgICAgICAgU2VydmljZU5hbWUgQVMgc2VydmljZU5hbWUsCiAgICAgICAgICAgIFBhcmVudFNwYW5JZCBBUyBwYXJlbnRTcGFuSWQsCiAgICAgICAgICAgIFN0YXR1c0NvZGUgQVMgc3RhdHVzQ29kZQogICAgICAgIEZST00gb3RlbF92Mi5vdGVsX3RyYWNlcwogICAgICAgIFdIRVJFICgoVGltZXN0YW1wID49IHRzX2Zyb20pIEFORCAoVGltZXN0YW1wIDw9IHRzX3RvKSkgQU5EICgoKGNpdHlIYXNoNjQoVHJhY2VJZCkgJSAxMCkgPSAwKSBBTkQgKFNwYW5LaW5kIElOICgnU2VydmVyJywgJ0NvbnN1bWVyJykpKQogICAgKSwKICAgIENsaWVudFNwYW5zIEFTCiAgICAoCiAgICAgICAgU0VMRUNUCiAgICAgICAgICAgIFRyYWNlSWQgQVMgdHJhY2VJZCwKICAgICAgICAgICAgU3BhbklkIEFTIHNwYW5JZCwKICAgICAgICAgICAgU2VydmljZU5hbWUgQVMgc2VydmljZU5hbWUsCiAgICAgICAgICAgIFBhcmVudFNwYW5JZCBBUyBwYXJlbnRTcGFuSWQsCiAgICAgICAgICAgIFN0YXR1c0NvZGUgQVMgc3RhdHVzQ29kZQogICAgICAgIEZST00gb3RlbF92Mi5vdGVsX3RyYWNlcwogICAgICAgIFdIRVJFICgoVGltZXN0YW1wID49IHRzX2Zyb20pIEFORCAoVGltZXN0YW1wIDw9IHRzX3RvKSkgQU5EICgoKGNpdHlIYXNoNjQoVHJhY2VJZCkgJSAxMCkgPSAwKSBBTkQgKFNwYW5LaW5kIElOICgnQ2xpZW50JywgJ1Byb2R1Y2VyJykpKQogICAgKQpTRUxFQ1QKICAgIFNlcnZlclNwYW5zLnNlcnZpY2VOYW1lIEFTIHNlcnZlclNlcnZpY2VOYW1lLAogICAgU2VydmVyU3BhbnMuc3RhdHVzQ29kZSBBUyBzZXJ2ZXJTdGF0dXNDb2RlLAogICAgQ2xpZW50U3BhbnMuc2VydmljZU5hbWUgQVMgY2xpZW50U2VydmljZU5hbWUsCiAgICBjb3VudCgqKSAqIDEwIEFTIHJlcXVlc3RDb3VudApGUk9NIFNlcnZlclNwYW5zCkxFRlQgSk9JTiBDbGllbnRTcGFucyBPTiAoU2VydmVyU3BhbnMudHJhY2VJZCA9IENsaWVudFNwYW5zLnRyYWNlSWQpIEFORCAoU2VydmVyU3BhbnMucGFyZW50U3BhbklkID0gQ2xpZW50U3BhbnMuc3BhbklkKQpXSEVSRSAoQ2xpZW50U3BhbnMuc2VydmljZU5hbWUgSVMgTlVMTCkgT1IgKFNlcnZlclNwYW5zLnNlcnZpY2VOYW1lICE9IENsaWVudFNwYW5zLnNlcnZpY2VOYW1lKQpHUk9VUCBCWQogICAgc2VydmVyU2VydmljZU5hbWUsCiAgICBzZXJ2ZXJTdGF0dXNDb2RlLAogICAgY2xpZW50U2VydmljZU5hbWUKT1JERVIgQlkKICAgIHNlcnZlclNlcnZpY2VOYW1lIEFTQywKICAgIHNlcnZlclN0YXR1c0NvZGUgQVNDLAogICAgY2xpZW50U2VydmljZU5hbWUgQVNDOw&chart=eyJ0eXBlIjoibGluZSIsImNvbmZpZyI6eyJ4YXhpcyI6InllYXIiLCJ5YXhpcyI6Ijk5dGhfYXZnX3RlbXAiLCJzZXJpZXMiOiJjb3VudHJ5IiwidGl0bGUiOiJUZW1wZXJhdHVyZSBieSBjb3VudHJ5IGFuZCB5ZWFyIn19'   show_statistics='true' 

>
WITH
    now64(3) AS ts_to,
    ts_to - INTERVAL 900 SECOND AS ts_from,
    ServerSpans AS
    (
        SELECT
            TraceId AS traceId,
            SpanId AS spanId,
            ServiceName AS serviceName,
            ParentSpanId AS parentSpanId,
            StatusCode AS statusCode
        FROM otel_v2.otel_traces
        WHERE ((Timestamp >= ts_from) AND (Timestamp <= ts_to)) AND (((cityHash64(TraceId) % 10) = 0) AND (SpanKind IN ('Server', 'Consumer')))
    ),
    ClientSpans AS
    (
        SELECT
            TraceId AS traceId,
            SpanId AS spanId,
            ServiceName AS serviceName,
            ParentSpanId AS parentSpanId,
            StatusCode AS statusCode
        FROM otel_v2.otel_traces
        WHERE ((Timestamp >= ts_from) AND (Timestamp <= ts_to)) AND (((cityHash64(TraceId) % 10) = 0) AND (SpanKind IN ('Client', 'Producer')))
    )
SELECT
    ServerSpans.serviceName AS serverServiceName,
    ServerSpans.statusCode AS serverStatusCode,
    ClientSpans.serviceName AS clientServiceName,
    count(*) * 10 AS requestCount
FROM ServerSpans
LEFT JOIN ClientSpans ON (ServerSpans.traceId = ClientSpans.traceId) AND (ServerSpans.parentSpanId = ClientSpans.spanId)
WHERE (ClientSpans.serviceName IS NULL) OR (ServerSpans.serviceName != ClientSpans.serviceName)
GROUP BY
    serverServiceName,
    serverStatusCode,
    clientServiceName
ORDER BY
    serverServiceName ASC,
    serverStatusCode ASC,
    clientServiceName ASC;
</code></pre>

## Root span filtering

Root span filtering is a simple but elegant improvement to the trace search experience. Historically, ClickStack returned every matching span in search results, whether it was a root span or a child span. This offers maximum flexibility and ensures complete coverage of your data, but it also means that a single trace can appear multiple times when several client spans match the query. For users navigating large volumes of traces, this often made browsing and triage more tricky.

The latest release introduces the option to filter search results to root spans only. By selecting the root span filter in the left navigation, users can limit results to top-level spans, making it far easier to scan, compare, and locate the traces that matter. It’s a small change that brings a meaningful improvement to trace navigation, especially in high-cardinality environments.

![image3.png](/uploads/image3_3985f9dc68.png)

If root span searches become a common access pattern for you, we recommend adjusting your primary key to optimize for this workflow. Since root spans represent a much smaller subset of the overall trace data, indexing them directly can significantly reduce the amount of data scanned and materially improve query latency. It’s not unusual for a single trace to contain thousands of spans, so narrowing the search space can have a major impact. For example, a typical primary key might look like:

```sql
ORDER BY (toStartOfMinute(Timestamp), StatusCode, flow, country, ServiceName, Timestamp)
```

When optimizing specifically for root span filtering, introducing empty(ParentSpanId) into the key can accelerate these lookups:


```sql
ORDER BY (toStartOfMinute(Timestamp), empty(ParentSpanId), StatusCode, flow, country, ServiceName, Timestamp)
```

## Attribute highlighting

As we work with users, we gain a clearer understanding of the workflows that matter most to them. One frequent request has been the ability to surface specific attributes directly in search results. These attributes need to be configurable per source, clickable for quick filtering, and easy to add to the current search context. Users also want these attributes to be visible when inspecting a trace, not only for the selected span but also whenever they appear anywhere in the trace. The latest version of ClickStack introduces two new concepts to support these needs.

The first is **Highlighted Attributes**. These can be configured per source, allowing users to define expressions that extract log or span-level fields to show in the row side panel. Each highlighted attribute supports an alias for readability as well as an optional expression that defines how it should be searched if the user selects it - if not specified, the column name will be used. This gives users full control over how important fields are displayed and how they can be queried. Below, we include two practical examples of highlighted attributes demonstrating how to extract values, assign aliases, and specify search expressions.

![image11.png](/uploads/image11_a6fb0760c5.png)

*> For this log source example, we extract the pod and node names from the resource attribute map, assigning them the aliases Pod and Node. Each attribute also includes an optional Lucene expression - if the user clicks a value to initiate a search, this syntax is used instead of SQL.*

When viewing search results, these highlighted attributes appear directly in the row-side panel, giving immediate visibility into key fields without expanding the full span or log entry. Clicking any of these values applies the associated search expression, letting users filter or refine their query with a single interaction. 

resource-attributes.mp4

The second feature is **Highlighted Trace Attributes**, which are also configured at the source level. These define fields for logs and traces that appear at the top of the trace view, and are rendered if they appear anywhere in the associated trace. Like Highlighted Attributes, these can also be used to construct links to external systems or surfacing important computed values. 

In the examples below, we extract fields from the ClickPy demo traces dataset. This dataset represents the traces captured by our [public demo ClickPy](https://clickpy.clickhouse.com), which provides free analytics over Python PyPI downloads data.

![image2.png](/uploads/image2_31a676665d.png)

The first definition constructs a hyperlink back to the ClickPy application for the package for which it is associated. This becomes clickable when detected. Another extracts execution time from any database spans and surfaces it prominently at the top of the waterfall, so that users can quickly understand performance characteristics.

![image6.png](/uploads/image6_c99e85f911.png)

When reviewing the trace associated with a log or span, these attributes are displayed above the waterfall. In the example above, all database response times from spans in the trace are surfaced, along with a generated link back to ClickPy.

## [Incident.io](http://Incident.io) integration

Following last month’s introduction of alerting, we’re continuing to make it easier for teams to route notifications to the tools they rely on. This release adds native support for sending alerts directly to incident.io, removing the need to configure raw webhook endpoints. Over the coming months, we plan to further enrich alert capabilities and streamline configuration so teams can set up meaningful and actionable alerts with minimal effort.

<img src="/uploads/image7_68e9e72b60.png" alt="image7.png" width="70%">

## Search within traces

A core principle in ClickStack is the ability to search your data at any point in the observability workflow. The latest release extends this further by adding full Lucene search capabilities directly to the trace waterfall view. Since the waterfall contains both spans and logs, often coming from different sources, we provide a separate search input for each source. This gives users a fast way to filter, refine, and explore the events that make up a trace without leaving the context of the waterfall itself.

![image5.png](/uploads/image5_eb1da198de.png)

## Line chart comparisons

When investigating system behavior over time, it’s common to compare current performance with how things looked in the past. Many tools rely on users visually inspecting a single line and mentally contrasting it with previous behavior. While this works, we wanted to make the process far easier. One of the features we’re excited to introduce in the November release is period comparison for line charts.

With this feature, users can chart a line for their selected time range and then enable a simple toggle to overlay the previous period. The previous period is defined as a date range of equal length, immediately preceding the current range. It appears on the same axis as a dashed line, allowing users to quickly spot changes, regressions, or improvements without manual calculations or separate charts. An example is shown below.

line-chart-comparisons.mp4

We'd love to get your feedback on this feature and all of the others released this month. 

Discover the world’s fastest and most scalable open source observability stack, in seconds.

Get started with ClickStack

Discover what’s new in ClickStack this month, including service maps, root span filtering, Incident.io integration, and more!

large_image9.png

medium_image9.png

small_image9.png

thumbnail_image9.png

image9.png

What's new in ClickStack. November '25.

When selected, ClickStack begins analyzing the matching data to identify recurring message structures. Importantly, it doesn’t attempt to cluster every single result - doing so would be computationally expensive and unnecessary. Instead, ClickStack samples a representative subset of the data, ensuring that the sample is large enough to be statistically meaningful and evenly distributed across the result set.

Once the analysis completes, you’re presented with a condensed view of your events grouped by similarity. Each cluster corresponds to a distinct log or event pattern, allowing you to instantly see which types of messages dominate the sample and which rare ones might warrant deeper investigation.

In the example above, we begin by filtering down to the **payment service**, then apply an **error filter**. Within seconds, event patterns surface an interesting and recurring error - something that would have taken far longer to spot by scrolling through raw logs or iteratively refining search queries.

## How ClickStack implements event patterns

Under the hood, event clustering in ClickStack is powered by [Drain3](https://github.com/logpai/Drain3), an open-source, high-performance log template mining library. Drain3 is an **online log clustering algorithm** that efficiently extracts templates, or “patterns”, from streams of log messages. It uses a fixed-depth parse tree to guide how log groups are formed, avoiding the deep and unbalanced structures that can make other algorithms slow or memory-heavy. This approach makes it fast enough to operate interactively, even on large log datasets.

We made a deliberate decision to integrate Drain3 **at query time** rather than at insert time. This design gives users the flexibility to discover and analyze patterns dynamically within any subset of their data, directly complementing the search experience. Running clustering at ingest would allow pre-tagging of log patterns but would also introduce heavy overhead - especially at ClickStack’s ingestion scales of gigabytes per second across petabytes of data. Performing the analysis on demand keeps ingestion lightweight and ensures the insights are always contextual and up to date.

Because Drain3 is written in Python, we use **Pyodide** to execute it directly in the browser. Pyodide is a full Python runtime compiled to WebAssembly that runs entirely client-side, with no backend processing required. It supports much of the Python standard library and key data science packages such as numpy, pandas, and matplotlib. Using Pyodide allows ClickStack to run Drain3 seamlessly within the browser environment while still interacting with JavaScript for visualization and UI updates.

Even with a lightweight algorithm like Drain3, running clustering over billions of rows at query time would be impractical. To balance accuracy and performance, ClickStack applies **sampling**, selecting a statistically representative subset of 10,000 events for analysis with the aim of returning clusters within a few seconds. This sampling is random but evenly distributed to ensure coverage across the query’s time range and services.

For example, when analyzing logs in the above example, ClickStack might issue a query like:
<pre><code

  run='false'   type='click-ui'   language='sql'  runnable='true'   clickhouse_settings='{"enable_parallel_replicas": 0}'   play_link='https://sql.clickhouse.com?query=V0lUSAogICAgbm93NjQoMykgQVMgdHNfdG8sCiAgICB0c190byAtIElOVEVSVkFMIDkwMCBTRUNPTkQgQVMgdHNfZnJvbSwKICAgIHRhYmxlU3RhdHMgQVMKICAgICgKICAgICAgICBTRUxFQ1QgY291bnQoKSBBUyB0b3RhbAogICAgICAgIEZST00gb3RlbF92Mi5vdGVsX2xvZ3MKICAgICAgICBXSEVSRSBUaW1lc3RhbXBUaW1lID49IHRzX2Zyb20KICAgICAgICAgIEFORCBUaW1lc3RhbXBUaW1lIDw9IHRzX3RvCiAgICApClNFTEVDVAogICAgQm9keSBBUyBfX2hkeF9wYXR0ZXJuX2ZpZWxkLAogICAgVGltZXN0YW1wVGltZSBBUyBfX2hkeF90aW1lc3RhbXAsCiAgICBTZXZlcml0eVRleHQgQVMgX19oZHhfc2V2ZXJpdHlfdGV4dCwKICAgIFRpbWVzdGFtcFRpbWUsCiAgICBTZXJ2aWNlTmFtZQpGUk9NIG90ZWxfdjIub3RlbF9sb2dzCldIRVJFCiAgICBUaW1lc3RhbXBUaW1lID49IHRzX2Zyb20KICAgIEFORCBUaW1lc3RhbXBUaW1lIDw9IHRzX3RvCiAgICBBTkQgaWYoCiAgICAgICAgKFNFTEVDVCB0b3RhbCBGUk9NIHRhYmxlU3RhdHMpIDw9IDEwMDAwLAogICAgICAgIDEsCiAgICAgICAgY2l0eUhhc2g2NChUaW1lc3RhbXBUaW1lLCByYW5kKCkpICUKICAgICAgICAgICAgZ3JlYXRlc3QoCiAgICAgICAgICAgICAgICBDQVNUKChTRUxFQ1QgdG90YWwgRlJPTSB0YWJsZVN0YXRzKSAvIDEwMDAwLCAnVUludDMyJyksCiAgICAgICAgICAgICAgICAxCiAgICAgICAgICAgICkgPSAwCiAgICApCkxJTUlUIDEwMDAwOw&chart=eyJ0eXBlIjoibGluZSIsImNvbmZpZyI6eyJ4YXhpcyI6InllYXIiLCJ5YXhpcyI6Ijk5dGhfYXZnX3RlbXAiLCJzZXJpZXMiOiJjb3VudHJ5IiwidGl0bGUiOiJUZW1wZXJhdHVyZSBieSBjb3VudHJ5IGFuZCB5ZWFyIn19'   show_statistics='true' 
>
WITH
    now64(3) AS ts_to,
    ts_to - INTERVAL 900 SECOND AS ts_from,
    tableStats AS
    (
        SELECT count() AS total
        FROM otel_v2.otel_logs
        WHERE TimestampTime >= ts_from
          AND TimestampTime <= ts_to
    )
SELECT
    Body AS __hdx_pattern_field,
    TimestampTime AS __hdx_timestamp,
    SeverityText AS __hdx_severity_text,
    TimestampTime,
    ServiceName
FROM otel_v2.otel_logs
WHERE
    TimestampTime >= ts_from
    AND TimestampTime <= ts_to
    AND if(
        (SELECT total FROM tableStats) <= 10000,
        1,
        cityHash64(TimestampTime, rand()) %
            greatest(
                CAST((SELECT total FROM tableStats) / 10000, 'UInt32'),
                1
            ) = 0
    )
LIMIT 10000;
</code>
</pre>

> This query employs some tricks to make sampling efficient, over the naive approach of just ordering by rand().

In practice, this sample size is sufficient to identify both dominant and anomalous patterns with high confidence - providing a fast, accurate snapshot of what’s happening without overwhelming compute resources or the user.

## Making the most of log patterns

Event patterns are most powerful when used as part of an iterative exploration process - bridging the gap between broad searches and precise debugging. Here are a few practical ways to get the most out of them in ClickStack:

**Start broad, then narrow down**

Begin with a wide search, such as filtering by service or severity, to capture a representative view of recent activity. Then, use Event Patterns to quickly identify dominant behaviors and narrow in on unusual or emerging signals. Once a pattern of interest appears, pivot back to traditional search or traces for deeper investigation.

**Combine with structured filters**

Event patterns shine when paired with meaningful filters like ServiceName, SeverityText, or deployment_id. This ensures that sampling focuses on relevant data, improving both performance and accuracy. For example, filtering by error logs before running pattern analysis will highlight failure types across a service, while filtering by warn might uncover degradation before it becomes critical. 

**Treat rare patterns as signals**

In most systems, rare patterns are sometimes more interesting than common ones. They may indicate new error paths, edge-case failures, or recently introduced regressions. Sorting or filtering by frequency within the event pattern view helps quickly surface these valuable outliers. 

In summary, pattern analysis is best for understanding *what’s happening* at a high level, not for pinpointing *why* it’s happening. Use it to orient yourself quickly and find leads, then switch to raw search, traces, or metrics for root cause analysis.

Finally, remember event patterns  don’t eliminate the need for better logs - but they make it far easier to see through the noise of an imperfect world.

## Looking ahead

Looking ahead, we see several opportunities to extend how event patterns are used across the ClickStack ecosystem. In a [recent blog post](https://clickhouse.com/blog/improve-compression-log-clustering), we explored how the Drain3 algorithm can be invoked directly as a **user-defined function (UDF)** within ClickHouse - allowing larger sets of data to be analyzed. Beyond just larger samples at query time, this opens some other interesting possibilities.

For example, event patterns can be used to identify common structures within specific log sources, such as those belonging to a single service or component. These learned templates could then be compiled to regular expression-based extraction rules, allowing ClickHouse to capture structured fields automatically at ingest time. Beyond accelerating queries, and avoiding schema-on-read operations, this structure can also significantly boost compression efficiency. We demonstrated this concept in a [recent blog post which allowed compression to be increased to 170x](https://clickhouse.com/blog/improve-compression-log-clustering).

We’re continuing to explore these directions and would love feedback from users who see potential applications in their own workflows. If you’re experimenting with event clustering in ClickStack, or using Drain3 as a UDF in ClickHouse, let us know how you’re using it and what capabilities you’d like to see next.

## Conclusion

Event patterns bring a powerful layer of intelligence to observability in ClickStack by helping teams navigate noisy, unstructured data with clarity and speed. By grouping similar events and surfacing anomalies in context at query time, they reduce time to insight and make exploratory log analysis far more approachable. While they don’t replace disciplined logging practices, they provide a practical way to uncover meaningful signals in complex systems. As ClickStack continues to evolve, event patterns lay the groundwork for smarter, more adaptive analytics that turn even the messiest data into actionable understanding.


* **Find structure in chaos:** Event patterns in ClickStack automatically cluster similar logs and traces, surfacing key behaviors and anomalies without predefined rules.

* **Analyze at query time:** Built on Drain3, ClickStack performs dynamic pattern analysis directly in the browser for any search result set.

* **Work smarter, not harder:** Use patterns to cut through noise, identify recurring issues, and uncover rare signals to accelerate observability analysis.

TL;DR

Analyzing logs at scale is one of the most persistent challenges in observability. Modern systems emit massive volumes of unstructured data - they are often repetitive, noisy, and lacking the context engineers need to act on. The result is an ocean of log lines where the signal is buried in noise, and finding meaning becomes a slow, expensive process.

Event patterns help you cope with this noise, providing a way to make sense of what’s already there and surfacing the key events.

In this post, we’ll explore why log analysis is inherently hard, what event patterns solve, how ClickStack implements them, and why we’ve chosen to do so at query time. We’ll also share practical guidance on how to make the most of event patterns today, and look ahead to how this technology could further improve compression and performance in the future.

## The problem: Why log analysis is hard

When something goes wrong, you often start with an alert or a trace, which is ideal - you already have a direction and context. But in exploratory scenarios, when you only know that “something’s off” and need to start from raw logs, the challenge multiplies.

Logs, especially from undisciplined systems or teams, can be incredibly verbose. They mix valuable signals with overwhelming amounts of noise, often repeating similar messages or recording irrelevant details. This verbosity makes it nearly impossible to chart every event or review each line individually. Engineers resort to iterative analysis with charting patterns, filtering errors, and refining queries over and over - hoping to gradually uncover the story beneath the noise.

> Event patterns aren’t a silver bullet. Most logs are noise - and the real solution starts upstream, with better decisions about what to log in the first place. The reality is that not every team or system logs perfectly, however, and event patterns still help surface the important signals amongst the noise.

While Lucene-style search helps when you know what you’re looking for, it offers little support when you don’t. In these cases, the process becomes one of pattern discovery: scanning massive datasets to find recurring behaviors or outliers. It’s an inherently difficult problem and akin to finding a needle in a haystack, except the haystack keeps growing every second.

![image1.png](/uploads/image1_c21ac2405f.png)

> Lucene (aka. Natural language querying) helps with needle in a haystack when you know what you’re looking for.

Event analysis extends beyond logs to traces and metrics, but the same challenge applies: when the data is unstructured and high-volume, extracting insight quickly is both computationally demanding and mentally taxing. 


Spin it up with a single command, download the open-source stack, and start exploring real-time analytics in minutes.

Get Started with ClickStack today


## How event patterns help log analysis

Event patterns in ClickStack make log and event analysis simpler by turning overwhelming noise into clear, structured insight. Instead of forcing you to sift through millions of raw events, they automatically detect recurring structures and group similar messages together into clusters - each representing a generalized “pattern” of behavior.

These clusters give you a compact, high-level view of your data: you can immediately see which patterns occur most frequently, which are new or unusual, and how their frequency changes over time. 

![image3.png](/uploads/image3_79085a08e8.png)

Frequent patterns often represent healthy or expected system behavior, while rare ones tend to reveal anomalies or emerging issues.  
Because event patterns are generated dynamically, you don’t need to define regular expressions or maintain brittle parsing rules. ClickStack adapts automatically to whatever data your systems produce, whether structured or free-text. This flexibility makes it easy to spot recurring errors, uncover noisy sources, or understand the types of logs a service emits - all without prior knowledge of what to search for.

Beyond incident response, event patterns also help teams improve their observability hygiene. By visualizing which logs dominate the volume, you can identify redundant or low-value messages to trim - giving you patterns you can add to your pipeline to filter events before ingestion or make sure your developers are “educated” on logging discipline!

## Using event patterns in ClickStack

Event patterns in ClickStack are designed to complement, not replace, traditional search. The workflow starts with a familiar step: issuing a search query to narrow the full corpus down to a relevant subset of events. That subset might still include millions of results - too many to inspect directly. Instead of displaying a giant table of raw logs, you can select **“Event Patterns”** on the left side to trigger a structured analysis of those returned events.

event_patterns.mp4

In this post, we’ll explore why log analysis is inherently hard, what event patterns solve, how ClickStack implements them, and why we’ve chosen to do so at query time. We’ll also share practical guidance on how to make the most of event patterns today, a

large_image2.png

medium_image2.png

small_image2.png

thumbnail_image2.png

image2.png

Using Event Patterns in ClickStack to accelerate observability analysis

Alerting is now available in ClickStack for ClickHouse Cloud, bringing feature parity with the open source version and eliminating the need for custom setup.

Users in the private preview can create search-based or chart-based alerts directly in the HyperDX UI, across logs, metrics, and traces. Alerts integrate out-of-the-box with Slack, PagerDuty, and webhooks, helping teams respond to spikes in latency, errors, or KPIs in real time.

TLDR;

Back in August, [we announced ClickStack in ClickHouse Cloud](https://clickhouse.com/blog/announcing-clickstack-in-clickhouse-cloud), marking the start of a unified future for observability and analytics. Since then, we’ve been busy onboarding customers, scaling the HyperDX UI to handle petabyte-scale workloads, and delivering enterprise features.

Today, we're excited to share one of the most requested capabilities - alerting, which is now available for ClickStack users in ClickHouse Cloud.


## What's new: Alerting in ClickHouse Cloud

Users participating in the private preview can now create and manage alerts directly within the HyperDX interface. These alerts work seamlessly across your ClickStack signals:  logs, metrics, and traces, and integrate with the notification tools you already use.

Out-of-the-box, ClickStack alerting supports Slack, PagerDuty, and generic webhooks, giving you the flexibility to connect any system or workflow. Whether you’re monitoring latency spikes, error rates, or business KPIs stored in ClickHouse, alerting helps you take action immediately.

![clickstack-webhook.png](/uploads/clickstack_webhook_9f59806558.png)

## Why alerting matters

This release achieves feature parity with open-source ClickStack, while simplifying setup for ClickHouse Cloud users. There’s no infrastructure to manage, and no custom configuration is required - alerting just works, right from the Cloud-hosted ClickStack UI in HyperDX.

We've also made optimizations to the UI and backend services to ensure alert processing scales with the size and performance demands we see in ClickHouse Cloud.

Deploy the world’s fastest and most scalable open source observability stack, with one command.


Get started with ClickStack 

## Alerting in ClickStack

ClickStack alerting in ClickHouse Cloud offers flexible ways to detect and respond to issues in real time. 

Within the HyperDX UI, users can create alerts in two ways - search-based or chart-based.

**Search-based** alerts let you define a query using either SQL or Lucene syntax, then specify a threshold for the number of matching events over a given time window for a grouping column e.g., Service name. When the count rises above or falls below that threshold for any of the lines produced by the GROUP BY, the alert triggers automatically - sending an event to the configured Webhook.

![create-clickstack-alert.png](/uploads/create_clickstack_alert_2fc88a80ab.png)

For more advanced use cases, you can build **chart-based alerts** directly from dashboards, powered by full SQL aggregations. This allows users to leverage any ClickHouse function for rich aggregations and statistical calculations. When a computed metric again exceeds a threshold, an alert can be fired for the chart.

For example, consider our public [ClickPy demo environment](https://clickpy.clickhouse.com), which provides analytics for Python package downloads. 

This public-facing demo has been instrumented with OpenTelemetry tracing to allow us to debug performance issues. Using this data, we can visualize query performance across user interactions in the public application. In this example, we track the 99th percentile of query latency and configure an alert to trigger whenever it exceeds a defined threshold - signaling that users may be experiencing slower performance.


simple_dashboard_alert.mp4

> Note: to add an alert, the underlying dashboard for the chart must be saved.

## Looking forward: What's next for ClickStack

Looking ahead, we’ll continue expanding ClickStack alerting with broader integrations such as email, and more advanced alerting logic, such as anomaly detection, to support richer use cases. 

We’re also exploring dedicated compute pools for alerts, allowing users with large-scale alerting needs to leverage [ClickHouse warehouses](https://clickhouse.com/docs/cloud/reference/warehouses) to isolate alert processing from their primary read and write workloads.

## Conclusion

Users in the private preview can now explore alerting in ClickStack integration for ClickHouse Cloud and enjoy the same powerful, ClickHouse-powered alerts available in the open source version. 

If you’d like to join the private preview, reach out to your ClickHouse account rep or complete the[ interest form](https://clickhouse.com/cloud/clickstack-private-preview).


Stay informed on feature ClickStack releases, roadmap, and community events!

Subscribe to our observability newsletter

Alerting has arrived in ClickStack for ClickHouse Cloud! Now you can create and manage real-time alerts across logs, metrics, and traces directly in the ClickStack UI - integrated with Slack, PagerDuty, and more

large_Alerting arrives in ClickStack for ClickHouse Cloud #1152.jpg

medium_Alerting arrives in ClickStack for ClickHouse Cloud #1152.jpg

small_Alerting arrives in ClickStack for ClickHouse Cloud #1152.jpg

thumbnail_Alerting arrives in ClickStack for ClickHouse Cloud #1152.jpg

Alerting arrives in ClickStack for ClickHouse Cloud #1152.jpg

Alerting arrives in ClickStack for ClickHouse Cloud

Observability costs rise because legacy search-first architectures and ingest/per-host pricing models conflict with modern analytics workflows. Learn the root causes and the open blueprint: OTel-first collection, unified storage, and affordable long reten

large_banner-breaking-free-observability.png

medium_banner-breaking-free-observability.png

small_banner-breaking-free-observability.png

thumbnail_banner-breaking-free-observability.png

banner-breaking-free-observability.png

Breaking free from rising observability costs with open, cost-efficient architectures

## ClickStackの発表

本日、ClickHouse上に構築された新しいオープンソースのオブザーバビリティソリューションである [ClickStack](https://clickhouse.com/o11y) を発表できることを嬉しく思います。ClickStackは、ログ、メトリクス、トレース、セッションリプレイを一括で扱える、即利用可能な完全な体験を提供します。ClickHouseのパフォーマンスと効率性を基盤としながらも、すべての人が利用できるオープンでアクセスしやすい、完全なオブザーバビリティスタックとして設計されています。

ClickStackを始めるには、ドキュメント内の [Getting Started](https://clickhouse.com/docs/use-cases/observability/clickstack/getting-started) ガイドをご覧ください。

![hyperdx-landing.png](/uploads/hyperdx_landing_c7bdd6582c.png)

これまで、NetflixやeBayなどの大規模なエンジニアリングチームは、オブザーバビリティ用途にClickHouseを選んできました。カラム指向設計、圧縮、高スループットのベクトル化クエリエンジンにより、コンテキストが豊富でユニーク数（カーディナリティ）の高い「wide event（ワイドイベント）」の保存に理想的だったためです。こうしたモダンなオブザーバビリティのアプローチ（"Observability 2.0"と呼ばれることもあります）は、従来の「三本柱モデル」から脱却し、分断されたテレメトリソースを繋ぎ合わせる複雑さを排除します。

しかし今までは、このモデルの恩恵を最大限に受けられるのは、ClickHouse上に独自のオブザーバビリティ体験を構築できるリソースを持ったチームに限られていました。他の多くのチームは、一般的な可視化ツールや、ClickHouse上に構築されたサードパーティの商用プラットフォームに頼っていました。こうしたツールはClickHouse用の基本的なインターフェースを提供していましたが、日常的な観測作業で長いSQLクエリが必要になったり、ClickHouseの性能やオープンなアーキテクチャを十分に活用できないこともありました。

それが、今日から変わります。HyperDXによって支えられたClickStackのリリースにより、すべてのユーザーに公平な選択肢が提供されます。この完全オープンソースのスタックには、すぐに使えるOpenTelemetryコレクター、wide eventに最適化されたUI、自然言語クエリ、セッションリプレイ、アラートなどが含まれています。

そしてそれらすべてが、オブザーバビリティ分野で信頼されている、あの高性能・高圧縮のClickHouseエンジン上で動作します。

これまでは、多くのチームが、高価なクローズドソースSaaS製品を選ぶか、オープンソースを組み合わせて自力で構築するか、という選択を迫られてきました。検索エンジンは柔軟で高速なクエリを提供していましたが、スケール運用や高速な集計には課題がありました。一方、メトリクスストアは集計性能に優れるものの、事前集計の必要があり、柔軟性や検索性能に欠けていました。どちらのアプローチも高カーディナリティのデータには向いておらず、それらを統合しようとすると、複雑さが増すばかりで、本質的な問題解決にはなりませんでした。

ClickStackを使えば、もはや選ぶ必要はありません。ユニーク数の多いwide eventデータに対して、高速な検索と集計をスケール可能な形で実現できます。オープンソースで、誰にでも。

## オブザーバビリティにおけるClickHouseの進化

![clickstack\_timeline.png](/uploads/clickstack_timeline_c2f60d65eb.png)

### それは「単なるデータの問題」

ClickHouseの初期ユーザーたちは、オブザーバビリティが本質的には「データの問題」であることを早くから認識していました。選ぶデータベースが、オブザーバビリティプラットフォームのコスト、スケール、機能性を決定づけるため、正しいデータベースを選ぶことは、自社構築でも観測系スタートアップでも、最も重要なアーキテクチャの決断の1つです。

この理由から、ClickHouseは長年にわたり、オブザーバビリティスタックの中核であり続けてきました。たとえば [Netflix](https://www.youtube.com/watch?v=HRh5setqGCU) や [eBay](https://clickhouse.com/videos/monitorama-pdx-2024-distributed-tracing-all-the-warning-signs-were-out-there) のような大企業から、SentryやDash0といったオブザーバビリティ系スタートアップに至るまで、ClickHouseは大規模なログ・メトリクス・トレースを支えています。カラム指向のストレージ、積極的な圧縮、ベクトル化実行エンジンにより、コストを大幅に削減し、サブセカンドでのクエリ応答を実現します。これにより、エンジニアはツールの遅さを待たずにリアルタイムでシステムをデバッグできます。

### 必要なのは、「wide event」とカラムストア

以前のブログ投稿「[The State of SQL-Based Observability](https://clickhouse.com/blog/the-state-of-sql-based-observability)」およびその[続編](https://clickhouse.com/blog/evolution-of-sql-based-observability-with-clickhouse)で、このトレンドについて深く掘り下げました。そこで当時は明確な名前を付けませんでしたが、これは今日の「Observability 2.0」と完全に一致する考え方です。つまり、ログ・メトリクス・トレースの「柱」ではなく、「wide event（幅広いイベント）」を中心に構築された統一モデルです。

これまで、多くのチームはログ・メトリクス・トレースを別々のストアに保存していたため、断片化、手動での相関付け、不要な複雑さが生まれていました。wide eventは、すべてのオブザーバビリティ信号を1つのクエリ可能な構造に統合することで、こうしたサイロを解消します。

wide eventは、ユーザー、サービス、HTTPパス、ステータスコード、キャッシュ結果など、アプリケーションの文脈すべてを1つのレコードに記録します。この統一された構造により、ユニーク数の多いデータに対しても高速な検索と集計が可能になります ― もちろん、それを効率的に圧縮・保存できるストレージエンジンがあればの話です！

NoSQLソリューション、特に検索エンジンはこの構造を取り入れたものの、広範な範囲を集計する性能には欠けていました。検索や「銀河の中の針探し」には適していても、大規模集計には不向きだったのです。

この課題を解決するClickHouseの秘密は今も変わりません。すなわち、カラムストレージ、深い圧縮を実現する豊富なコーデックライブラリ、そして分析ワークロードに最適化された大規模並列エンジンです。

### リソース効率に優れ、スケーラブル

ClickHouse Cloudではさらに進化を遂げ、ストレージとコンピュートの分離を実現するためにオブジェクトストレージを採用しました。これは、オブザーバビリティをペタバイト級、さらにはそれ以上にスケールさせ、弾力的に拡張するために不可欠です。さらに要求の厳しいユースケースをサポートするために、同じデータソースを使用しながら、取り込みとクエリなどのワークロードごとに計算リソースを分離できる「コンピュート・コンピュート分離」も導入しました。

オブザーバビリティのニーズがより複雑になるにつれ、半構造化イベントに対するネイティブなJSON対応が不可欠であると認識しました。ClickHouseはこのニーズに応える形で進化し、カラム指向処理のメリットを維持しながら、半構造化データへの一級対応を追加しました。データの到着に応じてカラムが自動生成され、型の昇格やカラムの拡張もClickHouseが自動で管理します。これこそが、オブザーバビリティに必要な「書き込み時スキーマ」であり、現代的な分析エンジンに求められるパフォーマンス、圧縮率、柔軟性を兼ね備えています。

### OpenTelemetryの台頭

この進化は、ログ・メトリクス・トレース全体のテレメトリ収集における事実上の標準である OpenTelemetry（OTel）の台頭と時を同じくしました。私たちは **ClickHouse用OpenTelemetry Exporter** の公式サポートと開発への貢献を開始しました。

OpenTelemetryは、私たちのエコシステムにとって大きなブレークスルーとなりました。ベンダーニュートラルで標準化された方法でオブザーバビリティデータを収集・エクスポートでき、CollectorはClickHouseにデータを直接送信するよう設定できます。私たちは、エクスポーターが堅牢でスケーラブルであり、ClickHouseの設計原則に沿うよう、コミュニティと密接に連携してきました。

最初に直面した最大の課題の1つがスキーマ設計でした。オブザーバビリティにおいては[万能なスキーマは存在せず](https://clickhouse.com/blog/clickhouse-and-open-telemtry)、各チームごとにクエリパターンや保持要件、サービス構成が異なります。そのため、このエクスポーターにはログ・メトリクス・トレースに対応したデフォルトスキーマが付属していますが、チームごとにワークロードに合わせて自由にカスタマイズすることを推奨しています。

### 欠けていたピース

しかし、優れたデータベースとスキーマ、収集・取り込みの手段が整っていても、それだけでは十分ではないことがすぐに分かりました。エンジニアには、すぐに使える取り込み、可視化、アラート、そして自分たちのワークフローに適したUIが必要です。これまでは、OpenTelemetryによる収集とGrafanaによるダッシュボードに頼るのが一般的でした。

これでも十分には機能していました。実際、[社内のオブザーバビリティチームはDatadogをClickHouseベースのスタックに置き換え](https://clickhouse.com/blog/building-a-logging-platform-with-clickhouse-and-saving-millions-over-datadog)、数億円規模のコスト削減と200倍以上のコスト効率を実現しました。現在、私たちの社内ログシステムは43ペタバイトを超えるOpenTelemetryデータを保存しており、スキーマや主キーはそのスケールに最適化されています。このアプローチのパフォーマンスとコスト効率は証明されましたが、体験はもっとシンプルにできると感じていました。

私たちが求めていたのは、より明確な設計思想と、より簡単に始められる仕組み、そして最も重要なことに、ClickHouseのために作られたUIです。効率的なクエリを構築し、wide eventからパターンを浮かび上がらせ、ClickHouseのパワーを隠さず、かつ優れたユーザー体験を提供できるUIが必要だったのです。

そして、SQLベースのオブザーバビリティがwide eventモデルを後押ししてきたことは事実ですが、同時に私たちはユーザーが「今どこにいるか」にも対応する必要があると考えていました。ELKスタックのような検索エンジンが成功した理由は、直感的なログ検索インターフェースを提供していたからです。その体験をClickHouse上で実現したいと考えたのです。

## HyperDXの登場

そんなときに出会ったのが、ClickHouse向けに設計されたオープンソースのオブザーバビリティレイヤー **HyperDX** です。2024年後半、HyperDXがv2のUIをオープンソースとして公開した際、私たちはすぐに社内テストを行い、これこそが欠けていたピースだと確信しました。セットアップはシームレスで、開発者体験も素晴らしく、ユーザーにもこの体験を届けたいと思いました。

HyperDXは、私たちが求めていたすべてを備えていました：

* **標準に基づくデータ収集**：HyperDXは初期からOpenTelemetryを採用しており、ClickHouseのOpenTelemetryエクスポーターへの私たちの投資と完全に一致しています。
* **オープンソース第一**：堅牢なオブザーバビリティツールは誰にでも利用可能であるべきという理念を共有しており、HyperDXのクラウドネイティブな設計により、運用はシンプルかつコスト効率的です。

標準への準拠に加えて、HyperDXはClickHouseのことを理解した設計になっています。チームはクエリ最適化に真剣に取り組んでおり、ユーザーが意識せずとも高速・信頼性の高いパフォーマンスを得られるようになっています。特にインシデント調査時には、ミリ秒単位の違いが重要です。

ClickStackは、OpenTelemetryコレクターの組み込み取り込み、HyperDX UI向けに最適化されたスキーマとともに、取り込み・保存・可視化を統合した体験として提供されます。デフォルトのスキーマはそのまま動作するよう設計されており、特に意識せずともすぐに使い始めることができます（もちろん、必要であればカスタマイズも可能です）。

![clickstack\_simple.png](/uploads/clickstack_simple_6ae8ee85d0.png)

ClickStackのシンプルな構造により、各レイヤーは独立してスケールできます。取り込みスループットを上げたいなら、OpenTelemetryコレクターゲートウェイを追加するだけ。クエリやストレージ容量を増やしたいなら、ClickHouse自体をスケールすればOKです。このモジュール構成により、データやチームの成長に合わせて、スタック全体を再構築することなく拡張できます。

HyperDXを取得して以来、私たちはプロダクトの簡素化と柔軟性の拡大に注力してきました。デフォルトスキーマを使って即時利用もできますし、自分たちのユースケースに合わせた独自スキーマを持ち込むことも可能です。ClickHouse同様、「1つのサイズですべてに対応」はしないという信念のもと、スケーラビリティの鍵は柔軟性にあると考えています。

<img preview="/uploads/clickstack_simple_v2_0c8e38c417.gif"  src="/uploads/simple_clickstack_3546eb282e.gif" alt="catalogue_lakehouse.png" class="h-auto w-auto max-w-full"  style="width: 100%;">

同時に、私たちはSQLの原点も忘れていません。SQLは今でもデータの共通言語であり、多くのClickHouse上級ユーザーにとっては、最も表現力があり、効率的な探索手段です。そのため、HyperDX UIにはネイティブSQLクエリのサポートが含まれており、上級ユーザーは妥協なしにエンジンへ直接アクセスできます。

<img preview="/uploads/clickstack_sql_small_a55c2027c3.gif"  src="/uploads/clickstack_sql_1eb4e0e9a2.gif" alt="catalogue_lakehouse.png" class="h-auto w-auto max-w-full"  style="width: 100%;">

さらに、デバッグや探索を容易にする新機能も導入しました。たとえば「イベントデルタ」は、異常やパフォーマンスの退行を素早く特定するのに役立ちます。特定の属性におけるユニーク値をサンプリングすることで、UIはパフォーマンスの差異や変動を可視化し、何がどう変化したのかを簡単に把握できるようにします。

<img preview="/uploads/event_deltas_simple_3bcfb720b3.gif"  src="/uploads/event_deltas_6e27437203.gif" alt="catalogue_lakehouse.png" class="h-auto w-auto max-w-full"  style="width: 100%;">

そして最も重要なのは、このスタックがよりシンプルになったことです。OpenTelemetryが事実上の標準となった今、すべてのデータはOTelコレクター経由で取り込まれるようになりました。デフォルト構成では、すぐに使えることを前提とした設計思想の強いスキーマが用意されていますが、ユーザーが自由に変更・拡張することも可能です。このスタックは **OpenTelemetryネイティブ** でありながら、**OpenTelemetry専用** ではありません。オープンスキーマのおかげで、HyperDXは既存のデータパイプラインやスキーマとも連携可能です。

## 結論と今後の展望

ClickStackは、ClickHouseがオブザーバビリティ分野における取り組みをさらに進化させた結果生まれたものであり、直感的かつ設計思想の明確なフルスタック体験を、オープンソースとオープンスタンダードの力で提供します。ClickHouseの高性能カラム型エンジン、OpenTelemetryの標準化されたインスツルメンテーション、HyperDXの特化UIを統合することで、モダンなオブザーバビリティのアプローチをついに誰でも使えるものにしました。

オープンソースへの継続的な取り組みにより、ClickStackは単一サービスの導入からペタバイト級のシステムに至るまで、誰でも利用可能です。私たちは今後も、高性能オブザーバビリティのためのデータベースコアと、Grafanaのような既存ツールとの統合の両方に投資を続けていきます。

ClickStackは単なるツールではありません。あらゆるテレメトリ信号を統合する、高性能なカラム型データベースを基盤とした統一プラットフォームであり、自然言語クエリ、セッションリプレイ、アラート機能までもすぐに使える形で提供されます。

ClickStackを始めるには、ドキュメント内の [Getting Started](https://clickhouse.com/docs/use-cases/observability/clickstack/getting-started) ガイドをご覧ください。


ClickStackの登場をお知らせします。ClickHouse上に構築されたオープンソースのオブザーバビリティスタックで、ログ・メトリクス・トレース・セッションリプレイを、超高速かつ開発者フレンドリーなプラットフォームで一元的に提供します。

Mike Shi

Blog posts by Mike Shi

Observability - a year in review

What's new in ClickStack. November '25.

Using Event Patterns in ClickStack to accelerate observability analysis

Alerting arrives in ClickStack for ClickHouse Cloud

Breaking free from rising observability costs with open, cost-efficient architectures

ClickStack: ClickHouse上に構築された高性能なOSSオブザーバビリティスタック