Luzmo uses ClickHouse Cloud to power fast, embedded analytics for data-driven companies with highly variable, customer-defined data. After years on open-source ClickHouse, Luzmo moved to ClickHouse Cloud to offload infrastructure management and focus on building analytics, not running clusters. ClickHouse now supports embedded analytics, query and plugin log analysis, and vector search, delivering high performance, flexibility, and cost-efficiency.


Summary

[Luzmo](https://www.luzmo.com/) empowers data-heavy applications to deliver lightning-fast, accessible analytics to their users. By integrating powerful visualizations and AI-driven exploration directly into the application, Luzmo helps teams building data-centric software turn high-volume information into intuitive, actionable insights for their customers.

Some companies use Luzmo to surface familiar SaaS metrics (e.g. campaign performance, ROI, usage trends) inside marketing or sales platforms. Others, like Belgium's largest telecom provider, Proximus, operate in very different domains, where location intelligence and large-scale operational data need to be visualized, explored, and monetized. In every case, analytics need to feel instant, even when the underlying data is complex.

From the beginning, delivering that kind of experience meant choosing infrastructure that could keep up. We caught up with Luzmo's co-founder and CTO, Haroen Vermylen, to chat about why the team originally adopted ClickHouse, what prompted their move from open source to [ClickHouse Cloud](https://clickhouse.com/cloud), and how ClickHouse has since spread across the company, powering everything from end-user analytics to observability-related workloads and vector search.

<iframe width="748" height="432" src="https://www.youtube.com/embed/dEwnRl1wH-Q" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

## Why Luzmo originally chose ClickHouse

Luzmo was founded (as Cumul.io) in Leuven, Belgium, in 2015\. Two years later, the team discovered ClickHouse. "It's been quite a ride," Haroen says with a smile.

At the time, Luzmo needed a fast internal warehouse to store and analyze data for customers who preferred the platform to host it on their behalf. While Luzmo is federated (many customers keep their data in their own systems), others rely on it to store and query that data efficiently.

Back then, much of the analytics world still equated serious performance with sprawling distributed systems and large clusters. Scaling typically meant spending heavily on infrastructure and taking on significant operational complexity.

ClickHouse, open-sourced just a year earlier, offered a different set of tradeoffs. Rather than requiring massive infrastructure to unlock speed, it delivered major performance gains even at small scale, while still offering a clear path to grow as needed. "ClickHouse came out, and it worked on my laptop," Haroen recalls. "That was pretty amazing."

The team evaluated a number of alternatives, but none offered ClickHouse's combination of performance, flexibility, and scalability. "We want to make things fast for our customers," Haroen says. "That's why we settled on ClickHouse."

## From open-source to ClickHouse Cloud

In late 2022, ClickHouse released [ClickHouse Cloud](https://clickhouse.com/cloud), giving teams a way to run ClickHouse without having to manage clusters themselves. Luzmo was one of the first customers to sign up.

For Haroen and the team, the motivation was simple enough: they never set out to manage infrastructure. "We're in the business of building analytics," he says. "Running clusters isn't really our core business, so that was a task that we wanted to get rid of."

Moving to a managed service allowed the team to offload day-to-day operational work and stay focused on building analytics and reporting for customers. It was a "best of both worlds" situation, with ClickHouse Cloud handling provisioning, upgrades, and ongoing maintenance, while still preserving the performance benefits and scalability that attracted Haroen and the team to ClickHouse in the first place.

## One platform, many use cases

Over the years, Luzmo's use of ClickHouse has evolved well beyond customer-facing analytics. As Haroen jokes, "ClickHouse has been quite infectious inside our company."

For example, Luzmo now uses ClickHouse for observability-related workloads, storing query logs and plugin logs so both the team and its customers can understand how queries behave in production. That visibility helps customers identify slow queries, trace issues back to specific data sources, and make informed decisions about how their analytics are powered.

More recently, ClickHouse has become part of Luzmo's work with [vector search](https://clickhouse.com/docs/knowledgebase/vector-search). Rather than introducing a separate system, the team uses ClickHouse to store embeddings, metadata, and related data, extending the platform into what Haroen describes as "non-obvious use cases."

In practice, ClickHouse has become a kind of default. Once in place, it's often the simplest and most effective choice for the next problem, without adding unnecessary architectural sprawl.

## Engineering for the unknown

Luzmo's architecture comes with a few unique challenges. As a multi-tenant platform, the team often doesn't know what data it will receive until it arrives. That leads to tens of thousands of tables, highly variable schemas, and workloads that don't fit neatly into predefined models.

On top of that, many customer queries span multiple systems. To manage that complexity, Luzmo has built its own query engine to optimize and coordinate those requests, ordering and grouping them to reduce load and improve efficiency. In some cases, large volumes of incoming queries can be consolidated into a single, more efficient operation, reducing pressure on ClickHouse while still delivering fast results.

Some of these challenges have required custom engineering. But over time, Haroen says, improvements in ClickHouse itself have smoothed many of the rough edges. New features and data types continue to expand what's possible, even in environments where the data is unpredictable by nature.

## "Fast, versatile, cost-efficient"

Luzmo bills itself as "the fastest way to embed analytics in your product." For Haroen, that promise goes hand in hand with the role ClickHouse plays under the hood.

Today, ClickHouse Cloud supports everything from end-user analytics to observability and emerging workloads like vector search. It gives Luzmo the performance it needs, the flexibility to take on new use cases, and the cost profile to scale without friction. And it does all of that while keeping analytics fast and effortless for end users.

Asked to sum up Luzmo's ClickHouse experience in a few words, Haroen thinks for a moment before responding: "extremely fast, extremely versatile, extremely cost-efficient."


Interested in seeing how ClickHouse works on your data? Get started with ClickHouse Cloud in minutes and receive $300 in free credits.

Need speed and scalability from your database?

“ClickHouse Cloud has been quite infectious inside our company. It’s extremely fast, extremely versatile, and extremely cost-efficient.”  Haroen Vermylen, Co-Founder and CTO

large_Luzmo Blog Issue 1237.jpg

medium_Luzmo Blog Issue 1237.jpg

small_Luzmo Blog Issue 1237.jpg

thumbnail_Luzmo Blog Issue 1237.jpg

Luzmo Blog Issue 1237.jpg

How Luzmo uses ClickHouse Cloud to power fast end-user analytics

For [Cloudflare](https://www.cloudflare.com/en-ca/)’s Jamie Herre, trouble is the engineer’s version of Murphy’s Law: whatever *can* go wrong *will* go wrong, and at scale it’s not a matter of *if* but *when*. A drive fails. A server goes down. A certificate expires. A link breaks. “If you carry a pager, you know what I’m talking about,” he says. “You can always assume something is broken somewhere.”

When Jamie joined Cloudflare as Senior Director of Engineering in 2018, its events pipeline was already “by far” the biggest he’d ever seen. Seven years later, the company has grown to serve roughly one-fifth of the world’s websites. What once looked enormous now feels modest compared to the quadrillions of events Cloudflare processes daily.

Scaling, Jamie argues, isn’t a milestone or a box to check. It’s a journey of constant adaptation: to more data, more complexity, more failure. Each new ceiling eventually becomes the floor for what comes next. “When we talk about scaling, it’s not a process that ends,” he says. “As you go up in scale, things are going to fail more frequently.”

Trouble and scaling, in this sense, are two sides of the same coin. A sudden surge in traffic can look a lot like losing half your capacity to an outage, and vice versa. In both cases, the system is stretched beyond yesterday’s limits. The challenge for designers and engineers, Jamie says, isn’t to avoid that stress—that’s impossible—but to build infrastructure that bends without breaking, and keeps delivering answers in the face of inevitable failure.

## What good trouble looks like

To show what “bending without breaking” looks like in practice, Jamie demoed his team’s analytics system at an [August 2025 ClickHouse meetup](https://clickhouse.com/videos/meetupsf_august_2025_2) at Cloudflare’s office in San Francisco. 

The first results showed the system’s scale. A single query scanned 96 trillion events in an hour and returned in less than two seconds, with a margin of error under one percent. Zooming out to a full day, the same query covered 1.61 quadrillion events—and still finished in less than two seconds. For Jamie, part of the fun was being able to say “quadrillion” out loud (“it’s a vanity thing,” he jokes) but his point was serious: even at volumes that, for many teams, would defy imagination, the system continued to respond instantly and accurately.

Then came the stress test. What happens when trouble arrives not as a traffic surge, but as a loss of capacity? Jamie simulated disconnecting a major North American data center, and then all of North America at once. Errors spiked, as expected, but the queries kept returning results. Thanks to Cloudflare’s distributed, active-active design, with more than 300 data centers around the world, European clusters automatically picked up the load, and the results remained consistent within the same tight margin of error.

Even when scaling the query window—from an hour to a day, a week, a month, and even a year—the system’s performance held steady. “No matter how much I ask it for, it will return results in less than two seconds,” Jamie says.

The demo proved that Cloudflare’s analytics system is both resilient and responsive. It can withstand large-scale outages and changes in scale without collapsing, and return queries in seconds regardless of load, capacity, or volume. In other systems, maintaining that level of responsiveness at extreme scale would require complex coordination, aggressive tuning, and major architectural tradeoffs, but ClickHouse lets Cloudflare operate this way by design. The fundamental assumption may be that “something is always sub-optimal,” but that doesn’t prevent successful responses.

“At Cloudflare, we’re always scaling,” Jamie says. “There’s always more trouble tomorrow. But we’ve designed our system around ClickHouse to be able to deal with that.”

## Why ClickHouse works for Cloudflare

“So what’s so great about ClickHouse?” Jamie asks. It turns out, quite a lot.

Cloudflare has been running on open-source ClickHouse for nearly 10 years, making it one of the OLAP database’s earliest large-scale adopters. That long history has shaped how Jamie and his team think about resilience and performance at global scale.

One “underrated feature,” according to Jamie, is the HTTP protocol. Cloudflare’s analytics clients interact with ClickHouse entirely over HTTP, which makes integration simple and universal. “There are all these tools and modes that you can leverage,” he says.

He also highlights the “minimal coordination required.” Unlike systems that rely on constant Raft or Paxos negotiation, ClickHouse nodes don’t need heavy orchestration to keep working. “When I take away a third of the capacity, like we did in the demo, not that much goes wrong, because all of those nodes are still there,” he explains.

That philosophy extends to what Jamie calls “soft clusters”—the ability to interrogate any node and choose dynamic combinations. “This gives us a lot of control and flexibility about how to find the nodes that are working and happy,” he says. There’s also what he calls “optional complexity,” or the ability to turn features on and off as needed. “We’ve tried to leverage the things that work well for our specific use case,” he adds.

Even the SQL dialect has turned out to be a “surprising advantage.” It took some getting used to, but over time Jamie has come to value how expressive and efficient it is for the team’s workloads. “It’s really cool,” he says.

Finally, Jamie emphasizes the value of ClickHouse’s open-source community and source code. For Cloudflare, being able to understand, contribute to, and rely on the database’s evolution has been valuable. “We’ve gotten a lot over the years by being part of this community,” he says.

## Jamie’s advice—just do it!

Jamie closed his talk with a reminder that scaling is never finished. The best time to prepare is right now. “It’s never too early, and it’s never too late,” he says. “You’ll never be done.”

The important thing, he argues, is to think about scaling *before* trouble forces your hand. “Say your workload suddenly scales 10x or 100x—would it fail in a good way, or would it fail in a bad way? And then the inverse of that is, what if you lose nine-tenths of the capacity? Would it just keel over? Or would you still be able to use it successfully?”

For Cloudflare, these aren’t hypotheticals. With quadrillions of events a day, hundreds of data centers around the world, and the omnipresent inevitability of failure, the company has to design for explosive growth and catastrophic loss at the same time. ClickHouse gives Jamie and his team the flexibility to handle either scenario without sacrificing speed or resilience.

Your company may not operate at Cloudflare’s scale, but the same design principles apply whether you’re starting with gigabytes or scaling from terabytes to petabytes—the goal is simply to be ready when scale arrives.

The lesson is universal. Trouble is guaranteed. The only question is whether you’ll be ready when it finds you.

Ready to make your systems more resilient?

“At Cloudflare, we’re always scaling. There’s always more trouble tomorrow. But we’ve designed our system around ClickHouse to be able to deal with that.”  Jamie Herre, Senior Director of Engineering

large_Cloudflare Blog Issue 1235.jpg

medium_Cloudflare Blog Issue 1235.jpg

small_Cloudflare Blog Issue 1235.jpg

thumbnail_Cloudflare Blog Issue 1235.jpg

Cloudflare Blog Issue 1235.jpg

Trouble will find you: How Cloudflare uses ClickHouse to scale analytics at quadrillion-row scale

ClickHouse 26.1 is here! In this post, you will learn about deduplication of asynchronous inserts with materialized views, new syntax for indexing projections, support for Variant in all functions, and more!

large_image4.png

medium_image4.png

small_image4.png

thumbnail_image4.png

image4.png

ClickHouse Release 26.1


[Wix](https://wix.com), a leading global platform for creating, managing, and growing a complete digital presence, operates one of the most complex production environments globally. The platform serves about  300 million users, runs 4,000+ microservices, and processes billions of requests per day. Any individual user action can traverse long, dynamic chains of services, each with the potential to introduce latency, errors, or cascading failures.

The company is known for its zero-error tolerance culture, where resilience and error handling are designed into systems from day one. As Wix continued to scale, the engineering team became increasingly committed to finding a way to make incident response faster, more accurate, and less disruptive to developers or the underlying data infrastructure.

This initiative ultimately led Wix to AI-driven incident response, leveraging Wix’s foundations of handling real-world production incidents at scale. 

## The Challenge: Complexity, Scale, and the Limits of Manual Response

To push incident response beyond the limits of manual investigation, Wix experimented with several internal approaches. The team built automation tools and ran early AI experiments that showed promise, but they struggled to scale. Turning these experiments into reliable, production-grade systems would have required significant ongoing investment and maintenance.

At the same time, any AI-driven approach posed a new risk of query amplification. Investigating an alert with AI means running many more queries than a human would, often in parallel, while engineers still need full access to logs and dashboards. Without the right data layer, AI would either be too slow to trust or too disruptive to deploy.

Wix needed an approach that matched its zero-error tolerance culture without compromising system stability.

## ClickHouse and the Data Foundation for Agentic Workloads

Long before introducing AI agents, Wix had already made a critical architectural decision.

Nearly eight years ago, Alex Ulstein, head of visibility at Wix, set out to support centralized logging at massive scale. As the team began defining requirements, it quickly became clear that conventional approaches would struggle to meet the demands ahead.

Within weeks, Wix needed to ingest roughly one million log events per minute with no sampling, making existing search-based architectures impractical from both a performance and cost perspective.

Because Wix controls its logging libraries end to end, logs follow a fixed schema. That minimized the need for schema-on-read flexibility or full-text search, shifting the focus instead to raw ingestion speed, compression efficiency, and analytical query performance.

After benchmarking multiple open-source options, ClickHouse emerged as the clear choice. It delivered the fastest performance and the most complete SQL support among columnar databases. “ClickHouse was by far the fastest,” says Ulstein. “Compression was excellent, and the depth of its SQL feature set exceeded what we needed.”

Today, Wix’s largest application logging cluster runs on 60 machines, handles 300 queries per second, and ingests an average of 300 million events per minute without sampling. Most queries return in sub-second latency. “Ingestion speed is a really important part, and where ClickHouse shines,” Ulstein adds.

Critically, this architecture gave Wix something many teams lack: full-fidelity observability data at scale. That foundation would later prove essential for AI-driven incident response.

## Wild Moose: AI SRE Built from Tribal Knowledge

As alert volumes continued to grow, Wix set out to operationalize AI for incident response by evaluating external, production-ready solutions. They defined a clear set of requirements:

* AI must work the way engineers investigate, not as a black box  
* It must integrate into existing workflows (Slack, Grafana, logs)  
* It needs to automate tribal knowledge without replacing human judgment  
* And it must scale without disrupting human access to data

Together, these requirements made Wild Moose the right choice - AI built to work the way engineers do, at production scale.

[Wild Moose](https://wildmoose.ai) is an agentic AI SRE platform designed to codify how experienced engineers debug incidents. Instead of static rules or generic summaries, Wild Moose ingests institutional knowledge and automatically executes investigation workflows. It runs incident analysis the same way engineers do, only faster and in parallel.

“Wild Moose had the most mature approach for automating tribal knowledge. The responsiveness in tuning and adapting the product to our unique needs made it clear this was the right partnership,” Aviva Peisach, head of backend engineering at Wix shares.

When an alert fires, Wild Moose agents query logs, correlate signals across services, examine recent deployments, and compare against historical patterns. Enriched findings are delivered directly to Slack in under a minute, giving engineers high-confidence starting points instead of raw alerts or noisy summaries.

"With Wild Moose, we’re able to spot the root cause - whether it’s a recent GA, traffic switch, new experiment, or something environmental. Wild Moose helps pinpoint the actual root cause service causing the problem in a very complex environment with billions of requests coming in through a very long chain of calls between services,” says Peisach.

## How ClickHouse and Wild Moose Work Together

At Wix, ClickHouse and Wild Moose together formed a system capable of supporting AI-driven incident response under real production constraints.

### 1. AI-driven execution operationalizes incident response

By shifting incident investigations from human-driven or rule-based workflows to continuously running AI agents, Wild Moose transforms ClickHouse into a central hotspot for reliability logic at Wix. Unlike human-driven or rule-based systems, Wild Moose systematically leverages ClickHouse’s concurrency model by issuing many parallel queries per alert, fully exercising the database’s real-time analytics capabilities. Wix teams can now compose, test, and iterate on SQL queries directly inside Wild Moose agents, where those queries are executed automatically during incidents. 

### 2. Query performance enables parallel investigations

Wild Moose agents are able to run hundreds of queries per alert in parallel to build a complete picture of what's happening. ClickHouse's ability to sustain high query concurrency allows Wild Moose to heavily parallelize investigations without slowing down dashboards or human workflows. "With AI that approaches investigations the same way people do, solutions that work well for humans work well for the AI. Except automating these operations increases the scale and concurrency by orders of magnitude, so the fast and efficient querying that made us choose ClickHouse in the first place becomes even more crucial," Ulstein says.

### 3. Full-fidelity retention improves root cause accuracy

Because ClickHouse stores unsampled logs with efficient compression, Wild Moose has access to complete historical context for analysis and comparison. Rare failure patterns, edge cases, and subtle correlations remain detectable, and the AI learns from long-tail, high-volume examples that might be filtered out in sampled data. This long-term retention ensures the agent can draw on rich examples of past incidents that would otherwise be lost, strengthening its ability to reason about and respond to new failures.

### 4. SQL expressiveness accelerates insight

Wild Moose uses SQL as its native query interface, allowing AI models to generate highly problem-specific queries without relying on custom query languages. This lets the system reuse and adapt existing queries that Wix engineers already trust. ClickHouse’s rich SQL functions and join support enable Wild Moose to push certain correlations down to the database where they’re optimized, making investigations both faster and more accurate.

This allows Wild Moose to focus its AI on higher-level guidance that shapes the investigation, rather than on routine query mechanics, preserving valuable context for complex problem-solving.

### 5. Decoupled architecture enables flexibility

ClickHouse separates the data layer from the UI layer. AI agents can access raw logs directly while dashboards remain fast and independent, avoiding the “walled garden” constraints of traditional observability platforms.

This decoupling proved essential for rapid iteration and tuning. When Wild Moose needed to adjust investigation strategies or add new correlation techniques, they could do so without worrying about breaking existing dashboards or workflows.

## The Results: Higher Accuracy, Faster MTTR, and Boosted Team Morale

Wix’s existing ClickHouse infrastructure meant Wild Moose could be deployed quickly. Logs were structured for ClickHouse queries, full SQL support simplified query translation, and many existing human-written queries could be reused with minimal optimization. This supported rollout to hundreds of teams in a matter of weeks.

Within weeks, Wix could clearly see how their work with Wild Moose delivered results:

* **90% root cause accuracy** sustained across thousands of alerts  
* **50% projected reduction in MTTR**, giving engineers time back  
* **30,000+ alerts enriched per month**, with volume continuing to grow  
* **Hundreds of teams supported** across 200+ Slack channels  
* Reduced on-call stress and improved developer morale

The impact was felt in the day-to-day experience of developers on call. Instead of firefighting, engineers now start investigations with context, hypotheses, and likely root causes already surfaced. 

"We were able to get to a stage of over 80% accuracy in just three weeks, which is mind-blowing for a system as complex as ours," Peisach says.

On-call rotations have become less stressful. Engineers feel more confident knowing they have a system that could guide them quickly to the right answer. And unlike the hard-coded rules of Wix’s internal automation tools, Wild Moose learns and adapts through natural-language feedback, automatically improving accuracy over time.

## Looking Forward: A System for AI-Driven Reliability

Wix's experience highlights what large-scale engineering teams can achieve when AI, observability, and data infrastructure form a coherent architecture.

Wix designed and operates an AI-driven reliability system grounded in its own large-scale production experience. ClickHouse serves as the scalable data backbone for observability and AI workloads, while Wild Moose encodes hard-won operational knowledge into automated investigations. This architecture allows Wix to sustain a zero-error tolerance culture while continuing to scale one of the most complex platforms in the world.

What emerged was an investigation engine where AI agents continuously learn, and the data layer continuously scales. As the environment grows more complex, the system becomes more valuable. Tribal knowledge turns into durable institutional knowledge, alert storms become manageable, and developers spend more time building instead of firefighting.   


Get started today

Wix paired ClickHouse with Wild Moose's AI agents to automate incident response at scale, achieving 90% root cause accuracy across 30,000+ monthly alerts.

large_Wild Moose Wix ClickHouse Issue 1236.jpg

medium_Wild Moose Wix ClickHouse Issue 1236.jpg

small_Wild Moose Wix ClickHouse Issue 1236.jpg

thumbnail_Wild Moose Wix ClickHouse Issue 1236.jpg

Wild Moose Wix ClickHouse Issue 1236.jpg

How Wix Built AI-Driven Incident Response at Scale with ClickHouse and Wild Moose

- LINEヤフーはClickHouseを使用して世界有数のKafkaデプロイメントを観測し、1日に1兆以上のメッセージと2.6ペタバイトのI/Oを処理している。
- ClickHouseはAPIログを毎秒700万行インジェストし、これまでの観測ツールでは想像もできないほど大規模な、ミリ秒レベルのデバッグを可能にする。
- LINEヤフーはわずか24台のサーバーで4.1兆行をリアルタイムに分析し、Kafkaのアップストリームのバグ特定や、パフォーマンス改善に役立てている。


概要

[LINEヤフー](https://www.lycorp.co.jp/)については、まずその規模を説明する必要があるでしょう。「巨大テックカンパニー」では足りず、「国民的デジタルインフラ」と理解していただく必要があります。LINEとYahoo!の合併により生まれたLINEヤフーは、日本で最も有名なメッセージングアプリと、国内有数のニュースポータル、および巨大な支払いエコシステムを運営し、それらの間のデータ転送を、数えきれないほどの基幹サービスによって稼働させています。

それほどのスケールを堅実に動かしているのが、同社が全社的に使用しているApache Kafkaプラットフォームです。この中枢のプラットフォームがまるで神経系のように、アプリログ、マイクロサービスイベント、ゲーム内テレメトリなど、あらゆるデータを運んでいます。ピーク時には毎秒3,100万件のメッセージを処理することもあり、1日に流れるメッセージの数は1兆、さらにI/Oは2.6ペタバイトを超えるほどです。

「これはかなり大規模なKafkaクラスターです」と、LINEヤフーのKafkaプラットフォームチームで上級ソフトウェアエンジニアとテックリードを兼務する岡田 遥来氏は言います。「巨大です。おそらく世界でも有数の規模でしょう。」

ClickHouseが2025年6月に開催した[Observability happy hour in Tokyo](https://clickhouse.com/jp/videos/tokyo-o11y-happy-hour-line-corp-18jun25)で、岡田氏はKafkaを巨大な規模で運用していることや、まだ誰も知らない問題を発見したことを説明しました。その問題を解決するために、ほとんどの企業が想像もできないようなレベルのオブザーバビリティと、データベース速度、コスト効率、そして開発者にとって使いやすいことなどが求められました。


## 稀有なKafkaデプロイメント
岡田氏は、「社内で最もよく使用しているミドルウェアソリューションはKafkaです。」と説明しました。これは大袈裟ではありません。LINEヤフーはKafkaを、たとえばサービス間のクラシックなPub/Sub通信、ログパイプライン、非同期処理、ゲーム内データのストリーミング、そして、内部システム間でデータベースの同期を保つためのCDCフローなど、あらゆる処理に使用しています。サービスから送信されるデータのほとんどがKafkaに到達するといった状態です。

しかし、彼らにとって重要なのはそこではなく、そのボリュームです。LINEヤフーのKafkaプラットフォームは、200を超えるブローカーで構成され、25,000以上のクライアントサービスに機能を提供しています。負荷がそのレベルに達すると、ほんの些細な問題でも雪だるまが膨らむように効率が急降下し、稼働停止に追い込まれます。

また、KafkaはもともとLinkedInが開発し、2011年にオープンソース化されたプラットフォームです。1日に1兆を超えるほどのワークロードは想定されていません。未知の領域に足を踏み入れるようなチャレンジを、LINEヤフーは続けてきました。「これほどの規模で運用すると、世界中でまだ誰も目にしたことのない問題にも遭遇します。」（岡田氏）これまで、彼らはログの削除でレースコンディションが起きること、予期しないfsyncコールによりパフォーマンスがわずかに低下すること、Linuxカーネルの問題により接続が急増した際にパフォーマンスが急低下することなど、アップストリームの問題を修正およびレポートし、貢献してきました。

「こういった問題が日々起こったり起こらなかったりするので、オブザーバビリティが非常に重要になります」と岡田氏は言います。「各問題のルート構造を分析し、根本原因を突き止めるために、システムスタックのどの部分に関しても最大限のオブザーバビリティが必要です。」


## マルチレイヤで構成されるLINEヤフーのオブザーバビリティスタック
巨大なシステムで問題をいち早く検出するため、社内でオブザーバビリティスタックを構築しました。それは一般的なモニタリングセットアップというよりも、むしろ完璧な調査研究システムに近いものでした。

![lineyahoo1.png](https://cms.clickhouse-dev.com:1337/uploads/lineyahoo1_43fc07e3ae.png)
（図）Kafka、JVM、カーネル、ディスクのメトリクスをキャプチャし、リアルタイムで診断するマルチレイヤオブザーバビリティスタック

このスタックの最上部で実行されるプルーブは、エンドツーエンドのパフォーマンスを定期的に測定し、メッセージがプロデューサーからブローカーを通過してコンシューマーに到達するまでの時間を調べます。その下の層では、Async Profilerを使用してKafkaの処理をプロファイリングし、CPU動作の情報とJVMレベルのインサイトを取得して、問題をピンポイントで特定するために使用します。そこからさらに、ディレイアカウンティング、eBPFツール、各ディスクのリードコレクター、およびSMARTデータエクスポーターを使用してLinuxカーネルを詳しく測定します。カーネルスケジューラに問題が起きたり、ディスクが異常な動作を始めたりすると、わかるようになっています。

ですが、何か問題が起きた際に最も活躍するのは、やはりClickHouseを活用したKafka APIリクエストログパイプラインです。LINEヤフーはこのレイヤで、システムのあらゆる瞬間の動作を正確に把握していると言います。プロデュース、フェッチ、コンシューム、ブローカー間レプリケーション、管理オペレーションなど、Kafkaへのあらゆるリクエストがインターセプトされ、プロトコルバッファ経由でシリアル化されて、内部のKafkaクラスターにプッシュされます。その後、「ログインジェスター」にコンシュームされて、ClickHouseに大規模バッチで書き込まれます。

![lineyahoo2.png](https://cms.clickhouse-dev.com:1337/uploads/lineyahoo2_a90bc9b693.png)
（図）Kafka APIのリクエストフロー。インターセプト後にClickHouseに送られる大容量のクエリ可能データをオブザーバビリティに使用。

これにより、莫大な量のデータが生成されます。同社では毎秒700万行のAPIリクエストログをClickHouseに格納しますが、やがてそれは4.1兆行を超えるデータセットにまで拡大します。「ClickHouseにインジェストされるデータ量はとんでもなく大きいです」（岡田氏）


## ClickHouseを選んだ理由
LINEヤフーがClickHouseを選んだ理由は主に3つあります。まず、ClickHouseは[SQLに対応](https://clickhouse.com/docs/sql-reference)しており、エンジニアが特別なトレーニングを受けなくてもデータをクエリできることです。「SQLは社内の開発者にとって最も使い慣れたクエリ言語です」と岡田氏は言います。「これは非常に大きな利点です。」

2つ目は、ClickHouseの[圧縮機能](https://clickhouse.com/docs/data-compression/compression-in-clickhouse)です。「ClickHouseはとにかくストレージ効率に優れています。カラムナーアーキテクチャのおかげで、非常に高い圧縮率が得られます。データ量が多くなっても、高いストレージ効率を維持できます。」（岡田氏）

最後は何と言っても、ClickHouseの[パフォーマンス](https://clickhouse.com/docs/concepts/why-clickhouse-is-so-fast)と[水平方向への拡張性](https://clickhouse.com/docs/architecture/horizontal-scaling)です。「パフォーマンスは非常に重要」と岡田氏は言います。「毎秒700万行のインジェストを24のClickHouseクラスターノードだけで実行できます。エネルギー効率にも、コスト効果にも非常に優れています。」

データアクセスにはRedashのダッシュボードを使用しますが、デバッグのセッション中にSQLクエリをダイレクトに実行して行うこともあります。また、岡田氏がClickHouseの[Tokyo happy hour](https://clickhouse.com/jp/videos/tokyo-o11y-happy-hour-line-corp-18jun25)で説明したように、このクエリこそが、まだ誰も知らないKafkaのバグを捕捉する立役者となるのです。


## Kafkaのバグを運用環境で捉える
ことの始まりは、Kafkaのパーティションでプロデュースリクエストが失敗したという、過去に体験したことのないエラーに遭遇したことでした。岡田氏はいつものようにダッシュボードをチェックし、レプリケーションが完全に停止したという、非常に深刻なアラームを確認します。リーダーはメッセージを受信するものの、フォロワーがそれに追いついておらず、プロデューサーがクォーラムを失うため、新しいメッセージが拒否されています。

最初はフォロワーの問題のように思えたのですが、フォロワーのログを詳しく調査すると、「offset out of order」や「non-monotonic offsets」という奇妙なメッセージに気付きました。これはあってはならないことです。Kafkaパーティションのオフセットは常に1つずつ前進するように作られており、1つでも後退すると、レプリケーションのパイプライン全体が壊れる可能性があります。そこで岡田氏はバッファのダンプを調査しますが、そのとき、「奇妙な現象」に遭遇します。正常にカウントアップされていたオフセットが、突如として大幅に巻き戻されているのです。Kafkaの奥深くに何か問題があると思いました。

岡田氏は、ProduceとListOffsetsの2つのKafka API（どちらもディスクのログセグメントにタッチする）の間で[レースコンディション](https://en.wikipedia.org/wiki/Race_condition)が起きていると推測します。極度に短い瞬間に2つのスレッドが同時にアクセスし、オフセットの状態を変えることで矛盾が起きたと仮定したのです。「この時点ではコードを解読しただけであり、単なる仮定に過ぎませんでした」と岡田氏は言います。「これをどう証明するかが重要でした。」

これを調査するため、どのAPIコールがどの瞬間に起きているかを厳密に知る必要がありました。岡田氏は言います。「ClickHouseで収集したAPIリクエストログが非常に役に立ちました。」ClickHouseに格納されたAPIリクエストログを使用し、エラーが発生した瞬間の数百万行に対してSQLクエリを実行しました。そして岡田氏の推測どおり、ListOffsetsコールはProduceリクエストが失敗したまさにその瞬間に起きていたことを突き止めました。2つのオペレーションが衝突したという明確な証拠により、アップストリームのバグであると証明されたのです。

岡田氏はチームでKafkaコミュニティに修正プログラムを提出し、イシューは現在レビュー中とのことです。数兆ものAPIログを簡単にクエリできるClickHouseのパワーがなければ、これほど簡単には解決しなかったはずです。


## LINEヤフーのスケールに欠かせないClickHouse
ほとんどの企業にとって、Kafkaは堅牢で予測可能なインフラストラクチャです。LINEヤフーほどの規模であっても、Kafkaというオープンソースプラットフォームに与えられた処理を何でも高速に実行します。

そこにClickHouseを組み込むことで、同社はメッセージ件数が1日に1兆を超えるほどのシステムで、リアルタイムなフォレンジックをプレーンなSQLで可能にしています。エンジニアは仮定を立て、クエリを作成し、数十億、あるいは数兆ものレコードに対して動作の追跡をミリ秒単位で行うことができます。

岡田氏は最後に、「ClickHouseは我々のオブザーバビリティスタックの中でも非常に重要な役割を果たしています。」と付け加えました。「毎秒700万行という莫大な量のデータ処理を24台のサーバーで賄うという、驚くようなコストパフォーマンスを達成しています。」

LINEヤフーほどのスケールで運用していると、何が起きても不思議ではありません。ですが、オブザーバビリティプラットフォームの中枢でClickHouseが機能している限り、どれほど奇妙で稀なバグであろうと必ず手掛かりは残ります。その手掛かりのおかげで、岡田氏のチームは世界有数の規模を誇るKafkaシステムを毎日無事に稼働させているのです。


社内オブザーバビリティスタックのアップグレードに関心を持たれた方は、ClickHouse Cloudを30日間無料でお試しください。

ClickHouse Cloudを無料トライアル

ClickHouseはLINEヤフー社のオブザーバビリティスタックの中でも非常に重要な役割を果たしています。毎秒700万行という莫大な量のデータ処理を24台のサーバーで賄うという、驚くようなコストパフォーマンスを達成しています。ー 上級ソフトウェアエンジニア 岡田 遥来氏

large_Line Yahoo User Story Issue 1233.png

medium_Line Yahoo User Story Issue 1233.png

small_Line Yahoo User Story Issue 1233.png

thumbnail_Line Yahoo User Story Issue 1233.png

Line Yahoo User Story Issue 1233.png

LINEヤフーが世界有数のKafkaデプロイメントのオブザーバビリティにClickHouseを活用

To understand [LY Corporation](https://www.lycorp.co.jp/en/), you have to think big. Not just “large tech company” big, but “nationwide digital infrastructure” big. Born from the merger of LINE and Yahoo! Japan, LY runs Japan’s most popular messaging app, one of the country’s biggest news portals, a massive payments ecosystem, and countless backend services moving data between them.

Few systems capture that scale better than LY’s company-wide Apache Kafka platform. It’s their central nervous system—the message bus carrying everything from app logs to microservice events to in-game telemetry. At peak, the platform processes 31 million messages per second. That adds up to more than 1 trillion messages and 2.6 petabytes of I/O every day.

“This is an extremely large-scale Kafka cluster,” says Haruki Okada, Senior Software Engineer and tech lead on LY’s Kafka platform team. “The scale is massive—it’s probably one of the largest deployments in the world.”

As Haruki explained at our [June 2025 observability happy hour in Tokyo](https://clickhouse.com/jp/videos/tokyo-o11y-happy-hour-line-corp-18jun25), running Kafka at that scale means encountering issues no one has ever seen before. Solving them requires a level of observability most companies never come close to—and a database fast, cost-efficient, and developer-friendly enough to keep up.

## A Kafka deployment like no other

In Haruki’s words, Kafka is “one of our most popular middleware solutions.” He’s not kidding. LY uses it for just about everything: classic pub/sub communication between services, logging pipelines, async processing, streaming in-game data, and CDC flows that keep databases in sync across internal systems. If a service emits data, odds are it touches Kafka.

But the real challenge isn’t merely the variety of use cases. It’s the *volume*. LY’s Kafka platform operates across more than 200 brokers, serving over 25,000 client servers. When you hit that level of load, even tiny inefficiencies can snowball into production outages.

And because Kafka (originally developed at LinkedIn and open-sourced in 2011) was never built with trillion-message-per-day workloads in mind, LY often finds itself in uncharted territory. As Haruki puts it, “When operating at this scale, we run into issues that no one else in the world has encountered yet.” Along the way, they’ve traced and contributed upstream fixes for everything from a race condition in log deletion, to a subtle performance issue caused by an unexpected fsync call, to a Linux kernel problem that tanked performance during sudden connection spikes.

“Since issues like these can pop up on a daily basis, observability has become extremely important,” Haruki says. “We analyze the root structure of each problem, and to get to the root cause, we need maximum observability across every part of the system stack.”

## LY’s multi-layer observability stack

To stay ahead of problems at this scale, the LY team built an observability stack that looks more like a full research laboratory than a typical monitoring setup.

![Line Yahoo User Story Issue 1233 (2).jpg](/uploads/Line_Yahoo_User_Story_Issue_1233_2_e2c7e6071a.jpg)

Multi-layer observability stack capturing Kafka, JVM, kernel, and disk metrics for real-time diagnosis.

At the top, they run a probe that constantly measures end-to-end performance—how long it takes for a message to make the full trip from producer to broker to consumer. Beneath that, they continuously profile the Kafka process using Async Profiler, capturing CPU behavior and JVM-level insights that help pinpoint hotspots. From there, they go even deeper, instrumenting the Linux kernel using delay accounting, eBPF tools, per-disk read collectors, and SMART data exporters. If the kernel scheduler blips, or if a disk starts behaving strangely, they know.

But the layer that matters most when things go wrong is the Kafka API request log pipeline, powered by ClickHouse—because that’s where LY sees exactly what the system was doing at any given moment. Every request to Kafka—produce, fetch, consume, inter-broker replication, admin operations—is intercepted, serialized via Protocol Buffers, pushed into an internal Kafka cluster, and then consumed by a “log ingestor” that writes it into ClickHouse in large batches.

![Line Yahoo User Story Issue 1233.jpg](/uploads/Line_Yahoo_User_Story_Issue_1233_23e232957b.jpg)

Kafka API requests flow through interceptors into ClickHouse for high-volume, queryable observability.

This produces an enormous amount of data. LY stores 7 million rows per second of API request logs in ClickHouse. Over time, that’s grown into a dataset of more than 4.1 trillion rows. As Haruki puts it, “The amount of data ingested into ClickHouse is extremely large.”

## Why they chose ClickHouse

The LY team chose ClickHouse for three main reasons. The first is its [SQL compatibility](https://clickhouse.com/docs/sql-reference), which means engineers can query the data without specialized training. “SQL is the query language our developers are most familiar with,” Haruki says. “This is a major advantage.”

The second is its [compression capabilities](https://clickhouse.com/docs/data-compression/compression-in-clickhouse). “ClickHouse is extremely efficient in terms of storage,” Haruki says. “Thanks to its columnar architecture, it achieves very high compression rates. Even with large volumes of data, storage efficiency remains high.”

Last but not least is [performance](https://clickhouse.com/docs/concepts/why-clickhouse-is-so-fast) and the [ability to scale horizontally](https://clickhouse.com/docs/architecture/horizontal-scaling). “Performance is key,” he says. “We’re ingesting 7 million rows per second, and we’re able to do this with just 24 ClickHouse cluster nodes, which is both energy-efficient and cost-effective.”

Engineers access the data through Redash dashboards or direct SQL queries, often during live debugging sessions. And as Haruki showed at our [Tokyo happy hour](https://clickhouse.com/jp/videos/tokyo-o11y-happy-hour-line-corp-18jun25), that queryability is how the team caught one of their strangest Kafka bugs to date.

## Catching a Kafka bug in production

The story started with something LY almost never sees: produce requests failing in a specific Kafka partition. When Haruki checked the usual dashboards, he saw something even more alarming—replication had completely stopped. The leader was receiving messages, but the followers weren’t keeping up, causing the producer to lose quorum and reject new messages.

At first glance, it looked like a follower issue. But when he dug into the follower logs, he found odd messages about “offset out of order” and “non-monotonic offsets.” That’s not supposed to happen. Offsets in Kafka partitions always move forward by exactly one. A single jump backward can break the entire replication pipeline. So he dumped the relevant buffer. That’s when he encountered a “strange phenomenon.” Offsets were counting up normally… and then suddenly jumping back dozens of values. Something deep in Kafka’s internals had gone wrong.

Haruki suspected it might be a [race condition](https://en.wikipedia.org/wiki/Race_condition) between two Kafka APIs—Produce and ListOffsets—which both touch the underlying log segment on disk. Under an extremely narrow timing window, two threads might both access or modify the offset state in a conflicting way. “At this point, I’m just forming hypotheses based on reading the code,” Haruki says. “The key was figuring out how to prove it.”

To confirm his theory, Haruki needed to know exactly which API calls happened at exactly which moments. Here, he says, “the API request log we collected with ClickHouse was really helpful.” Using the API request logs stored in ClickHouse, Haruki ran a SQL query across billions of rows for the timeframe of the incident. And there it was: a ListOffsets call had landed at the precise moment the Produce request started failing. With clear evidence that the two operations had collided, the hypothesis turned into a confirmed upstream bug.

Haruki and the LY team submitted a patch to the Kafka community, and the issue is now under review. Without ClickHouse giving them the ability to easily query trillions of API logs, the problem would have been much harder to solve.

## Why ClickHouse matters at LY’s scale

For most companies, Kafka is sturdy, predictable infrastructure. At LY Corporation’s scale, it’s a high-performance machine running at the edge of what the open-source platform was ever meant to handle.

ClickHouse gives LY something special: the ability to perform real-time forensics on a trillion-messages-per-day system using plain SQL. Engineers can take a hunch, write a query, and trace behavior down to the millisecond, across billions or even trillions of records.

As Haruki puts it, “ClickHouse plays a really important role in our automated monitoring efforts. We’re handling a massive amount of data—about seven million rows per second—but with just 24 servers, we’ve achieved outstanding cost performance.”

When you’re operating at LY’s scale, surprises are unavoidable. But with ClickHouse at the heart of their observability platform, even the oddest, rarest bugs leave a trail. And that trail is what lets Haruki and the LY team keep one of the world’s largest Kafka systems running smoothly every single day.


Interested in seeing how Managed ClickStack works for your observability data? Get started in minutes and receive $300 in free credits.

Looking to upgrade your team’s observability stack? 

“ClickHouse plays a really important role in our automated monitoring efforts. We’re handling a massive amount of data—about seven million rows per second—but with just 24 servers, we’ve achieved outstanding cost performance.” - Haruki Okada, Senior Softw

ClickHouse

Blog posts by ClickHouse

How Luzmo uses ClickHouse Cloud to power fast end-user analytics

Trouble will find you: How Cloudflare uses ClickHouse to scale analytics at quadrillion-row scale

ClickHouse Release 26.1

How Wix Built AI-Driven Incident Response at Scale with ClickHouse and Wild Moose

LINEヤフーが世界有数のKafkaデプロイメントのオブザーバビリティにClickHouseを活用

How LY Corporation uses ClickHouse to observe one of the largest Kafka deployments on earth