RunReveal: 10 years using ClickHouse taught us why it's perfect for AI

A decade of operating ClickHouse at scale, and why ClickHouse is an ideal database for security, AI applications and agents

Architects and engineers face pressure when choosing a data stack for new AI features. Databases built for batch analytics are often too slow for real-time AI agents, while specialized vector databases are too niche for comprehensive analytical needs. A database that operates fundamentally as a compression algorithm solves both problems.

Alan Braithwaite, co-founder of the SIEM platform Run Reveal, shared: "Neural networks... they're more than just thinking machines. They're also a compression algorithm." AI models and ClickHouse share this core principle: both take mountains of raw data and transform it into a highly efficient, queryable state. Braithwaite built a petabyte-scale, AI-powered security platform on this concept.

TL;DR

• ClickHouse and AI share a foundational principle: highly efficient data compression. ClickHouse is the fastest open-source real-time analytics database.
• LLMs use lossy compression to store knowledge. ClickHouse uses lossless compression via its MergeTree engine to scan and query petabytes of data in sub-second times.
• Traditional architectures, such as SIEMs (Elastic, Splunk) and batch data warehouses (Snowflake, Databricks), work well for standard BI and daily reporting but can introduce write amplification, higher storage overhead, or latency that bottlenecks real-time AI agents.
• By natively handling complex JSON, providing a vector database alternative, and executing performant JOINs at scale, ClickHouse reduces LLM context payloads and acts as a pre-processing engine for AI.

How compression makes ClickHouse and AI models efficient

The comparison between a database and a neural network hinges on the distinction between lossy and lossless compression.

AI models, particularly LLMs, act as lossy compression. They ingest terabytes of unstructured information and distill it into a smaller model file containing weights and biases. Querying the model "decompresses" an answer. The model cannot reproduce its source text verbatim, but it retains the core meaning.

ClickHouse provides lossless compression. This allows users to fully reconstruct data without losing detail. It achieves this through its MergeTree engine, which uses concepts from Log-Structured Merge-Trees (LSMT) and columnar storage.

In an LSMT model, the engine rapidly writes incoming data to immutable parts on disk, and continuously merges them in the background. Before writing data, ClickHouse sorts it based on a defined primary key. Storing these sorted, similar values together in columns makes them highly compressible.

Higher compression means less data to scan from disk. This translates directly to faster queries and significantly lower compute costs.

How Run Reveal built a petabyte-scale AI security platform on ClickHouse

Braithwaite’s journey with ClickHouse began with skepticism. Working at Cloudflare in 2016, he was wary of a crowded database market and concerned about ClickHouse's eventually consistent nature.

His perspective shifted when he observed ClickHouse power internal DNS analytics. The database delivered fast queries over massive datasets, and its high availability outweighed his initial concerns about consistency. This experience led him to champion its adoption at Segment for product observability, where a hack week project proved its real-time ingestion capabilities by processing 300,000 events per second.

This background culminated in the founding of Run Reveal, a security data platform built around what Braithwaite identified as the number one pain point for security teams: data management. Traditional SIEM platforms like Splunk and Elastic weren't built for the query volume and speed that modern, AI-assisted investigations demand, forcing teams to search across many terabytes of logs on engines slower than a purpose-built analytical database. By building on ClickHouse from day one, Run Reveal could lean on its compression and query performance to cut those searches down to just the signal an investigation needs.

Choosing ClickHouse Cloud allowed Run Reveal to bypass these legacy limitations. They scaled to handle millions of events per second and over four petabytes of compressed data without managing infrastructure.

Building an AI-native data platform with ClickHouse

Modern security data platforms use AI to combat alert fatigue, but an AI model is only as effective as the data it accesses. To perform well, an LLM must query terabytes of historical logs in seconds to retrieve context before making a decision.

Feeding a raw log firehose directly into an LLM is not feasible, as it would overwhelm the model's context window and escalate token costs.

Instead, ClickHouse acts as a pre-processing engine. By rapidly executing complex aggregations and precise analytical filters across petabytes of data, ClickHouse distills millions of raw events into a concise, high-signal summary. This reduces the context payload the LLM needs to process. This lowers latency and ensures accurate AI responses.

Native JSON support for real-time AI and security data

Security logs and event streams often arrive as complex, semi-structured JSON objects. ClickHouse handles this natively with its native JSON data type. This format provides the flexibility of a schemaless structure and the performance of columnar storage. It also avoids the performance pitfalls of traditional JSONB types. ClickHouse stores each JSON path as its own sub-column. This eliminates the need for cumbersome string parsing.

Lightweight updates and deletes allow for efficient data corrections and point-in-time removals to meet internal data governance requirements. The engine applies these changes immediately at query time via patch parts to provide read-time masking, then materializes them asynchronously during the standard background merge process.

SELECT
    json.path.to.field
FROM logs
WHERE ...

Using ClickHouse instead of a vector database

Integrating generative AI often leads engineering teams to adopt specialized vector databases to store and query embeddings. This creates a fragmented stack where developers must maintain one database for analytical metadata and another for semantic search.

ClickHouse eliminates this complexity by treating vectors as first-class data types alongside logs and structured event data. Applications can combine analytical filtering with vector similarity search in a single SQL query. AI agents can use ClickHouse to find semantic matches while strictly filtering by time ranges or event types without syncing data between isolated systems.

High-performance JOINs for AI and analytics workloads

A modern AI application often needs to join alert data with asset inventories and threat intelligence feeds. While denormalization remains standard engineering practice for optimal execution, ClickHouse's join optimizer performs well when operational flexibility or governance require joining tables at query time.

Recent join improvements include a sophisticated query planner, global join reordering, scale-oriented join algorithms, and dynamic adaptation to memory constraints. These updates ensure queries complete successfully. For a complete overview of supported algorithms, consult the official documentation or our deep dive on Faster JOINs in ClickHouse.

Handling high query concurrency for AI agents

An AI-powered system issues many concurrent queries from both human users and automated agents. The default max_concurrent_queries setting of 100 in ClickHouse is a configurable safeguard. ClickHouse Cloud abstracts this configuration by separating storage and compute. It dynamically scales to accommodate concurrency spikes without manual intervention.

Conclusion

Systems from AI models to databases rely on efficient compression. ClickHouse's speed and real-time capabilities make it an ideal data engine for powering AI-native applications.

Ready to build your own AI-powered application on a data engine built for real-time speed and scale? Start a free trial of ClickHouse Cloud today. Load your own data to test at a realistic scale, and benchmark the latency and compute cost against your existing architecture.

Frequently Asked Questions

How does ClickHouse's lossless compression differ from the lossy compression of AI models?

While AI models use lossy compression to distill unstructured data into weights that retain core meaning without verbatim reproduction, ClickHouse utilizes lossless compression to fully reconstruct original data. Using its MergeTree engine, incoming data is sorted and stored in highly compressible columnar formats, enabling exact detail retrieval alongside significantly faster queries and lower compute costs.

How does ClickHouse act as a pre-processing engine to optimize Large Language Model performance?

By executing complex aggregations across petabytes of historical data, this data engine distills millions of raw events into concise summaries before feeding them to a Large Language Model. Instead of overwhelming the context window with raw logs, this targeted pre-processing reduces payload sizes, drastically lowers token costs, and ensures AI agents return highly accurate responses with minimal latency.

How does ClickHouse process complex, semi-structured JSON security logs without sacrificing speed?

The native JSON data type maintains query speeds by automatically storing each JSON path as an individual sub-column instead of relying on slower JSONB formats. This architectural choice entirely removes the need for cumbersome string parsing. Engineering teams get the flexibility of schemaless ingestion while keeping the performance benefits of strict columnar storage.

Why should engineering teams choose ClickHouse over a specialized vector database for AI applications?

Choosing ClickHouse prevents the stack fragmentation inherent to specialized vector databases by treating embeddings as first-class data types alongside traditional logs and metrics. Rather than syncing data across isolated systems, developers can combine vector similarity searches with strict analytical filters—like time ranges or user IDs—within a single, unified SQL query.

How does ClickHouse scale to handle the high query concurrency generated by automated AI agents?

ClickHouse Cloud dynamically scales to absorb concurrency spikes from automated agents by separating storage and compute architectures. While open-source deployments include a configurable baseline limit for concurrent queries, the cloud environment completely abstracts this safeguard. This ensures heavy analytical workloads and high-frequency automated requests run simultaneously without requiring manual infrastructure intervention.