CostBench: Putting real-time analytics to the test

This is Part 1 of our CostBench series.
Other parts: Snowflake · Databricks · BigQuery · Redshift
Coming next: Why ClickHouse Cloud won

The path from fresh data to fast queries shapes performance per dollar #

A real-time analytics system serves queries over a dataset that is constantly changing. New rows keep arriving while users, applications, and agents query data that may already span billions or trillions of rows. Each new row must become query-ready while those queries continue to run.

CostBench measures the end-to-end performance and cost of real-time analytics across cloud data warehouses. Its first results show that query engines are only part of what separates these systems:

Systems vary dramatically in the cost and efficiency of carrying incoming data from arrival to query readiness.

Under sustained load, that work can shape performance per dollar as much as query execution itself: it carries its own direct cost, and the state it produces determines how much work remains for the query engine.

This round tested ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift Serverless using push-based ingestion. Fresh quotes kept arriving while aggregate and drill-down queries ran, so each system had to prepare new data and answer queries at the same time.

We’ll first show what makes data query-ready, then put the systems under continuous load. The results reveal a gap in both runtime and cost. From there, we’ll explain how CostBench combines those dimensions, and why the path from an incoming row to a current answer matters so much.

What makes data query-ready #

Consider a common analytical question about stock quotes: How many AAPL quote updates arrived each day?

The query filters for AAPL, groups its quotes by day, and counts the rows in each group:

SELECT day, COUNT(*)
FROM quotes
WHERE sym = 'AAPL'
GROUP BY day;

This is exactly the kind of query analytical systems are built to run fast.

Query-ready data is organized and prepared so the query engine reads less data and does less work at query time.

For our “AAPL a day” query, three steps help the engine read less data and do less work.

Store data by column #

Our query needs sym to find AAPL’s quotes and day to group them. It does not need any other column.

Columnar storage lets the engine read only those two columns and skip the rest. In the example below, that means reading 2 columns instead of 7.

Organize data for vectorized execution and chunk pruning #

Columnar layouts also help the engine to process the data it reads efficiently. With a column’s values together in memory, the engine can apply an operation to a chunk of values in a simple loop. This vectorized execution reduces the overhead of handling rows individually. Suitable operations can also use SIMD instructions, which process several values with one instruction.

Before reading a chunk of stored data, the engine checks its metadata and skips it when the metadata shows it cannot contain values matching the query’s filter. This is chunk pruning.

Our query already reads only sym and day columns. But in the unsorted example, AAPL quotes appear in all three chunks, so the engine must read all three.

Sorting by sym keeps each stock’s quotes together. For WHERE sym = 'AAPL', the engine now reads one AAPL chunk and skips the MSFT and NVDA chunks. That leaves three AAPL quotes to group by day and count.

Pre-aggregate to reduce query-time work #

At billions or trillions of rows, scanning and aggregating raw data can be too slow for interactive analytics. When those queries run repeatedly, as they often do in dashboards, the compute cost can also add up.

Pre-aggregations move that work out of each query and into data preparation. They maintain a much smaller set of summary rows, reducing both the data read and the grouping and aggregation performed at query time. The summaries retain the grouping dimensions and aggregate results the queries need, allowing the engine to combine prepared results instead of calculating them from individual rows.

In our example, pruning leaves three AAPL quotes to group by day and count. Pre-aggregation prepares those counts in advance: it groups quotes by sym and day and calculates COUNT(*) for each group. This turns nine quotes into four summary rows, each containing a count for one stock on one day.

Because queries may still filter the summary rows, pre-aggregated data also benefits from a pruning-friendly physical layout. Our summary rows are ordered by sym. For WHERE sym = 'AAPL', the engine reads only the two AAPL summary rows and skips the MSFT and NVDA rows. Those two rows already contain the daily counts - two quotes on D1 and one on D2 - so the engine no longer needs to group and count the individual AAPL quotes on every execution.

For time-sensitive use cases such as monitoring market activity, these summaries must stay current with the underlying quotes, ideally updating as new quotes become queryable.

For our AAPL query, columnar storage lets the engine skip unused fields, sorting by sym lets it skip unrelated stocks, and current pre-aggregations let it skip grouping and counting individual quotes. Together, these preparations make data query-ready for analytical queries like this one: the query engine reads less data and performs fewer calculations.

Real-time systems must make fresh data query-ready as it arrives #

With a fixed dataset, that preparation can happen before queries begin. In a real-time system, new rows arrive continuously while queries keep running.

“A real-time system should have access to real-time data so it's viable for making real-time decisions based on the freshest data.” — Instacart Engineering

We call the continuous work that carries each new row from arrival to query readiness the fresh-data path. It comprises three responsibilities that must remain active in parallel:

① Ingest fresh data into columnar storage as it arrives.

② Maintain pruning-friendly physical layouts for event-level and pre-aggregated data through sorting, clustering, partitioning, or equivalent structures.

③ Maintain current pre-aggregations as new data arrives, keeping common grouping and aggregation work out of query time.

These are logical responsibilities, not a strict execution sequence. A system may fuse them in a single write path - for example, sorting incoming rows, computing pre-aggregations, and sorting those results before either representation is written to storage. Other systems complete some of this work asynchronously after ingestion.

Indexes, metadata, compaction, and other provider-specific structures also affect the cost and efficiency of this work.

Alongside the fresh-data path, the query engine serves two typical analytical paths. Both depend on the preparation described above:

• Pre-aggregated path: Interactive aggregate queries depend on ③ to keep repeated grouping and aggregation out of query time. If they filter the summarized rows, they also depend on ② to prune irrelevant chunks.

• Event-level path: Selective drill-down queries read event-level data directly and therefore depend on ② to avoid scanning most of the data. Even when these queries take seconds, pruning is what makes them feasible at scale; they may still group or aggregate the remaining rows.

An efficient fresh-data path lowers its own cost - and the work left for queries #

A system can sustain ingestion while its query readiness falls behind.

Keeping up with incoming data does not guarantee that the system is keeping up with everything needed to query it efficiently.

That gap can affect answer freshness: new stock quotes may be ingested promptly, yet a risk query can still return results based on an outdated view of the market.

It can also affect query efficiency, even when queries can see the latest data. If ordering or pre-aggregation falls behind, the query engine may need to scan more data, reconcile newer rows, or repeat grouping and aggregation at query time.

The interactive diagram below shows this second effect while data keeps flowing. Watch the loop or drag the slider to see how the query engine’s scanning and aggregation work shrinks as ordering and pre-aggregation catch up - and grows when preparation falls behind.

That extra query work consumes read-side compute, making queries more expensive and answers slower. An efficient fresh-data path therefore pays off in three ways: lower preparation cost, lower query cost, and faster answers.

CostBench measures preparation cost, query cost, and runtime under continuous load #

CostBench follows the same continuously growing dataset from ingestion through row visibility, event-level layout maintenance, and pre-aggregation freshness, while aggregate and drill-down queries run on a fixed schedule. The query definitions stay fixed, but their answers must continually account for newly arriving data. This makes both sides visible: the direct cost of keeping data query-ready and the performance and cost of serving queries from the state the system actually achieved.

The first results show that some systems sustained ingestion while pre-aggregation lag accumulated or query latency rose - moving more work onto the read path and driving up query-serving cost.

How CostBench keeps the comparison fair #

This round used push-based ingestion across ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift Serverless: one shared client sent the stream directly through each system’s recommended low-latency ingest path. Four controls kept the comparison like-for-like while respecting each system’s architecture:

• Same workload: Each system received the same 113.2-billion-row National Best Bid and Offer (NBBO) stock-market quotes dataset, schema, layout intent, target ingestion rate of 1 million rows per second, queries, and cadence.

• Comparable resources: We aligned read-side compute where a meaningful comparison was possible and used the smallest tested configurable fresh-data-path compute that sustained the target during calibration.

• Complete platform paths: Each system used its recommended real-time architecture with a low latency push-based ingestion path. Managed and serverless services retained their native scaling, and their metered work remained in the benchmark’s cost.

• One shared harness: The same source files, decoding logic, client host, rate controller, scheduler, timing, and recording logic drove every system; only the destination adapter changed.

We configured matching sorting or clustering keys across systems for both raw data and daily pre-aggregations. These layouts support the workload’s stock-symbol filters. The pre-aggregations group quotes by stock symbol and day, maintaining the counts, sums, and price minima and maxima that the aggregate queries use. Each system therefore had to keep the same preparation current for the same query workload as new rows arrived.

Later rounds may also test pull-based variants using one shared external stream, to give a more complete picture of real-time performance per dollar.

The shared harness holds the workload constant. CostBench measures how each system handles it.

The full benchmark definition, accepted runs, results, cost summaries, and reproduction steps are open in the CostBench repository.

The gap opens while the data is still arriving #

As the dataset grew toward 113.2 billion rows, the systems kept preparing fresh quotes and answering queries. The three charts below show what accumulated along the way: fresh-data-path cost, normalized query-serving cost, and query runtime.

ClickHouse Cloud stays near the bottom in all three. Across the accepted comparisons, it spent less keeping incoming data query-ready, less serving the measured queries, and less time answering them. Its advantage was already visible while ingestion was running.

Those three results are connected. Preparing data efficiently carries a direct cost, but it also changes the work left for the query engine. CostBench combines preparation cost, query-serving cost, and query runtime into one score to show their overall effect.

Turning runtime and cost into performance per dollar #

A fast query is useful; delivering it economically while new data keeps arriving is the full challenge. CostBench combines the three inputs shown above:

1. Fresh-data-path cost: ingestion, layout maintenance, and pre-aggregation during the complete ingest.
2. Normalized query-serving cost: the measured query workload priced at the applicable read-side compute rates.
3. Accumulated query runtime: the time spent answering accepted queries while ingestion continued.

The score multiplies combined cost by accumulated query runtime. Lower cost improves the score; faster queries improve it too. Each provider’s result is compared with its matched ClickHouse baseline.

The score rewards systems that keep incoming data query-ready economically and serve the workload quickly. Lower is better.

Database storage costs are excluded from the score because their impact is small over the benchmark’s duration; see the calculation note below.

The fresh-data path widens the performance-per-dollar gap #

We began with a thesis:

Under sustained load, keeping data query-ready can shape performance per dollar as much as query execution itself.

The results now show just how large that impact can be.

Query-side performance: a 23–101× advantage #

The earlier query-side comparison began with an already-loaded dataset and measured query-serving cost and runtime. ClickHouse led that comparison: Databricks, Snowflake, Redshift Serverless, and BigQuery Capacity had query-side scores 23–101× higher.

This round asks a broader question: what happens to performance per dollar when fresh data keeps arriving and each system must prepare it while serving queries?

End-to-end performance: a 412–1,996× advantage #

In the continuous-ingestion benchmark, Snowflake’s end-to-end score was 412× higher than its matched ClickHouse baseline; BigQuery Capacity’s was 438×, Databricks’ 752×, and Redshift Serverless’ 1,996×. Lower is better.

The larger range comes from a broader benchmark scope: data preparation and query serving running together. It shows how much the complete path can matter to performance per dollar.

Why the complete path matters #

The final chart brings runtime and cost together. ClickHouse Cloud combines the lowest accumulated query runtime with the lowest combined preparation and normalized query-serving cost in the accepted comparisons.

Its fresh-data path performs ordering and pre-aggregation as incoming blocks are inserted. Raw data and summaries advance together, so queries can use current, prepared data without waiting for a separate summary-refresh cycle.

That connects the result back to the AAPL example. Reading fewer columns, pruning irrelevant data, and reusing prepared aggregates all reduce query work.

A real-time system must keep those advantages as the dataset changes. ClickHouse Cloud did so at the lowest measured cost in these comparisons.

The provider deep dives examine where each system spends that preparation effort and what happens when it falls behind. The ClickHouse Cloud–Snowflake comparison follows the 412× result through ingestion, materialized-view refresh, query compilation, and billing.