Measuring real-time performance per dollar under continuous load: CostBench’s first end-to-end results

The path from fresh data to fast queries shapes performance per dollar #

A real-time analytics system never works on a finished dataset. New rows keep arriving while users, applications, and agents query data that may already span billions or trillions of rows. Each new row must become query-ready while those queries continue to run.

CostBench’s first results show that query engines are only part of what separates cloud data warehouses:

Systems vary dramatically in the cost and efficiency of carrying incoming data from arrival to query readiness.

Under sustained load, that work can shape performance per dollar as much as query execution itself: it carries its own direct cost, and the state it produces determines how much work remains for the query engine.

This first end-to-end round tested ClickHouse Cloud, Snowflake, BigQuery, and Redshift Serverless using push-based ingestion.

CostBench measured the complete workload-ingestion, ongoing query-readiness work, and aggregate and drill-down query serving-all running together. This post first explains what query-ready means, then shows how the benchmark measures both effects and presents the overall results. The 1:1 posts that follow trace those results through each provider’s architecture and billing model.

What makes data query-ready #

Consider a common analytical pattern: filter a contiguous range of rows, such as all events for a specific day in a web analytics table, then group and aggregate the results:

SELECT
    URL,
    COUNT(*) AS pageviews,
    COUNT(DISTINCT User) AS users
FROM hits
WHERE Day = 'D2'
GROUP BY URL;

This is exactly the kind of query analytical systems are built to run fast. To do so, they keep data query-ready by minimizing how much data the query engine must read and how much repeated work it must do.

For a typical analytical query like this one, that translates into three requirements: store data by column, organize it for chunk pruning, and pre-aggregate repeated calculations.

Store data by column #

Most analytical queries do not need every column in a table. They may filter by one column, group by another, and calculate aggregates from a third. The example query uses only Day, User, and URL.

Columnar storage lets the engine read those three columns directly and skip every other column, even if the table contains hundreds.

Organize data for chunk pruning #

Analytical engines process column values in chunks, which enables efficient vectorized execution.

Chunks are also the unit of pruning. Before reading a chunk, the engine checks its metadata and skips it when its values fall outside the query’s filter. Pruning works best when the physical layout keeps similar values together, for example by sorting on the filtered column. In the example below, ordering by Day lets the engine read only the D2 chunk and skip the D1 and D3 chunks.

Pre-aggregate repeated calculations #

At billions or trillions of rows, scanning and aggregating the raw data for every query is too slow for interactive analytics.

Pre-aggregations move that repeated work out of each query. They maintain a much smaller set of summarized rows, reducing both the data read and the grouping and aggregation performed at query time. The summaries retain the grouping dimensions and aggregate results the queries need, allowing the query engine to combine prepared results instead of repeating those calculations over individual events.

For time-sensitive use cases such as fraud detection, the pre-aggregated rows must also remain current with the event-level data, ideally updating at the same time new events become queryable.

Because analytical queries often still filter these summarized rows, pre-aggregated data also benefits from a pruning-friendly physical layout.

The example below turns nine events into three daily rows ordered by Day. For WHERE Day = 'D1', the engine reads only the D1 row and skips the D2 and D3 rows.

Together, columnar storage, pruning-friendly layouts, and current pre-aggregations make analytical data query-ready.

Real-time systems must make fresh data query-ready as it arrives #

With a fixed dataset, that preparation can happen before queries begin. In a real-time system, new rows arrive continuously while queries keep running.

“A real-time system should have access to real-time data so it's viable for making real-time decisions based on the freshest data.” — Instacart Engineering

We call the continuous work that carries each new row from arrival to query readiness the fresh-data path. It comprises three responsibilities that must remain active in parallel:

① Ingest fresh data into columnar storage as it arrives.

② Maintain pruning-friendly physical layouts for event-level and pre-aggregated data through sorting, clustering, partitioning, or equivalent structures.

③ Maintain current pre-aggregations as new data arrives, keeping common grouping and aggregation work out of query time.

These are logical responsibilities, not a strict execution sequence. A system may fuse them in a single write path - for example, sorting incoming rows, computing pre-aggregations, and sorting those results before either representation is written to storage. Other systems complete some of this work asynchronously after ingestion.

Indexes, metadata, compaction, and other provider-specific structures also affect the cost and efficiency of this work.

Alongside the fresh-data path, the query engine serves two typical analytical paths. Both depend on the preparation described above:

• Pre-aggregated path: Interactive aggregate queries depend on ③ to keep repeated grouping and aggregation out of query time. If they filter the summarized rows, they also depend on ② to prune irrelevant chunks.

• Event-level path: Selective drill-down queries read event-level data directly and therefore depend on ② to avoid scanning most of the data. Even when these queries take seconds, pruning is what makes them feasible at scale; they may still group or aggregate the remaining rows.

An efficient fresh-data path lowers its own cost - and the work left for queries #

A system can sustain ingestion while its query readiness falls behind.

That can affect:

• Answer freshness: A suspicious payment may be ingested immediately yet reach the fraud query too late.
• Query efficiency: When layout or pre-aggregation work remains unfinished, queries must scan more data, reconcile newer rows, or repeat grouping and aggregation at query time.

That extra query work increases:

• latency
• read-side compute
• query-serving cost.

An efficient fresh-data path therefore reduces both the direct cost of preparing data and the runtime and cost of the queries it serves.

CostBench measures both effects together #

CostBench follows the same continuously growing dataset from ingestion through row visibility, event-level layout maintenance, and pre-aggregation freshness, while aggregate and drill-down queries run on a fixed schedule. The query definitions stay fixed, but their answers must continually account for newly arriving data. This makes both sides visible: the direct cost of keeping data query-ready and the performance and cost of serving queries from the state the system actually achieved.

The first results show that some systems sustained ingestion while pre-aggregation lag accumulated or query latency rose - moving more work onto the read path and driving up query-serving cost.

How CostBench keeps the comparison fair #

This round used push-based ingestion across ClickHouse Cloud, Snowflake, BigQuery, and Redshift Serverless: one shared client sent the stream directly through each system’s recommended low-latency ingest path. Four controls kept the comparison like-for-like while respecting each system’s architecture:

• Same workload: Each system received the same 113.2-billion-row National Best Bid and Offer (NBBO) stock-market quotes dataset, schema, layout intent, target ingestion rate of 1 million rows per second, queries, and cadence.

• Comparable resources: We aligned read-side compute where a meaningful comparison was possible and used the smallest tested configurable fresh-data-path compute that sustained the target during calibration.

• Complete platform paths: Each system used its recommended real-time architecture with a low latency push-based ingestion path. Managed and serverless services retained their native scaling, and their metered work remained in the benchmark’s cost.

• One shared harness: The same source files, decoding logic, client host, rate controller, scheduler, timing, and recording logic drove every system; only the destination adapter changed.

We configured matching sorting or clustering keys across systems for both raw data and daily pre-aggregations. These layouts support the workload’s stock-symbol filters. The pre-aggregations group quotes by stock symbol and day, maintaining the counts, sums, and price minima and maxima that the aggregate queries use. Each system therefore had to keep the same preparation current for the same query workload as new rows arrived.

Later rounds may also test pull-based variants using one shared external stream, to give a more complete picture of real-time performance per dollar.

The shared harness holds the workload constant. CostBench measures how each system handles it.

The full benchmark definition, accepted runs, results, cost summaries, and reproduction steps are open in the CostBench repository.

Databricks Lakehouse/RT remains in beta, so it is outside this first comparison and will be tested after general availability.

CostBench ranks systems by real-time performance per dollar #

Cloud platforms pair fundamentally different billing models with often radically different fresh-data paths and query engines. To compare cost and performance across them, CostBench combines three inputs into one lower-is-better score:

① Fresh-data-path cost: Complete-ingest cost for continuous ingestion, event-level layout maintenance, and pre-aggregation.

② Normalized query-serving cost: Cost of executing the same scheduled queries at each system’s applicable read-side rate.

③ Total query runtime: Accumulated end-to-end runtime for that query workload during active ingestion.

The score rewards systems that keep incoming data query-ready economically and serve the workload quickly. Lower is better.

Database storage costs are excluded from the score because their impact is small over the benchmark’s duration; see the calculation note below.

The fresh-data path widens the performance-per-dollar gap #

We began with a thesis:

Under sustained load, keeping data query-ready can shape performance per dollar as much as query execution itself.

The results now show just how large that impact can be.

Query-side only: a 32–101× gap #

The earlier query-side comparison measured already-loaded, query-ready data, isolating query-serving cost and runtime.

Within that boundary, Snowflake, Redshift Serverless, and BigQuery Capacity delivered 32× to 101× worse query-side cost-performance than ClickHouse.

With the fresh-data path: a 412–1,996× gap #

The end-to-end comparison adds the continuous cost of ingesting new rows and maintaining their query-ready state. The benchmark adds that cost to normalized query-serving cost and combines the total with accumulated query runtime during active ingestion.

The result closes the loop:

Once the fresh-data path is included, the performance-per-dollar gap expands from 32–101× on the isolated query side to 412–1,996× end to end.

The final animation separates the two dimensions behind that wider score: end-to-end cost and accumulated query runtime.

ClickHouse Cloud is purpose-built end to end for fast answers on continuously fresh data - and the yellow point shows the effect on both sides of the score:

• Fresh-data path: As the 1:1 analyses will show, ClickHouse Cloud had the lowest-cost fresh-data path and was the only tested system that kept both its event-level and pre-aggregated data current as ingestion continued.

• Query serving: Because it completed that work as rows arrived, its query engine had less filtering, grouping, and aggregation work left to do - resulting in the lowest accumulated query runtime and the lowest normalized query-serving cost.

ClickHouse Cloud’s advantage compounded: the lowest-cost fresh-data path kept data query-ready as it arrived, leaving less work - and therefore less runtime and cost - for the measured query workload.

The CostBench end-to-end real-time path series continues with 1:1 analyses of each provider’s architecture and billing model, tracing where query-readiness work happens, where it falls behind, and how those choices shape performance per dollar.