How Databricks prepares incoming data
Pause
FOR CONTEXT
How real-time systems keep incoming data query-ready
Fresh-data path
Query-ready data
Newly
written
Ordered
raw data
Ordered pre-
aggregated
Query engine
Query
compute
Fresh
data
continuously
Fast
answers
continuously
1
Store in
columns
2
Order
data
3
Order and pre-
aggregate data
Databricks
Fresh-data path
Query-ready data
1
Zerobus
Ingest
Arrow Flight
streams
Liquid-clustered
Delta table
Incremental MV
Ordered
raw data
Ordered pre-
aggregated
NBBO
quotes
1M rows/s
2
Async
serverless
clustering
3
Async
serverless
MV refresh
Serverless SQL
warehouse
X-Small
≈16 worker vCPUs
Query
results