- Understand how the project’s views and tables land in ClickHouse.
- Load data with seeds and control the ClickHouse types and table layout.
- Configure a table model with a ClickHouse engine, sorting key and partitioning.
- Turn a table into an incremental model and pick an incremental strategy.
- Create a snapshot.
- Use ClickHouse materialized views.
Before you start
Follow the README of ClickHouse/jaffle-shop-clickhouse first. It explains how to set the project up with dbt Core 1.x, dbt OSS, dbt v2 or the dbt platform, how to point it at a local ClickHouse (docker) or ClickHouse Cloud, how to load the sample data withdbt seed, and how to run the first dbt build. Once dbt build completes successfully, come back here for the ClickHouse-specific examples and configurations.
After the README steps you should have two databases in ClickHouse:
raw: the six source tables loaded from CSV files bydbt seed(raw_customers,raw_orders,raw_items,raw_products,raw_stores,raw_supplies).jaffle_shop(theschemaof your profile): six staging views (stg_*) and seven mart tables (customers,orders,order_items,products,locations,supplies,metricflow_time_spine).
schema, replace jaffle_shop in the queries below with your value.
dbt Core 1.x, dbt OSS, dbt v2 and the dbt platform. Every command and model in this guide is the same on all of them. The examples were tested with dbt Core 1.12 with
dbt-clickhouse 1.10 and with dbt OSS 2.0 against ClickHouse 26.8; dbt v2 runs the same adapter, and the dbt platform runs dbt v2. The console output shown is from dbt Core 1.x, and the few places where the engines behave differently are called out. See the dbt OSS, dbt v2 and dbt platform page for the current status of the v2 adapter, and Connect ClickHouse in the dbt documentation to get started on the dbt platform.clickhouse client, the ClickHouse Cloud SQL console or the SQL client of your choice.
How the project is materialized
The Jaffle Shop configures its materializations indbt_project.yml: staging models are views and marts are tables.
CREATE OR REPLACE VIEW statement on every run. It stores no data, so it costs nothing to build, but every query against it runs the model’s SQL against the source tables. ClickHouse keeps the compiled SQL of the model in the view definition:
INSERT INTO ... SELECT with the model’s SQL and atomically exchanges it with the previous version. Query performance is much better than a view, at the cost of storage and of rebuilding the whole table every time. Look at the table dbt created for the orders mart:
MergeTree, and it doesn’t declare a sorting key, so the adapter uses ORDER BY tuple(), meaning the data isn’t sorted at all. This is fine for a sample project, but for a real table you’ll want to choose both, which is what the next sections do. The materializations page lists every table configuration the adapter supports.
Loading data with seeds
The Jaffle Shop uses dbt seeds to load its raw data from the CSV files inseeds/jaffle-data. Seeds are meant for small, static reference data (code tables, mappings), not for loading a warehouse; the project uses them for convenience so you can get going without another ingestion tool, which is why the seeds are disabled unless you pass --vars '{"load_source_data": true}'.
Seeds are still a good place to learn how dbt creates ClickHouse tables. dbt infers a column type for each CSV column, and the inferred types differ between engines:
When the type matters, pin it with
column_types. The project already does this for the opened_at column of the raw_stores seed in dbt_project.yml:
engine, order_by and partition_by. For example, to sort the raw_orders seed by order time and partition it by month, add a properties file next to the CSVs, seeds/jaffle-data/_raw_orders.yml:
Use a properties file for these ClickHouse seed configurations rather than
+order_by or +engine keys under seeds: in dbt_project.yml. dbt Core 1.x accepts both forms, but dbt v2 only recognizes them in a properties file and rejects the dbt_project.yml keys with Unrecognized key ... Custom keys must go under +meta.dbt seed --full-refresh drops and recreates the table, so run it before you build anything that depends on the table’s data directly, such as the materialized view later in this guide.
Configuring a table for ClickHouse
Theorders mart is the natural place to start: it’s queried by the customers mart and by the project’s metrics, and it’s an event-style table with a timestamp. Add a config block at the top of models/marts/orders.sql to choose the engine, the sorting key and a partitioning scheme:
materialized='table' repeats what dbt_project.yml already says for marts, which keeps the model self-describing when you switch it to incremental later. Rebuild only this model:
engine, order_by and partition_by, table models accept primary_key, ttl, settings, query_settings, projections and indexes, and columns can carry codec and ttl through a model contract. They’re all described in the materializations page.
Creating an incremental model
Rebuildingorders from scratch on every run is fine for 62,000 rows, but not for a table that grows by millions of rows a day. dbt’s incremental materialization only processes the rows that changed since the last run. Converting the orders model requires two additions:
unique_key: the column that identifies a row,order_idhere. The adapter uses it to replace rows that are processed again instead of duplicating them.- An incremental filter: a
whereclause wrapped in{% if is_incremental() %}that selects only the rows to process. It’s applied on incremental runs but not when the table is first built (or rebuilt with--full-refresh). Orders carry a timestamp, so the filter comparesordered_atagainst the latest value already in the table, referenced through the{{ this }}variable.
models/marts/orders.sql so the config block and the end of the model look like this:
stg_orders truncates ordered_at to the day, so the filter uses >=: on every run the whole latest day is processed again and, thanks to unique_key, the rows already loaded are replaced rather than duplicated. That’s what makes it safe for orders that arrive later on the same day.
Run the model. The table already exists, so this first run is already an incremental one: only the latest day is re-processed.
nutellaphone who dis? jaffle at 11.00 and the tax is Philadelphia’s 6%, so the project’s data tests still pass. Run the whole project so the staging views and the order_items table see the new rows before orders does:
customers mart, rebuilt from it, knows about the new customer:
Internals
ClickHouse’s query log shows the statements the adapter ran for the incremental update:- A table
orders__dbt_new_datais created and the model’s SQL, including the incremental filter, is inserted into it. In the run above, 378 rows were written: the 377 orders of the latest day already loaded plus the new one. - A table
orders__dbt_tmpis created with the same structure asorders, and all rows oforderswhoseorder_idisn’t inorders__dbt_new_dataare copied into it. - All rows of
orders__dbt_new_dataare inserted intoorders__dbt_tmp. Steps 2 and 3 are what replaces the rows of the latest day instead of duplicating them. orders__dbt_new_datais dropped.orders__dbt_tmpis swapped withordersusing an atomicEXCHANGE TABLESstatement (through an intermediate rename toorders__dbt_backup), soordersnow holds the new version.- The old version is dropped.
Append strategy
Theappend strategy inserts the rows selected by the model straight into the target table. No temporary tables are created and nothing is copied, so it’s as cheap as an incremental run can be. The price is that nothing is deduplicated either: if the incremental filter selects a row that’s already in the table, you get it twice. Use it for immutable, event-style data, and make sure the filter only selects genuinely new rows.
With the day-truncated ordered_at, that means switching the filter to >. Change the model:
orders is a single INSERT INTO jaffle_shop.orders ... SELECT ... with the model’s SQL and the incremental filter, and it wrote one row.
Delete and insert strategy
Historically ClickHouse has had only limited support for updates and deletes, in the form of asynchronous mutations. These can be extremely IO-intensive and should generally be avoided. ClickHouse 22.8 introduced lightweight deletes and ClickHouse 25.7 introduced lightweight updates. With these, the effect of a single delete or update statement is visible immediately from the user’s perspective even though it’s materialized asynchronously. Thedelete+insert strategy relies on lightweight deletes and is configured through the incremental_strategy parameter:
- A temporary table (
orders__dbt_new_data_<run_id>) is created and the rows selected by the model are inserted into it. - A
DELETEis issued againstordersfor everyorder_idpresent in the temporary table. - The rows of the temporary table are inserted into
orders. - The temporary table is dropped.
Insert overwrite strategy (experimental)
Theinsert_overwrite strategy replaces whole partitions, so it needs a partition_by configuration like the monthly one on orders. It performs the following steps:
- Create a staging table (
orders__dbt_new_data_<run_id>) with the same structure asorders. - Insert only the rows selected by the model into the staging table.
- List the partitions present in the staging table from
system.parts. - Replace exactly those partitions in
orderswithALTER TABLE ... REPLACE PARTITION ... FROMthe staging table. - Drop the staging table.
- It’s faster than the default strategy because it doesn’t copy the entire table.
- It’s safer than the other strategies because it doesn’t modify the original table until the INSERT operation completes successfully: in case of an intermediate failure, the original table isn’t modified.
- It implements the “partition immutability” data engineering best practice, which simplifies incremental and parallel data processing, rollbacks, etc.
microbatch strategy and on_schema_change.
Creating a snapshot
dbt snapshots record how the rows of a mutable table change over time, so analysts can look back at the state of the data at any point in the past. They implement type-2 slowly changing dimensions: each version of a row is stored with the interval during which it was valid. Thecustomers mart is a good candidate: count_lifetime_orders, lifetime_spend and customer_type all change every time a customer orders again. Before continuing, set the orders model back to the default incremental strategy from the incremental section (remove incremental_strategy='append' and change the filter back to >=), so orders placed later today are picked up.
Snapshots are defined in YAML since dbt 1.9. Create snapshots/customers_snapshot.yml:
check strategy compares the listed columns between the current snapshot and the source on every run and records a new version whenever any of them changed. If your model has a reliable “last updated” timestamp column, the timestamp strategy is cheaper: set strategy: timestamp and updated_at: <column>. The Jaffle Shop’s last_ordered_at is truncated to the day, so it wouldn’t catch a second order on the same day, which is why this example uses check.
Take the first snapshot:
generate_schema_name macro puts every relation in the target schema for non-production targets, so a schema config on the snapshot would only take effect with the prod target. It contains one row per customer, with the dbt bookkeeping columns dbt_valid_from and dbt_valid_to; the latter is NULL for the current version of a row:
orders and customers reflect the new order, then take a second snapshot:
dbt_valid_to, and the new version, now a returning customer with two orders, is open. Danny didn’t change, so his row is untouched:
customers_snapshot__snapshot_upsert and swaps it in with EXCHANGE TABLES (or a drop and rename where the server can’t exchange tables), so readers see either the previous or the new version of the snapshot. See the snapshot section of the materializations page for the configuration reference.
Using materialized views
Everything so far needs adbt run to bring new data into the models. ClickHouse materialized views work differently: they’re insert triggers. Every block of rows inserted into the source table is transformed by the view’s SELECT and written into a target table, with no scheduling involved. The adapter exposes them through the materialized_view materialization.
Create models/marts/daily_store_revenue.sql with the number of orders and the revenue per store and day, reading directly from the raw orders table:
engine and order_by apply to the target table. SummingMergeTree adds up the numeric columns of rows that share the same sorting key when it merges parts, which is exactly what a per-day, per-store aggregate needs.
_mv suffix, pointing at the target table with a TO clause. By default (catchup=True) the target table was also backfilled with the existing orders:
sum() and GROUP BY on purpose: SummingMergeTree only collapses rows with the same key when parts are merged in the background, so until then the two Brooklyn orders are two rows in the table. Always aggregate on read (or use FINAL) with summing and aggregating engines. Meanwhile the orders incremental model still has a single order for Danny until the next dbt run.
Later runs of dbt run keep the target table and its data and only update the view definition, with ALTER TABLE ... MODIFY QUERY when the change allows it, so it’s safe to keep the model in the project. dbt run --full-refresh rebuilds the target table and backfills it again (unless catchup is False). The materialized views page covers the rest: schema changes with on_schema_change, disabling the backfill with catchup, refreshable materialized views, several views feeding the same target and defining the target table as its own model.