│
└─────────────────┴──────────────────────┴───────────────────────────────────────────────────────────────┘
20 rows in set. Elapsed: 0.547 sec. Processed 7.88 million rows, 679.20 MB (14.42 million rows/s., 1.24 GB/s.)
```
We welcome exact and improved solutions here.
# GitHub events dataset
Source: https://clickhouse.com/docs/get-started/sample-datasets/github-events
Dataset containing all events on GitHub from 2011 to Dec 6 2020, with a size of 3.1 billion records.
Dataset contains all events on GitHub from 2011 to Dec 6 2020, the size is 3.1 billion records. Download size is 75 GB and it will require up to 200 GB space on disk if stored in a table with lz4 compression.
Full dataset description, insights, download instruction and interactive queries are posted in this [repository](https://github.com/ClickHouse/github-explorer)
# Hacker News dataset
Source: https://clickhouse.com/docs/get-started/sample-datasets/hacker-news
Dataset containing 28 million rows of hacker news data.
> In this tutorial, you'll insert 28 million rows of Hacker News data into a ClickHouse
> table from both CSV and Parquet formats and run some simple queries to explore the data.
## CSV
**1. Download CSV**
A CSV version of the dataset can be downloaded from our public [S3 bucket](https://datasets-documentation.s3.eu-west-3.amazonaws.com/hackernews/hacknernews.csv.gz), or by running this command:
```bash theme={null}
wget https://datasets-documentation.s3.eu-west-3.amazonaws.com/hackernews/hacknernews.csv.gz
```
At 4.6GB, and 28m rows, this compressed file should take 5-10 minutes to download.
**2. Sample the data**
[`clickhouse-local`](/docs/concepts/features/tools-and-utilities/clickhouse-local) allows you to perform fast processing on local files without
having to deploy and configure the ClickHouse server.
Before storing any data in ClickHouse, let's sample the file using clickhouse-local.
From the console run:
```bash theme={null}
clickhouse-local
```
Next, run the following command to explore the data:
```sql title="Query" theme={null}
SELECT *
FROM file('hacknernews.csv.gz', CSVWithNames)
LIMIT 2
SETTINGS input_format_try_infer_datetimes = 0
FORMAT Vertical
```
```response title="Response" theme={null}
Row 1:
──────
id: 344065
deleted: 0
type: comment
by: callmeed
time: 2008-10-26 05:06:58
text: What kind of reports do you need?ActiveMerchant just connects your app to a gateway for cc approval and processing.
Braintree has very nice reports on transactions and it's very easy to refund a payment.
Beyond that, you are dealing with Rails after all–it's pretty easy to scaffold out some reports from your subscriber base.
dead: 0
parent: 344038
poll: 0
kids: []
url:
score: 0
title:
parts: []
descendants: 0
Row 2:
──────
id: 344066
deleted: 0
type: story
by: acangiano
time: 2008-10-26 05:07:59
text:
dead: 0
parent: 0
poll: 0
kids: [344111,344202,344329,344606]
url: http://antoniocangiano.com/2008/10/26/what-arc-should-learn-from-ruby/
score: 33
title: What Arc should learn from Ruby
parts: []
descendants: 10
```
There are a lot of subtle capabilities in this command.
The [`file`](/docs/reference/functions/regular-functions/files#file) operator allows you to read the file from a local disk, specifying only the format `CSVWithNames`.
Most importantly, the schema is automatically inferred for you from the file contents.
Note also how `clickhouse-local` is able to read the compressed file, inferring the gzip format from the extension.
The `Vertical` format is used to more easily see the data for each column.
**3. Load the data with schema inference**
The simplest and most powerful tool for data loading is the `clickhouse-client`: a feature-rich native command-line client.
To load data, you can again exploit schema inference, relying on ClickHouse to determine the types of the columns.
Run the following command to create a table and insert the data directly from the remote CSV file, accessing the contents via the [`url`](/docs/reference/functions/table-functions/url) function.
The schema is automatically inferred:
```sql theme={null}
CREATE TABLE hackernews ENGINE = MergeTree ORDER BY tuple
(
) EMPTY AS SELECT * FROM url('https://datasets-documentation.s3.eu-west-3.amazonaws.com/hackernews/hacknernews.csv.gz', 'CSVWithNames');
```
This creates an empty table using the schema inferred from the data.
The [`DESCRIBE TABLE`](/docs/reference/statements/describe-table) command allows us to understand these assigned types.
```sql title="Query" theme={null}
DESCRIBE TABLE hackernews
```
```text title="Response" theme={null}
┌─name────────┬─type─────────────────────┬
│ id │ Nullable(Float64) │
│ deleted │ Nullable(Float64) │
│ type │ Nullable(String) │
│ by │ Nullable(String) │
│ time │ Nullable(String) │
│ text │ Nullable(String) │
│ dead │ Nullable(Float64) │
│ parent │ Nullable(Float64) │
│ poll │ Nullable(Float64) │
│ kids │ Array(Nullable(Float64)) │
│ url │ Nullable(String) │
│ score │ Nullable(Float64) │
│ title │ Nullable(String) │
│ parts │ Array(Nullable(Float64)) │
│ descendants │ Nullable(Float64) │
└─────────────┴──────────────────────────┴
```
To insert the data into this table, use the `INSERT INTO, SELECT` command.
Together with the `url` function, data will be streamed directly from the URL:
```sql theme={null}
INSERT INTO hackernews SELECT *
FROM url('https://datasets-documentation.s3.eu-west-3.amazonaws.com/hackernews/hacknernews.csv.gz', 'CSVWithNames')
```
You've successfully inserted 28 million rows into ClickHouse with a single command!
**4. Explore the data**
Sample the Hacker News stories and specific columns by running the following query:
```sql title="Query" theme={null}
SELECT
id,
title,
type,
by,
time,
url,
score
FROM hackernews
WHERE type = 'story'
LIMIT 3
FORMAT Vertical
```
```response title="Response" theme={null}
Row 1:
──────
id: 2596866
title:
type: story
by:
time: 1306685152
url:
score: 0
Row 2:
──────
id: 2596870
title: WordPress capture users last login date and time
type: story
by: wpsnipp
time: 1306685252
url: http://wpsnipp.com/index.php/date/capture-users-last-login-date-and-time/
score: 1
Row 3:
──────
id: 2596872
title: Recent college graduates get some startup wisdom
type: story
by: whenimgone
time: 1306685352
url: http://articles.chicagotribune.com/2011-05-27/business/sc-cons-0526-started-20110527_1_business-plan-recession-college-graduates
score: 1
```
While schema inference is a great tool for initial data exploration, it is “best effort” and not a long-term substitute for defining an optimal schema for your data.
**5. Define a schema**
An obvious immediate optimization is to define a type for each field.
In addition to declaring the time field as a `DateTime` type, we define an appropriate type for each of the fields below after dropping our existing dataset.
In ClickHouse the primary key id for the data is defined via the `ORDER BY` clause.
Selecting appropriate types and choosing which columns to include in the `ORDER BY`
clause will help to improve query speed and compression.
Run the query below to drop the old schema and create the improved schema:
```sql title="Query" theme={null}
DROP TABLE IF EXISTS hackernews;
CREATE TABLE hackernews
(
`id` UInt32,
`deleted` UInt8,
`type` Enum('story' = 1, 'comment' = 2, 'poll' = 3, 'pollopt' = 4, 'job' = 5),
`by` LowCardinality(String),
`time` DateTime,
`text` String,
`dead` UInt8,
`parent` UInt32,
`poll` UInt32,
`kids` Array(UInt32),
`url` String,
`score` Int32,
`title` String,
`parts` Array(UInt32),
`descendants` Int32
)
ENGINE = MergeTree
ORDER BY id
```
With an optimized schema, you can now insert the data from the local file system.
Again using `clickhouse-client`, insert the file using the `INFILE` clause with an explicit `INSERT INTO`.
```sql title="Query" theme={null}
INSERT INTO hackernews FROM INFILE '/data/hacknernews.csv.gz' FORMAT CSVWithNames
```
**6. Run sample queries**
Some sample queries are presented below to give you inspiration for writing your
own queries.
#### How pervasive a topic is "ClickHouse" in Hacker News?
The score field provides a metric of popularity for stories, while the `id` field and `||`
concatenation operator can be used to produce a link to the original post.
```sql title="Query" theme={null}
SELECT
time,
score,
descendants,
title,
url,
'https://news.ycombinator.com/item?id=' || toString(id) AS hn_url
FROM hackernews
WHERE (type = 'story') AND (title ILIKE '%ClickHouse%')
ORDER BY score DESC
LIMIT 5 FORMAT Vertical
```
```response title="Response" theme={null}
Row 1:
──────
time: 1632154428
score: 519
descendants: 159
title: ClickHouse, Inc.
url: https://github.com/ClickHouse/ClickHouse/blob/master/website/blog/en/2021/clickhouse-inc.md
hn_url: https://news.ycombinator.com/item?id=28595419
Row 2:
──────
time: 1614699632
score: 383
descendants: 134
title: ClickHouse as an alternative to Elasticsearch for log storage and analysis
url: https://pixeljets.com/blog/clickhouse-vs-elasticsearch/
hn_url: https://news.ycombinator.com/item?id=26316401
Row 3:
──────
time: 1465985177
score: 243
descendants: 70
title: ClickHouse – high-performance open-source distributed column-oriented DBMS
url: https://clickhouse.yandex/reference_en.html
hn_url: https://news.ycombinator.com/item?id=11908254
Row 4:
──────
time: 1578331410
score: 216
descendants: 86
title: ClickHouse cost-efficiency in action: analyzing 500B rows on an Intel NUC
url: https://www.altinity.com/blog/2020/1/1/clickhouse-cost-efficiency-in-action-analyzing-500-billion-rows-on-an-intel-nuc
hn_url: https://news.ycombinator.com/item?id=21970952
Row 5:
──────
time: 1622160768
score: 198
descendants: 55
title: ClickHouse: An open-source column-oriented database management system
url: https://github.com/ClickHouse/ClickHouse
hn_url: https://news.ycombinator.com/item?id=27310247
```
Is ClickHouse generating more noise over time? Here the usefulness of defining the `time` field
as a `DateTime` is shown, as using a proper data type allows you to use the `toYYYYMM()` function:
```sql title="Query" theme={null}
SELECT
toYYYYMM(time) AS monthYear,
bar(count(), 0, 120, 20)
FROM hackernews
WHERE (type IN ('story', 'comment')) AND ((title ILIKE '%ClickHouse%') OR (text ILIKE '%ClickHouse%'))
GROUP BY monthYear
ORDER BY monthYear ASC
```
```response title="Response" theme={null}
┌─monthYear─┬─bar(count(), 0, 120, 20)─┐
│ 201606 │ ██▎ │
│ 201607 │ ▏ │
│ 201610 │ ▎ │
│ 201612 │ ▏ │
│ 201701 │ ▎ │
│ 201702 │ █ │
│ 201703 │ ▋ │
│ 201704 │ █ │
│ 201705 │ ██ │
│ 201706 │ ▎ │
│ 201707 │ ▎ │
│ 201708 │ ▏ │
│ 201709 │ ▎ │
│ 201710 │ █▌ │
│ 201711 │ █▌ │
│ 201712 │ ▌ │
│ 201801 │ █▌ │
│ 201802 │ ▋ │
│ 201803 │ ███▏ │
│ 201804 │ ██▏ │
│ 201805 │ ▋ │
│ 201806 │ █▏ │
│ 201807 │ █▌ │
│ 201808 │ ▋ │
│ 201809 │ █▌ │
│ 201810 │ ███▌ │
│ 201811 │ ████ │
│ 201812 │ █▌ │
│ 201901 │ ████▋ │
│ 201902 │ ███ │
│ 201903 │ ▋ │
│ 201904 │ █ │
│ 201905 │ ███▋ │
│ 201906 │ █▏ │
│ 201907 │ ██▎ │
│ 201908 │ ██▋ │
│ 201909 │ █▋ │
│ 201910 │ █ │
│ 201911 │ ███ │
│ 201912 │ █▎ │
│ 202001 │ ███████████▋ │
│ 202002 │ ██████▌ │
│ 202003 │ ███████████▋ │
│ 202004 │ ███████▎ │
│ 202005 │ ██████▏ │
│ 202006 │ ██████▏ │
│ 202007 │ ███████▋ │
│ 202008 │ ███▋ │
│ 202009 │ ████ │
│ 202010 │ ████▌ │
│ 202011 │ █████▏ │
│ 202012 │ ███▋ │
│ 202101 │ ███▏ │
│ 202102 │ █████████ │
│ 202103 │ █████████████▋ │
│ 202104 │ ███▏ │
│ 202105 │ ████████████▋ │
│ 202106 │ ███ │
│ 202107 │ █████▏ │
│ 202108 │ ████▎ │
│ 202109 │ ██████████████████▎ │
│ 202110 │ ▏ │
└───────────┴──────────────────────────┘
```
It looks like "ClickHouse" is growing in popularity with time.
#### Who are the top commenters on ClickHouse related articles?
```sql title="Query" theme={null}
SELECT
by,
count() AS comments
FROM hackernews
WHERE (type IN ('story', 'comment')) AND ((title ILIKE '%ClickHouse%') OR (text ILIKE '%ClickHouse%'))
GROUP BY by
ORDER BY comments DESC
LIMIT 5
```
```response title="Response" theme={null}
┌─by──────────┬─comments─┐
│ hodgesrm │ 78 │
│ zX41ZdbW │ 45 │
│ manigandham │ 39 │
│ pachico │ 35 │
│ valyala │ 27 │
└─────────────┴──────────┘
```
#### Which comments generate the most interest?
```sql title="Query" theme={null}
SELECT
by,
sum(score) AS total_score,
sum(length(kids)) AS total_sub_comments
FROM hackernews
WHERE (type IN ('story', 'comment')) AND ((title ILIKE '%ClickHouse%') OR (text ILIKE '%ClickHouse%'))
GROUP BY by
ORDER BY total_score DESC
LIMIT 5
```
```response title="Response" theme={null}
┌─by───────┬─total_score─┬─total_sub_comments─┐
│ zX41ZdbW │ 571 │ 50 │
│ jetter │ 386 │ 30 │
│ hodgesrm │ 312 │ 50 │
│ mechmind │ 243 │ 16 │
│ tosh │ 198 │ 12 │
└──────────┴─────────────┴────────────────────┘
```
## Parquet
One of the strengths of ClickHouse is its ability to handle any number of [formats](/docs/reference/formats/index).
CSV represents a rather ideal use case, and isn't the most efficient for data exchange.
Next, you'll load the data from a Parquet file which is an efficient column-oriented format.
Parquet has minimal types, which ClickHouse needs to respect, and this type information is encoded in the format itself.
Type inference on a Parquet file will invariably lead to a slightly different schema than the one for the CSV file.
**1. Insert the data**
Run the following query to read the same data in Parquet format, again using the `url` function to read the remote data:
```sql theme={null}
DROP TABLE IF EXISTS hackernews;
CREATE TABLE hackernews
ENGINE = MergeTree
ORDER BY id
SETTINGS allow_nullable_key = 1 EMPTY AS
SELECT *
FROM url('https://datasets-documentation.s3.eu-west-3.amazonaws.com/hackernews/hacknernews.parquet', 'Parquet');
INSERT INTO hackernews SELECT *
FROM url('https://datasets-documentation.s3.eu-west-3.amazonaws.com/hackernews/hacknernews.parquet', 'Parquet');
```
> [!NOTE]
>
> **Null keys with Parquet**
>
> The inferred schema makes the columns `Nullable`, so `allow_nullable_key` is required even though this dataset contains
> no null IDs.
Run the following command to view the inferred schema:
```sql title="Query" theme={null}
DESCRIBE TABLE hackernews;
```
```response title="Response" theme={null}
┌─name────────┬─type───────────────────┬
│ id │ Nullable(Int64) │
│ deleted │ Nullable(UInt8) │
│ type │ Nullable(String) │
│ by │ Nullable(String) │
│ time │ Nullable(Int64) │
│ text │ Nullable(String) │
│ dead │ Nullable(UInt8) │
│ parent │ Nullable(Int64) │
│ poll │ Nullable(Int64) │
│ kids │ Array(Nullable(Int64)) │
│ url │ Nullable(String) │
│ score │ Nullable(Int32) │
│ title │ Nullable(String) │
│ parts │ Array(Nullable(Int64)) │
│ descendants │ Nullable(Int32) │
└─────────────┴────────────────────────┴
```
The remaining steps use clearer column names such as `author` and `comment`, so continue with a manually specified schema.
First drop the inferred table, then create the table and insert the data directly from the public S3 bucket:
```sql theme={null}
DROP TABLE IF EXISTS hackernews;
CREATE TABLE hackernews
(
`id` UInt64,
`deleted` UInt8,
`type` String,
`author` String,
`timestamp` DateTime,
`comment` String,
`dead` UInt8,
`parent` UInt64,
`poll` UInt64,
`children` Array(UInt32),
`url` String,
`score` UInt32,
`title` String,
`parts` Array(UInt32),
`descendants` UInt32
)
ENGINE = MergeTree
ORDER BY (type, author);
INSERT INTO hackernews
SELECT * FROM s3(
'https://datasets-documentation.s3.eu-west-3.amazonaws.com/hackernews/hacknernews.parquet',
NOSIGN,
'Parquet',
'id UInt64,
deleted UInt8,
type String,
by String,
time DateTime,
text String,
dead UInt8,
parent UInt64,
poll UInt64,
kids Array(UInt32),
url String,
score UInt32,
title String,
parts Array(UInt32),
descendants UInt32');
```
**2. Add a text index to speed up searches**
To find out how many comments mention "ClickHouse", run the following query:
```sql title="Query" theme={null}
SELECT count(*)
FROM hackernews
WHERE hasAnyTokens(lower(comment), 'clickhouse');
```
```response title="Response" highlight={1} theme={null}
┌─count()─┐
│ 1145 │
└─────────┘
1 row in set. Elapsed: 3.251 sec. Processed 28.74 million rows, 9.60 GB (8.84 million rows/s., 2.95 GB/s.)
```
Next, create a [text index](/docs/reference/engines/table-engines/mergetree-family/textindexes) on the `comment` column
to speed up this query. A text index uses an inverted index that maps tokens to the rows that contain them.
The `splitByNonAlpha` tokenizer splits text on non-alphanumeric characters. The index and queries use `lower(comment)`
with lowercase search terms so matching is case-insensitive. The query expression must match the indexed expression.
Run the following commands to create the index:
```sql theme={null}
ALTER TABLE hackernews
ADD INDEX comment_idx lower(comment)
TYPE text(tokenizer = splitByNonAlpha);
ALTER TABLE hackernews
MATERIALIZE INDEX comment_idx
SETTINGS mutations_sync = 2;
```
Materialization builds the index for the existing data. The `mutations_sync` setting waits for the materialization to finish.
You can check the index definition in the `system.data_skipping_indices` table.
Run the same query again once the index has been materialized:
```sql title="Query" theme={null}
SELECT count(*)
FROM hackernews
WHERE hasAnyTokens(lower(comment), 'clickhouse');
```
```response title="Response" highlight={1} theme={null}
┌─count()─┐
│ 1145 │
└─────────┘
1 row in set. Elapsed: 0.019 sec. Processed 4.48 million rows, 4.48 MB (232.23 million rows/s., 232.23 MB/s.)
```
The result remains the same because the index changes how ClickHouse finds matching rows, not which rows match. The indexed
query processes substantially less data and completes much faster.
Use [`EXPLAIN`](/docs/reference/statements/explain) to confirm that ClickHouse plans to apply the index:
```sql title="Query" theme={null}
EXPLAIN indexes = 1
SELECT count(*)
FROM hackernews
WHERE hasAnyTokens(lower(comment), 'clickhouse');
```
```response title="Response" theme={null}
Output: count()
Aggregating
│ Keys:
│ Aggregates: count()
│ Skip merging: 0
└──Filter
│ Filter column: __text_index_comment_idx_hasAnyTokens_7b9491bd22343c822f64c95a9bda20f9
└──ReadFromMergeTree (default.hackernews)
Read type: Default
Parts: 4 | Granules: 547
Output: __text_index_comment_idx_hasAnyTokens_7b9491bd22343c822f64c95a9bda20f9
Indexes:
PrimaryKey
Condition: true
Parts: 4/4
Granules: 3527/3527
Skip
Name: comment_idx
Description: text GRANULARITY 100000000
Condition: (mode: Any; tokens: ["clickhouse"])
Parts: 4/4
Granules: 547/3527
Ranges: 437
```
The `comment_idx` entry shows that ClickHouse plans to apply the text index. In this example, the plan selects 547 of 3527
granules, substantially reducing the amount of data examined.
You can also search for any or all of multiple tokens. These functions match complete tokens produced by the index tokenizer.
Use `hasAnyTokens` when at least one token must match:
```sql title="Query" theme={null}
SELECT count(*)
FROM hackernews
WHERE hasAnyTokens(lower(comment), 'oltp olap');
```
```response title="Response" theme={null}
┌─count()─┐
│ 2020 │
└─────────┘
```
Use `hasAllTokens` when every token must match, in any order:
```sql title="Query" theme={null}
SELECT count(*)
FROM hackernews
WHERE hasAllTokens(lower(comment), 'avx sve');
```
```response title="Response" theme={null}
┌─count()─┐
│ 22 │
└─────────┘
```
# Hacker News vector search dataset
Source: https://clickhouse.com/docs/get-started/sample-datasets/hacker-news-vector-search
Dataset containing 28+ million Hacker News postings & their vector embeddings
## Introduction
The [Hacker News dataset](https://news.ycombinator.com/) contains 28.74 million
postings and their vector embeddings. The embeddings were generated using [SentenceTransformers](https://sbert.net/) model [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2). The dimension of each embedding vector is `384`.
This dataset can be used to walk through the design, sizing and performance aspects for a large scale,
real world vector search application built on top of user generated, textual data.
## Dataset details
The complete dataset with vector embeddings is made available by ClickHouse as a single `Parquet` file in a [S3 bucket](https://clickhouse-datasets.s3.amazonaws.com/hackernews-miniLM/hackernews_part_1_of_1.parquet)
We recommend users first run a sizing exercise to estimate the storage and memory requirements for this dataset by referring to the [documentation](/docs/reference/engines/table-engines/mergetree-family/annindexes).
## Steps
**1. Create table**
Create the `hackernews` table to store the postings & their embeddings and associated attributes:
```sql theme={null}
CREATE TABLE hackernews
(
`id` Int32,
`doc_id` Int32,
`text` String,
`vector` Array(Float32),
`node_info` Tuple(
start Nullable(UInt64),
end Nullable(UInt64)),
`metadata` String,
`type` Enum8('story' = 1, 'comment' = 2, 'poll' = 3, 'pollopt' = 4, 'job' = 5),
`by` LowCardinality(String),
`time` DateTime,
`title` String,
`post_score` Int32,
`dead` UInt8,
`deleted` UInt8,
`length` UInt32
)
ENGINE = MergeTree
ORDER BY id;
```
The `id` is just an incrementing integer. The additional attributes can be used in predicates to understand
vector similarity search combined with post-filtering/pre-filtering as explained in the [documentation](/docs/reference/engines/table-engines/mergetree-family/annindexes)
**2. Load data**
To load the dataset from the `Parquet` file, run the following SQL statement:
```sql theme={null}
INSERT INTO hackernews SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/hackernews-miniLM/hackernews_part_1_of_1.parquet', NOSIGN);
```
Inserting 28.74 million rows into the table will take a few minutes.
**3. Build a vector similarity index**
Run the following SQL to define and build a vector similarity index on the `vector` column of the `hackernews` table:
```sql theme={null}
ALTER TABLE hackernews ADD INDEX vector_index vector TYPE vector_similarity('hnsw', 'cosineDistance', 384, 'bf16', 64, 512);
ALTER TABLE hackernews MATERIALIZE INDEX vector_index SETTINGS mutations_sync = 2;
```
The parameters and performance considerations for index creation and search are described in the [documentation](/docs/reference/engines/table-engines/mergetree-family/annindexes).
The statement above uses values of 64 and 512 respectively for the HNSW hyperparameters `M` and `ef_construction`.
You need to carefully select optimal values for these parameters by evaluating index build time and search results quality
corresponding to selected values.
Building and saving the index could even take a few minutes/hour for the full 28.74 million dataset, depending on the number of CPU cores available and the storage bandwidth.
**4. Perform ANN search**
Once the vector similarity index has been built, vector search queries will automatically use the index:
```sql title="Query" theme={null}
SELECT id, title, text
FROM hackernews
ORDER BY cosineDistance( vector, )
LIMIT 10
```
The first time load of the vector index into memory could take a few seconds/minutes.
**5. Generate embeddings for search query**
[Sentence Transformers](https://www.sbert.net/) provide local, easy to use embedding
models for capturing the semantic meaning of sentences and paragraphs.
The dataset in this HackerNews dataset contains vector emebeddings generated from the
[all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model.
An example Python script is provided below to demonstrate how to programmatically generate
embedding vectors using `sentence_transformers1 Python package. The search embedding vector
is then passed as an argument to the [`cosineDistance()`](/reference/functions/regular-functions/distance-functions#cosineDistance) function in the `SELECT\` query.
```python theme={null}
from sentence_transformers import SentenceTransformer
import sys
import clickhouse_connect
print("Initializing...")
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
chclient = clickhouse_connect.get_client() # ClickHouse credentials here
while True:
# Take the search query from user
print("Enter a search query :")
input_query = sys.stdin.readline();
texts = [input_query]
# Run the model and obtain search vector
print("Generating the embedding for ", input_query);
embeddings = model.encode(texts)
print("Querying ClickHouse...")
params = {'v1':list(embeddings[0]), 'v2':20}
result = chclient.query("SELECT id, title, text FROM hackernews ORDER BY cosineDistance(vector, %(v1)s) LIMIT %(v2)s", parameters=params)
print("Results :")
for row in result.result_rows:
print(row[0], row[2][:100])
print("---------")
```
An example of running the above Python script and similarity search results are shown below
(only 100 characters from each of the top 20 posts are printed):
```text theme={null}
Initializing...
Enter a search query :
Are OLAP cubes useful
Generating the embedding for "Are OLAP cubes useful"
Querying ClickHouse...
Results :
27742647 smartmic:
slt2021: OLAP Cube is not dead, as long as you use some form of:1. GROUP BY multiple fi
---------
27744260 georgewfraser:A data mart is a logical organization of data to help humans understand the schema. Wh
---------
27761434 mwexler:"We model data according to rigorous frameworks like Kimball or Inmon because we must r
---------
28401230 chotmat:
erosenbe0: OLAP database is just a copy, replica, or archive of data with a schema designe
---------
22198879 Merick:+1 for Apache Kylin, it's a great project and awesome open source community. If anyone i
---------
27741776 crazydoggers:I always felt the value of an OLAP cube was uncovering questions you may not know to as
---------
22189480 shadowsun7:
_Codemonkeyism: After maintaining an OLAP cube system for some years, I'm not that
---------
27742029 smartmic:
gengstrand: My first exposure to OLAP was on a team developing a front end to Essbase that
---------
22364133 irfansharif:
simo7: I'm wondering how this technology could work for OLAP cubes.
An OLAP cube
---------
23292746 scoresmoke:When I was developing my pet project for Web analytics ( int:
encoding = tiktoken.encoding_for_model(encoding_name)
num_tokens = len(encoding.encode(string))
return num_tokens
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
chclient = clickhouse_connect.get_client(compress=False) # ClickHouse credentials here
while True:
# Take the search query from user
print("Enter a search topic :")
input_query = sys.stdin.readline();
texts = [input_query]
# Run the model and obtain search or reference vector
print("Generating the embedding for ----> ", input_query);
embeddings = model.encode(texts)
print("Querying ClickHouse...")
params = {'v1':list(embeddings[0]), 'v2':100}
result = chclient.query("SELECT id,title,text FROM hackernews ORDER BY cosineDistance(vector, %(v1)s) LIMIT %(v2)s", parameters=params)
# Just join all the search results
doc_results = ""
for row in result.result_rows:
doc_results = doc_results + "\n" + row[2]
print("Initializing chatgpt-3.5-turbo model")
model_name = "gpt-3.5-turbo"
text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
model_name=model_name
)
texts = text_splitter.split_text(doc_results)
docs = [Document(page_content=t) for t in texts]
llm = ChatOpenAI(temperature=0, model_name=model_name)
prompt_template = """
Write a concise summary of the following in not more than 10 sentences:
{text}
CONSCISE SUMMARY :
"""
prompt = PromptTemplate(template=prompt_template, input_variables=["text"])
num_tokens = num_tokens_from_string(doc_results, model_name)
gpt_35_turbo_max_tokens = 4096
verbose = False
print("Summarizing search results retrieved from ClickHouse...")
if num_tokens <= gpt_35_turbo_max_tokens:
chain = load_summarize_chain(llm, chain_type="stuff", prompt=prompt, verbose=verbose)
else:
chain = load_summarize_chain(llm, chain_type="map_reduce", map_prompt=prompt, combine_prompt=prompt, verbose=verbose)
summary = chain.run(docs)
print(f"Summary from chatgpt-3.5: {summary}")
```
# Tutorials and example datasets
Source: https://clickhouse.com/docs/get-started/sample-datasets/index
Get started with ClickHouse using our tutorials and example datasets
We have a lot of resources to help you get started and learn how ClickHouse works:
* To try ClickHouse without installing anything, you can sign up for a free [ClickHouse Cloud trial](https://clickhouse.cloud/signUp?loc=docs-get-started-sample-datasets) or query the read-only [ClickHouse playground](/docs/get-started/sample-datasets/playground).
* If you would rather run ClickHouse yourself, follow our [Quick Start](/docs/get-started/setup/install).
* If you're looking for a place to start learning, try the [New York taxi data](/docs/get-started/sample-datasets/nyc-taxi) tutorial, which analyzes a dataset of New York City taxi rides — or pick any of the tutorials below.
We have built tutorials based on a range of sample datasets. They give you hands-on experience with ClickHouse, teaching key techniques and demonstrating how to take advantage of many powerful ClickHouse functions. The sample dataset tutorials include:
# Join Order Benchmark (JOB)
Source: https://clickhouse.com/docs/get-started/sample-datasets/job
The Join Order Benchmark (JOB) data set and queries.
The Join Order Benchmark (JOB) stresses the query optimizer with 113 analytical queries over a real-world, highly-correlated dataset (a snapshot of IMDb). Since its introduction, the JOB benchmark has become the de facto standard to assess the performance of relational database query optimizers, including cardinality estimation and join order optimization. Unlike synthetic benchmarks that assume uniform, independent data, JOB uses real data with skew and correlations, which makes it a hard test for join ordering and cardinality estimation.
The dataset holds about 74 million rows across 21 tables and takes around 1.15 GiB compressed in ClickHouse.
The 113 queries are organized into 33 families (`1`–`33`). Queries within a family (`a`, `b`, `c`, ...) share the same join graph but differ in their selection predicates.
**References**
* [How Good Are Query Optimizers, Really?](https://www.vldb.org/pvldb/vol9/p204-leis.pdf) (Leis et al., VLDB 2015)
* [Join Order Benchmark](https://github.com/gregrahn/join-order-benchmark) repository
## Creating the tables
The JOB dataset is a snapshot of IMDb with 21 tables. The table definitions are available in [`init_cloud.sql`](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/job/init_cloud.sql) in the ClickHouse repository.
Each table uses the [`MergeTree`](/docs/reference/engines/table-engines/mergetree-family/mergetree) engine sorted by its primary key column `id`, mirroring the original PostgreSQL schema where every table declares `id integer NOT NULL PRIMARY KEY`. Nullable PostgreSQL columns map to `Nullable(...)` types.
Create the tables:
```bash theme={null}
curl -O https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/tests/benchmarks/job/init_cloud.sql
clickhouse client --query "CREATE DATABASE IF NOT EXISTS job"
clickhouse client --database job --queries-file init_cloud.sql
```
## Loading the data
The data comes from the original IMDb snapshot used by JOB, distributed as one CSV file per table (`aka_name.csv`, `title.csv`, ...). These CSVs use PostgreSQL `COPY` semantics with `ESCAPE '\'`: a backslash escapes the quote character only inside a quoted field, while outside quotes a backslash is a literal character. ClickHouse expects RFC 4180 CSV (doubled quotes, no backslash escaping), so the files must be re-encoded first.
[`convert_csv.py`](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/job/convert_csv.py) performs that re-encoding. It reads the original CSV on stdin and writes standard CSV on stdout, doubling embedded quotes and preserving empty unquoted fields (which ClickHouse maps to `NULL` for `Nullable` columns).
To build the tables from the original CSVs:
* Create the tables (see above).
* Download the IMDb data set as an `imdb.tgz` file, following the instructions in the Join Order Benchmark repository.
* Convert and import the data:
```bash theme={null}
set -euo pipefail
for table in aka_name aka_title cast_info char_name comp_cast_type company_name \
company_type complete_cast info_type keyword kind_type link_type \
movie_companies movie_info movie_info_idx movie_keyword movie_link \
name person_info role_type title; do
echo "Loading ${table} ..."
python3 convert_csv.py < "${table}.csv" > "${table}.clean.csv"
clickhouse client --database job --query "INSERT INTO ${table} FORMAT CSV" < "${table}.clean.csv"
done
```
Once the tables are populated, they can be exported to Parquet for faster re-import later, e.g. `clickhouse client --database job --query "SELECT * FROM title ORDER BY id FORMAT Parquet" > title.parquet`.
Detailed table sizes:
| Table | size (in rows) | size (compressed in ClickHouse) |
| - | - | - |
| aka\_name | 901,343 | 31.86 MiB |
| aka\_title | 361,472 | 14.32 MiB |
| cast\_info | 36,244,344 | 296.25 MiB |
| char\_name | 3,140,339 | 107.95 MiB |
| comp\_cast\_type | 4 | 132.00 B |
| company\_name | 234,997 | 8.38 MiB |
| company\_type | 4 | 162.00 B |
| complete\_cast | 135,086 | 748.80 KiB |
| info\_type | 113 | 1.25 KiB |
| keyword | 134,170 | 1.88 MiB |
| kind\_type | 7 | 177.00 B |
| link\_type | 18 | 284.00 B |
| movie\_companies | 2,609,129 | 21.20 MiB |
| movie\_info | 14,835,720 | 300.46 MiB |
| movie\_info\_idx | 1,380,035 | 8.01 MiB |
| movie\_keyword | 4,523,930 | 21.06 MiB |
| movie\_link | 29,997 | 178.21 KiB |
| name | 4,167,491 | 131.16 MiB |
| person\_info | 2,963,664 | 154.12 MiB |
| role\_type | 12 | 246.00 B |
| title | 2,528,312 | 78.04 MiB |
| **Total** | **74,190,187** | **1.15 GiB** |
(Compressed sizes in ClickHouse are taken from `system.tables.total_bytes` and based on the above table definitions.)
## Queries
The 113 JOB queries can be found [here](https://github.com/ClickHouse/ClickHouse/tree/master/tests/benchmarks/job/queries) in the ClickHouse repository. The settings used to run them are in [`settings.json`](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/job/settings.json). See the [README](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/job/README.md) for known issues and notes on specific queries.
The queries reference the tables by name, so run them against the `job` database (for example, with `clickhouse client --database job`).
An example query (`1a`):
```sql theme={null}
SELECT MIN(mc.note) AS production_note,
MIN(t.title) AS movie_title,
MIN(t.production_year) AS movie_year
FROM company_type AS ct,
info_type AS it,
movie_companies AS mc,
movie_info_idx AS mi_idx,
title AS t
WHERE ct.kind = 'production companies'
AND it.info = 'top 250 rank'
AND mc.note NOT LIKE '%(as Metro-Goldwyn-Mayer Pictures)%'
AND (mc.note LIKE '%(co-production)%'
OR mc.note LIKE '%(presents)%')
AND ct.id = mc.company_type_id
AND t.id = mc.movie_id
AND t.id = mi_idx.movie_id
AND mc.movie_id = mi_idx.movie_id
AND it.id = mi_idx.info_type_id;
```
## Performance benchmark
ClickHouse tracks JOB query performance across every released version. You can explore run times for all 113 JOB queries on the [ClickHouse versions benchmark](https://benchmark.clickhouse.com/versions/#version=+s\&dataset=+ob\&metric=hot\&queries=-) page to see how performance has evolved over time.
# Laion-400M dataset
Source: https://clickhouse.com/docs/get-started/sample-datasets/laion
Dataset containing 400 million images with English image captions
The [Laion-400M dataset](https://laion.ai/blog/laion-400-open-dataset/) contains 400 million images with English image captions. Laion nowadays provides [an even larger dataset](https://laion.ai/blog/laion-5b/) but working with it will be similar.
The dataset contains the image URL, embeddings for both the image and the image caption, a similarity score between the image and the image caption, as well as metadata, e.g. the image width/height, the licence and a NSFW flag. We can use the dataset to demonstrate [approximate nearest neighbor search](/docs/reference/engines/table-engines/mergetree-family/annindexes) in ClickHouse.
## Data preparation
The embeddings and the metadata are stored in separate files in the raw data. A data preparation step downloads the data, merges the files,
converts them to CSV and imports them into ClickHouse. You can use the following `download.sh` script for that:
```bash theme={null}
number=${1}
if [[ $number == '' ]]; then
number=1
fi;
wget --tries=100 https://deploy.laion.ai/8f83b608504d46bb81708ec86e912220/embeddings/img_emb/img_emb_${number}.npy # download image embedding
wget --tries=100 https://deploy.laion.ai/8f83b608504d46bb81708ec86e912220/embeddings/text_emb/text_emb_${number}.npy # download text embedding
wget --tries=100 https://deploy.laion.ai/8f83b608504d46bb81708ec86e912220/embeddings/metadata/metadata_${number}.parquet # download metadata
python3 process.py $number # merge files and convert to CSV
```
Script `process.py` is defined as follows:
```python theme={null}
import pandas as pd
import numpy as np
import os
import sys
str_i = str(sys.argv[1])
npy_file = "img_emb_" + str_i + '.npy'
metadata_file = "metadata_" + str_i + '.parquet'
text_npy = "text_emb_" + str_i + '.npy'
# load all files
im_emb = np.load(npy_file)
text_emb = np.load(text_npy)
data = pd.read_parquet(metadata_file)
# combine files
data = pd.concat([data, pd.DataFrame({"image_embedding" : [*im_emb]}), pd.DataFrame({"text_embedding" : [*text_emb]})], axis=1, copy=False)
# columns to be imported into ClickHouse
data = data[['url', 'caption', 'NSFW', 'similarity', "image_embedding", "text_embedding"]]
# transform np.arrays to lists
data['image_embedding'] = data['image_embedding'].apply(lambda x: x.tolist())
data['text_embedding'] = data['text_embedding'].apply(lambda x: x.tolist())
# this small hack is needed because caption sometimes contains all kind of quotes
data['caption'] = data['caption'].apply(lambda x: x.replace("'", " ").replace('"', " "))
# export data as CSV file
data.to_csv(str_i + '.csv', header=False)
# removed raw data files
os.system(f"rm {npy_file} {metadata_file} {text_npy}")
```
To start the data preparation pipeline, run:
```bash theme={null}
seq 0 409 | xargs -P1 -I{} bash -c './download.sh {}'
```
The dataset is split into 410 files, each file contains ca. 1 million rows. If you like to work with a smaller subset of the data, simply adjust the limits, e.g. `seq 0 9 | ...`.
(The python script above is very slow (\~2-10 minutes per file), takes a lot of memory (41 GB per file), and the resulting csv files are big (10 GB each), so be careful. If you have enough RAM, increase the `-P1` number for more parallelism. If this is still too slow, consider coming up with a better ingestion procedure - maybe converting the .npy files to parquet, then doing all the other processing with clickhouse.)
## Create table
To create a table initially without indexes, run:
```sql theme={null}
CREATE TABLE laion
(
`id` Int64,
`url` String,
`caption` String,
`NSFW` String,
`similarity` Float32,
`image_embedding` Array(Float32),
`text_embedding` Array(Float32)
)
ENGINE = MergeTree
ORDER BY id
```
To import the CSV files into ClickHouse:
```sql theme={null}
INSERT INTO laion FROM INFILE '{path_to_csv_files}/*.csv'
```
Note that the `id` column is just for illustration and is populated by the script with non-unique values.
## Run a brute-force vector similarity search
To run a brute-force approximate vector search, run:
```sql theme={null}
SELECT url, caption FROM laion ORDER BY cosineDistance(image_embedding, {target:Array(Float32)}) LIMIT 10
```
`target` is an array of 512 elements and a client parameter.
A convenient way to obtain such arrays will be presented at the end of the article.
For now, we can run the embedding of a random LEGO set picture as `target`.
**Result**
```markdown theme={null}
┌─url───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─caption──────────────────────────────────────────────────────────────────────────┐
1. │ https://s4.thcdn.com/productimg/600/600/11340490-9914447026352671.jpg │ LEGO Friends: Puppy Treats & Tricks (41304) │
2. │ https://www.avenuedelabrique.com/img/uploads/f20fd44bfa4bd49f2a3a5fad0f0dfed7d53c3d2f.jpg │ Nouveau LEGO Friends 41334 Andrea s Park Performance 2018 │
3. │ http://images.esellerpro.com/2489/I/667/303/3938_box_in.jpg │ 3938 LEGO Andreas Bunny House Girls Friends Heartlake Age 5-12 / 62 Pieces New! │
4. │ http://i.shopmania.org/180x180/7/7f/7f1e1a2ab33cde6af4573a9e0caea61293dfc58d.jpg?u=https%3A%2F%2Fs.s-bol.com%2Fimgbase0%2Fimagebase3%2Fextralarge%2FFC%2F4%2F0%2F9%2F9%2F9200000049789904.jpg │ LEGO Friends Avonturenkamp Boomhuis - 41122 │
5. │ https://s.s-bol.com/imgbase0/imagebase/large/FC/5/5/9/4/1004004011684955.jpg │ LEGO Friends Andrea s Theatershow - 3932 │
6. │ https://www.jucariicucubau.ro/30252-home_default/41445-lego-friends-ambulanta-clinicii-veterinare.jpg │ 41445 - LEGO Friends - Ambulanta clinicii veterinare │
7. │ https://cdn.awsli.com.br/600x1000/91/91201/produto/24833262/234c032725.jpg │ LEGO FRIENDS 41336 EMMA S ART CAFÉ │
8. │ https://media.4rgos.it/s/Argos/6174930_R_SET?$Thumb150$&$Web$ │ more details on LEGO Friends Stephanie s Friendship Cake Set - 41308. │
9. │ https://thumbs4.ebaystatic.com/d/l225/m/mG4k6qAONd10voI8NUUMOjw.jpg │ Lego Friends Gymnast 30400 Polybag 26 pcs │
10. │ http://www.ibrickcity.com/wp-content/gallery/41057/thumbs/thumbs_lego-41057-heartlake-horse-show-friends-3.jpg │ lego-41057-heartlake-horse-show-friends-3 │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┘
10 rows in set. Elapsed: 4.605 sec. Processed 100.38 million rows, 309.98 GB (21.80 million rows/s., 67.31 GB/s.)
```
## Run an approximate vector similarity search with a vector similarity index
Let's now define two vector similarity indexes on the table.
```sql theme={null}
ALTER TABLE laion ADD INDEX image_index image_embedding TYPE vector_similarity('hnsw', 'cosineDistance', 512, 'bf16', 64, 256)
ALTER TABLE laion ADD INDEX text_index text_embedding TYPE vector_similarity('hnsw', 'cosineDistance', 512, 'bf16', 64, 256)
```
The parameters and performance considerations for index creation and search are described in the [documentation](/docs/reference/engines/table-engines/mergetree-family/annindexes).
The above index definition specifies a HNSW index using the "cosine distance" as distance metric with the parameter "hnsw\_max\_connections\_per\_layer" set to 64 and parameter "hnsw\_candidate\_list\_size\_for\_construction" set to 256.
The index uses half-precision brain floats (bfloat16) as quantization to optimize memory usage.
To build and materialize the index, run these statements :
```sql theme={null}
ALTER TABLE laion MATERIALIZE INDEX image_index;
ALTER TABLE laion MATERIALIZE INDEX text_index;
```
Building and saving the index could take a few minutes or even hours, depending on the number of rows and HNSW index parameters.
To perform a vector search, just execute the same query again:
```sql theme={null}
SELECT url, caption FROM laion ORDER BY cosineDistance(image_embedding, {target:Array(Float32)}) LIMIT 10
```
**Result**
```response theme={null}
┌─url───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─caption──────────────────────────────────────────────────────────────────────────┐
1. │ https://s4.thcdn.com/productimg/600/600/11340490-9914447026352671.jpg │ LEGO Friends: Puppy Treats & Tricks (41304) │
2. │ https://www.avenuedelabrique.com/img/uploads/f20fd44bfa4bd49f2a3a5fad0f0dfed7d53c3d2f.jpg │ Nouveau LEGO Friends 41334 Andrea s Park Performance 2018 │
3. │ http://images.esellerpro.com/2489/I/667/303/3938_box_in.jpg │ 3938 LEGO Andreas Bunny House Girls Friends Heartlake Age 5-12 / 62 Pieces New! │
4. │ http://i.shopmania.org/180x180/7/7f/7f1e1a2ab33cde6af4573a9e0caea61293dfc58d.jpg?u=https%3A%2F%2Fs.s-bol.com%2Fimgbase0%2Fimagebase3%2Fextralarge%2FFC%2F4%2F0%2F9%2F9%2F9200000049789904.jpg │ LEGO Friends Avonturenkamp Boomhuis - 41122 │
5. │ https://s.s-bol.com/imgbase0/imagebase/large/FC/5/5/9/4/1004004011684955.jpg │ LEGO Friends Andrea s Theatershow - 3932 │
6. │ https://www.jucariicucubau.ro/30252-home_default/41445-lego-friends-ambulanta-clinicii-veterinare.jpg │ 41445 - LEGO Friends - Ambulanta clinicii veterinare │
7. │ https://cdn.awsli.com.br/600x1000/91/91201/produto/24833262/234c032725.jpg │ LEGO FRIENDS 41336 EMMA S ART CAFÉ │
8. │ https://media.4rgos.it/s/Argos/6174930_R_SET?$Thumb150$&$Web$ │ more details on LEGO Friends Stephanie s Friendship Cake Set - 41308. │
9. │ https://thumbs4.ebaystatic.com/d/l225/m/mG4k6qAONd10voI8NUUMOjw.jpg │ Lego Friends Gymnast 30400 Polybag 26 pcs │
10. │ http://www.ibrickcity.com/wp-content/gallery/41057/thumbs/thumbs_lego-41057-heartlake-horse-show-friends-3.jpg │ lego-41057-heartlake-horse-show-friends-3 │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┘
10 rows in set. Elapsed: 0.019 sec. Processed 137.27 thousand rows, 24.42 MB (7.38 million rows/s., 1.31 GB/s.)
```
The query latency decreased significantly because the nearest neighbours were retrieved using the vector index.
Vector similarity search using a vector similarity index may return results that differ slightly from the brute-force search results.
An HNSW index can potentially achieve a recall close to 1 (same accuracy as brute force search) with a careful selection of the HNSW parameters and evaluating the index quality.
## Creating embeddings with UDFs
One usually wants to create embeddings for new images or new image captions and search for similar image / image caption pairs in the data. We can use [UDF](/docs/reference/functions/regular-functions/udf) to create the `target` vector without leaving the client. It is important to use the same model to create the data and new embeddings for searches. The following scripts utilize the `ViT-B/32` model which also underlies the dataset.
### Text embeddings
First, store the following Python script in the `user_scripts/` directory of your ClickHouse data path and make it executable (`chmod +x encode_text.py`).
`encode_text.py`:
```python theme={null}
#!/usr/bin/python3
#!Note: Change the above python3 executable location if a virtual env is being used.
import clip
import torch
import numpy as np
import sys
if __name__ == '__main__':
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
for text in sys.stdin:
inputs = clip.tokenize(text)
with torch.no_grad():
text_features = model.encode_text(inputs)[0].tolist()
print(text_features)
sys.stdout.flush()
```
Then create `encode_text_function.xml` in a location referenced by `/path/to/*_function.xml` in your ClickHouse server configuration file.
```xml theme={null}
executable
encode_text
Array(Float32)
String
text
TabSeparated
encode_text.py
1000000
```
You can now simply use:
```sql theme={null}
SELECT encode_text('cat');
```
The first run will be slow because it loads the model, but repeated runs will be fast. We can then copy the output to `SET param_target=...` and can easily write queries. Alternatively, the `encode_text()` function can directly be used as a argument to the `cosineDistance` function :
```SQL theme={null}
SELECT url
FROM laion
ORDER BY cosineDistance(text_embedding, encode_text('a dog and a cat')) ASC
LIMIT 10
```
Note that the `encode_text()` UDF itself could require a few seconds to compute and emit the embedding vector.
### Image embeddings
Image embeddings can be created similarly and we provide a Python script that can generate an embedding of an image stored locally as a file.
`encode_image.py`
```python theme={null}
#!/usr/bin/python3
#!Note: Change the above python3 executable location if a virtual env is being used.
import clip
import torch
import numpy as np
from PIL import Image
import sys
if __name__ == '__main__':
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
for text in sys.stdin:
image = preprocess(Image.open(text.strip())).unsqueeze(0).to(device)
with torch.no_grad():
image_features = model.encode_image(image)[0].tolist()
print(image_features)
sys.stdout.flush()
```
`encode_image_function.xml`
```xml theme={null}
executable_pool
encode_image
Array(Float32)
String
path
TabSeparated
encode_image.py
1000000
```
Fetch an example image to search :
```shell theme={null}
# get a random image of a LEGO set
$ wget http://cdn.firstcry.com/brainbees/images/products/thumb/191325a.jpg
```
Then run this query to generate the embedding for above image :
```sql theme={null}
SELECT encode_image('/path/to/your/image');
```
The complete search query is :
```sql theme={null}
SELECT
url,
caption
FROM laion
ORDER BY cosineDistance(image_embedding, encode_image('/path/to/your/image')) ASC
LIMIT 10
```
# LAION 5B dataset
Source: https://clickhouse.com/docs/get-started/sample-datasets/laion5b
Dataset containing 10 million vectors from the LAION 5B dataset
## Introduction
The [LAION 5b dataset](https://laion.ai/blog/laion-5b/) contains 5.85 billion image-text embeddings and
associated image metadata. The embeddings were generated using `Open AI CLIP` model [ViT-L/14](https://huggingface.co/sentence-transformers/clip-ViT-L-14). The
dimension of each embedding vector is `768`.
This dataset can be used to model design, sizing and performance aspects for a large scale,
real world vector search application. The dataset can be used for both text to image search and
image to image search.
## Dataset details
The complete dataset can be downloaded using [opendatalab/laion5b-downloader](https://github.com/opendatalab/laion5b-downloader).
ClickHouse has made available a subset of 10 million vectors in a public `S3` bucket.
The subset is stored in one `Parquet` file.
We recommend users first run a sizing exercise to estimate the storage and memory requirements for this dataset by referring to the [documentation](/docs/reference/engines/table-engines/mergetree-family/annindexes).
## Steps
**1. Create table**
Create the `laion_5b_10m` table to store the embeddings and their associated attributes:
```sql theme={null}
CREATE TABLE laion_5b_10m
(
id UInt32,
image_path String,
caption String,
NSFW Nullable(String) default 'unknown',
similarity Float32,
LICENSE Nullable(String),
url String,
key String,
status LowCardinality(String),
width Int32,
height Int32,
original_width Int32,
original_height Int32,
exif Nullable(String),
md5 String,
vector Array(Float32) CODEC(NONE)
) ENGINE = MergeTree ORDER BY (id)
```
The `id` is just an incrementing integer. The additional attributes can be used in predicates to understand
vector similarity search combined with post-filtering/pre-filtering as explained in the [documentation](/docs/reference/engines/table-engines/mergetree-family/annindexes)
**2. Load data**
To load the dataset, run the following SQL statement:
```sql theme={null}
INSERT INTO laion_5b_10m SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/laion-5b/laion5b_100m_part_1_of_10.parquet', NOSIGN);
```
The loading of 10 million rows into the table will take a few minutes.
**3. Run a brute-force vector similarity search**
KNN (k - Nearest Neighbours) search or brute force search involves calculating the distance of each vector in the dataset
to the search embedding vector and then ordering the distances to get the nearest neighbours. We can use one of the vectors
from the dataset itself as the search vector. For example:
```sql title="Query" theme={null}
SELECT id, url
FROM laion_5b_10m
ORDER BY cosineDistance( vector, (SELECT vector FROM laion_5b_10m WHERE id = 9999) ) ASC
LIMIT 20
The vector in the row with id = 9999 is the embedding for an image of a Deli restaurant.
```
Note down the query latency so that we can compare it with the query latency of ANN (using vector index).
With 10 million rows, the above query without a vector index could take a few seconds/minutes to complete.
**4. Build a vector similarity index**
Run the following SQL to define and build a vector similarity index on the `vector` column of the `laion_5b_10m` table:
```sql theme={null}
ALTER TABLE laion_5b_10m ADD INDEX vector_index vector TYPE vector_similarity('hnsw', 'cosineDistance', 768, 'bf16', 64, 512);
ALTER TABLE laion_5b_10m MATERIALIZE INDEX vector_index SETTINGS mutations_sync = 2;
```
The parameters and performance considerations for index creation and search are described in the [documentation](/docs/reference/engines/table-engines/mergetree-family/annindexes).
The statement above uses values of 64 and 512 respectively for the HNSW hyperparameters `M` and `ef_construction`.
You need to carefully select optimal values for these parameters by evaluating index build time and search results quality
corresponding to selected values.
Building and saving the index can take significant time for the 10 million row subset, depending on the number of CPU cores available and the storage bandwidth.
**5. Perform ANN search**
Once the vector similarity index has been built, vector search queries will automatically use the index:
```sql title="Query" theme={null}
SELECT id, url
FROM laion_5b_10m
ORDER BY cosineDistance( vector, (SELECT vector FROM laion_5b_10m WHERE id = 9999) ) ASC
LIMIT 20
```
The first time load of the vector index into memory could take a few seconds/minutes.
**6. Generate embeddings for search query**
The `LAION 5b` dataset embedding vectors were generated using `OpenAI CLIP` model `ViT-L/14`.
An example Python script is provided below to demonstrate how to programmatically generate
embedding vectors using the `CLIP` APIs. The search embedding vector
is then passed as an argument to the [`cosineDistance()`](/docs/reference/functions/regular-functions/distance-functions#cosineDistance) function in the `SELECT` query.
To install the `clip` package, please refer to the [OpenAI GitHub repository](https://github.com/openai/clip).
```python theme={null}
import torch
import clip
import numpy as np
import sys
import clickhouse_connect
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-L/14", device=device)
# Search for images that contain both a dog and a cat
text = clip.tokenize(["a dog and a cat"]).to(device)
with torch.no_grad():
text_features = model.encode_text(text)
np_arr = text_features.detach().cpu().numpy()
# Pass ClickHouse credentials here
chclient = clickhouse_connect.get_client()
params = {'v1': list(np_arr[0])}
result = chclient.query("SELECT id, url FROM laion_5b_10m ORDER BY cosineDistance(vector, %(v1)s) LIMIT 100",
parameters=params)
# Write the results to a simple HTML page that can be opened in the browser. Some URLs may have become obsolete.
print("")
for r in result.result_rows:
print("
')
print("")
```
The result of the above search is shown below:
# New York Public Library "What's on the Menu?" dataset
Source: https://clickhouse.com/docs/get-started/sample-datasets/menus
Dataset containing 1.3 million records of historical data on the menus of hotels, restaurants and cafes with the dishes along with their prices.
The dataset is created by the New York Public Library. It contains historical data on the menus of hotels, restaurants and cafes with the dishes along with their prices.
Source: [https://www.nypl.org/research/support/whats-on-the-menu](https://www.nypl.org/research/support/whats-on-the-menu)
The data is in public domain.
The data is from library's archive and it may be incomplete and difficult for statistical analysis. Nevertheless it is also very yummy.
The size is just 1.3 million records about dishes in the menus — it's a very small data volume for ClickHouse, but it's still a good example.
## Download the dataset
Run the command:
```bash theme={null}
wget https://s3.amazonaws.com/menusdata.nypl.org/gzips/2021_08_01_07_01_17_data.tgz
# Option: Validate the checksum
md5sum 2021_08_01_07_01_17_data.tgz
# Checksum should be equal to: db6126724de939a5481e3160a2d67d15
```
Replace the link to the up to date link from [https://www.nypl.org/research/support/whats-on-the-menu](https://www.nypl.org/research/support/whats-on-the-menu) if needed.
Download size is about 35 MB.
## Unpack the dataset
```bash theme={null}
tar xvf 2021_08_01_07_01_17_data.tgz
```
Uncompressed size is about 150 MB.
The data is normalized consisted of four tables:
* `Menu` — Information about menus: the name of the restaurant, the date when menu was seen, etc.
* `Dish` — Information about dishes: the name of the dish along with some characteristic.
* `MenuPage` — Information about the pages in the menus, because every page belongs to some menu.
* `MenuItem` — An item of the menu. A dish along with its price on some menu page: links to dish and menu page.
## Create the tables
We use [Decimal](/docs/reference/data-types/decimal) data type to store prices.
```sql theme={null}
CREATE TABLE dish
(
id UInt32,
name String,
description String,
menus_appeared UInt32,
times_appeared Int32,
first_appeared UInt16,
last_appeared UInt16,
lowest_price Decimal64(3),
highest_price Decimal64(3)
) ENGINE = MergeTree ORDER BY id;
CREATE TABLE menu
(
id UInt32,
name String,
sponsor String,
event String,
venue String,
place String,
physical_description String,
occasion String,
notes String,
call_number String,
keywords String,
language String,
date String,
location String,
location_type String,
currency String,
currency_symbol String,
status String,
page_count UInt16,
dish_count UInt16
) ENGINE = MergeTree ORDER BY id;
CREATE TABLE menu_page
(
id UInt32,
menu_id UInt32,
page_number UInt16,
image_id String,
full_height UInt16,
full_width UInt16,
uuid UUID
) ENGINE = MergeTree ORDER BY id;
CREATE TABLE menu_item
(
id UInt32,
menu_page_id UInt32,
price Decimal64(3),
high_price Decimal64(3),
dish_id UInt32,
created_at DateTime,
updated_at DateTime,
xpos Float64,
ypos Float64
) ENGINE = MergeTree ORDER BY id;
```
## Import the data
Upload data into ClickHouse, run:
```bash theme={null}
clickhouse-client --format_csv_allow_single_quotes 0 --input_format_null_as_default 0 --query "INSERT INTO dish FORMAT CSVWithNames" < Dish.csv
clickhouse-client --format_csv_allow_single_quotes 0 --input_format_null_as_default 0 --query "INSERT INTO menu FORMAT CSVWithNames" < Menu.csv
clickhouse-client --format_csv_allow_single_quotes 0 --input_format_null_as_default 0 --query "INSERT INTO menu_page FORMAT CSVWithNames" < MenuPage.csv
clickhouse-client --format_csv_allow_single_quotes 0 --input_format_null_as_default 0 --date_time_input_format best_effort --query "INSERT INTO menu_item FORMAT CSVWithNames" < MenuItem.csv
```
We use [CSVWithNames](/docs/reference/formats/CSV/CSVWithNames) format as the data is represented by CSV with header.
We disable `format_csv_allow_single_quotes` as only double quotes are used for data fields and single quotes can be inside the values and shouldn't confuse the CSV parser.
We disable [input\_format\_null\_as\_default](/docs/reference/settings/formats/input-format#input_format_null_as_default) as our data doesn't have [NULL](/docs/reference/settings/formats/input-format#input_format_null_as_default). Otherwise ClickHouse will try to parse `\N` sequences and can be confused with `\` in data.
The setting [date\_time\_input\_format best\_effort](/docs/reference/settings/formats/date-time#date_time_input_format) allows to parse [DateTime](/docs/reference/data-types/datetime) fields in wide variety of formats. For example, ISO-8601 without seconds like '2000-01-01 01:02' will be recognized. Without this setting only fixed DateTime format is allowed.
## Denormalize the data
Data is presented in multiple tables in [normalized form](https://en.wikipedia.org/wiki/Database_normalization#Normal_forms). It means you have to perform [JOIN](/docs/reference/statements/select/join) if you want to query, e.g. dish names from menu items.
For typical analytical tasks it is way more efficient to deal with pre-JOINed data to avoid doing `JOIN` every time. It is called "denormalized" data.
We will create a table `menu_item_denorm` where will contain all the data JOINed together:
```sql theme={null}
CREATE TABLE menu_item_denorm
ENGINE = MergeTree ORDER BY (dish_name, created_at)
AS SELECT
price,
high_price,
created_at,
updated_at,
xpos,
ypos,
dish.id AS dish_id,
dish.name AS dish_name,
dish.description AS dish_description,
dish.menus_appeared AS dish_menus_appeared,
dish.times_appeared AS dish_times_appeared,
dish.first_appeared AS dish_first_appeared,
dish.last_appeared AS dish_last_appeared,
dish.lowest_price AS dish_lowest_price,
dish.highest_price AS dish_highest_price,
menu.id AS menu_id,
menu.name AS menu_name,
menu.sponsor AS menu_sponsor,
menu.event AS menu_event,
menu.venue AS menu_venue,
menu.place AS menu_place,
menu.physical_description AS menu_physical_description,
menu.occasion AS menu_occasion,
menu.notes AS menu_notes,
menu.call_number AS menu_call_number,
menu.keywords AS menu_keywords,
menu.language AS menu_language,
menu.date AS menu_date,
menu.location AS menu_location,
menu.location_type AS menu_location_type,
menu.currency AS menu_currency,
menu.currency_symbol AS menu_currency_symbol,
menu.status AS menu_status,
menu.page_count AS menu_page_count,
menu.dish_count AS menu_dish_count
FROM menu_item
JOIN dish ON menu_item.dish_id = dish.id
JOIN menu_page ON menu_item.menu_page_id = menu_page.id
JOIN menu ON menu_page.menu_id = menu.id;
```
## Validate the data
```sql title="Query" theme={null}
SELECT count() FROM menu_item_denorm;
```
```text title="Response" theme={null}
┌─count()─┐
│ 1329175 │
└─────────┘
```
## Run some queries
### Averaged historical prices of dishes
```sql title="Query" theme={null}
SELECT
round(toUInt32OrZero(extract(menu_date, '^\\d{4}')), -1) AS d,
count(),
round(avg(price), 2),
bar(avg(price), 0, 100, 100)
FROM menu_item_denorm
WHERE (menu_currency = 'Dollars') AND (d > 0) AND (d < 2022)
GROUP BY d
ORDER BY d ASC;
```
```text title="Response" theme={null}
┌────d─┬─count()─┬─round(avg(price), 2)─┬─bar(avg(price), 0, 100, 100)─┐
│ 1850 │ 618 │ 1.5 │ █▍ │
│ 1860 │ 1634 │ 1.29 │ █▎ │
│ 1870 │ 2215 │ 1.36 │ █▎ │
│ 1880 │ 3909 │ 1.01 │ █ │
│ 1890 │ 8837 │ 1.4 │ █▍ │
│ 1900 │ 176292 │ 0.68 │ ▋ │
│ 1910 │ 212196 │ 0.88 │ ▊ │
│ 1920 │ 179590 │ 0.74 │ ▋ │
│ 1930 │ 73707 │ 0.6 │ ▌ │
│ 1940 │ 58795 │ 0.57 │ ▌ │
│ 1950 │ 41407 │ 0.95 │ ▊ │
│ 1960 │ 51179 │ 1.32 │ █▎ │
│ 1970 │ 12914 │ 1.86 │ █▋ │
│ 1980 │ 7268 │ 4.35 │ ████▎ │
│ 1990 │ 11055 │ 6.03 │ ██████ │
│ 2000 │ 2467 │ 11.85 │ ███████████▋ │
│ 2010 │ 597 │ 25.66 │ █████████████████████████▋ │
└──────┴─────────┴──────────────────────┴──────────────────────────────┘
```
Take it with a grain of salt.
### Burger prices
```sql title="Query" theme={null}
SELECT
round(toUInt32OrZero(extract(menu_date, '^\\d{4}')), -1) AS d,
count(),
round(avg(price), 2),
bar(avg(price), 0, 50, 100)
FROM menu_item_denorm
WHERE (menu_currency = 'Dollars') AND (d > 0) AND (d < 2022) AND (dish_name ILIKE '%burger%')
GROUP BY d
ORDER BY d ASC;
```
```text title="Response" theme={null}
┌────d─┬─count()─┬─round(avg(price), 2)─┬─bar(avg(price), 0, 50, 100)───────────┐
│ 1880 │ 2 │ 0.42 │ ▋ │
│ 1890 │ 7 │ 0.85 │ █▋ │
│ 1900 │ 399 │ 0.49 │ ▊ │
│ 1910 │ 589 │ 0.68 │ █▎ │
│ 1920 │ 280 │ 0.56 │ █ │
│ 1930 │ 74 │ 0.42 │ ▋ │
│ 1940 │ 119 │ 0.59 │ █▏ │
│ 1950 │ 134 │ 1.09 │ ██▏ │
│ 1960 │ 272 │ 0.92 │ █▋ │
│ 1970 │ 108 │ 1.18 │ ██▎ │
│ 1980 │ 88 │ 2.82 │ █████▋ │
│ 1990 │ 184 │ 3.68 │ ███████▎ │
│ 2000 │ 21 │ 7.14 │ ██████████████▎ │
│ 2010 │ 6 │ 18.42 │ ████████████████████████████████████▋ │
└──────┴─────────┴──────────────────────┴───────────────────────────────────────┘
```
### Vodka
```sql title="Query" theme={null}
SELECT
round(toUInt32OrZero(extract(menu_date, '^\\d{4}')), -1) AS d,
count(),
round(avg(price), 2),
bar(avg(price), 0, 50, 100)
FROM menu_item_denorm
WHERE (menu_currency IN ('Dollars', '')) AND (d > 0) AND (d < 2022) AND (dish_name ILIKE '%vodka%')
GROUP BY d
ORDER BY d ASC;
```
```text title="Response" theme={null}
┌────d─┬─count()─┬─round(avg(price), 2)─┬─bar(avg(price), 0, 50, 100)─┐
│ 1910 │ 2 │ 0 │ │
│ 1920 │ 1 │ 0.3 │ ▌ │
│ 1940 │ 21 │ 0.42 │ ▋ │
│ 1950 │ 14 │ 0.59 │ █▏ │
│ 1960 │ 113 │ 2.17 │ ████▎ │
│ 1970 │ 37 │ 0.68 │ █▎ │
│ 1980 │ 19 │ 2.55 │ █████ │
│ 1990 │ 86 │ 3.6 │ ███████▏ │
│ 2000 │ 2 │ 3.98 │ ███████▊ │
└──────┴─────────┴──────────────────────┴─────────────────────────────┘
```
To get vodka we have to write `ILIKE '%vodka%'` and this definitely makes a statement.
### Caviar
Let's print caviar prices. Also let's print a name of any dish with caviar.
```sql title="Query" theme={null}
SELECT
round(toUInt32OrZero(extract(menu_date, '^\\d{4}')), -1) AS d,
count(),
round(avg(price), 2),
bar(avg(price), 0, 50, 100),
any(dish_name)
FROM menu_item_denorm
WHERE (menu_currency IN ('Dollars', '')) AND (d > 0) AND (d < 2022) AND (dish_name ILIKE '%caviar%')
GROUP BY d
ORDER BY d ASC;
```
```text title="Response" theme={null}
┌────d─┬─count()─┬─round(avg(price), 2)─┬─bar(avg(price), 0, 50, 100)──────┬─any(dish_name)──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1090 │ 1 │ 0 │ │ Caviar │
│ 1880 │ 3 │ 0 │ │ Caviar │
│ 1890 │ 39 │ 0.59 │ █▏ │ Butter and caviar │
│ 1900 │ 1014 │ 0.34 │ ▋ │ Anchovy Caviar on Toast │
│ 1910 │ 1588 │ 1.35 │ ██▋ │ 1/1 Brötchen Caviar │
│ 1920 │ 927 │ 1.37 │ ██▋ │ ASTRAKAN CAVIAR │
│ 1930 │ 289 │ 1.91 │ ███▋ │ Astrachan caviar │
│ 1940 │ 201 │ 0.83 │ █▋ │ (SPECIAL) Domestic Caviar Sandwich │
│ 1950 │ 81 │ 2.27 │ ████▌ │ Beluga Caviar │
│ 1960 │ 126 │ 2.21 │ ████▍ │ Beluga Caviar │
│ 1970 │ 105 │ 0.95 │ █▊ │ BELUGA MALOSSOL CAVIAR AMERICAN DRESSING │
│ 1980 │ 12 │ 7.22 │ ██████████████▍ │ Authentic Iranian Beluga Caviar the world's finest black caviar presented in ice garni and a sampling of chilled 100° Russian vodka │
│ 1990 │ 74 │ 14.42 │ ████████████████████████████▋ │ Avocado Salad, Fresh cut avocado with caviare │
│ 2000 │ 3 │ 7.82 │ ███████████████▋ │ Aufgeschlagenes Kartoffelsueppchen mit Forellencaviar │
│ 2010 │ 6 │ 15.58 │ ███████████████████████████████▏ │ "OYSTERS AND PEARLS" "Sabayon" of Pearl Tapioca with Island Creek Oysters and Russian Sevruga Caviar │
└──────┴─────────┴──────────────────────┴──────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
At least they have caviar with vodka. Very nice.
## Online playground
The data is uploaded to ClickHouse Playground, [example](https://sql.clickhouse.com?query_id=KB5KQJJFNBKHE5GBUJCP1B).
# NOAA Global Historical Climatology Network
Source: https://clickhouse.com/docs/get-started/sample-datasets/noaa
2.5 billion rows of climate data for the last 120 yrs
This dataset contains weather measurements for the last 120 years. Each row is a measurement for a point in time and station.
More precisely and according to the [origin of this data](https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-ghcn):
> GHCN-Daily is a dataset that contains daily observations over global land areas. It contains station-based measurements from land-based stations worldwide, about two-thirds of which are for precipitation measurements only (Menne et al., 2012). GHCN-Daily is a composite of climate records from numerous sources that were merged together and subjected to a common suite of quality assurance reviews (Durre et al., 2010). The archive includes the following meteorological elements:
* Daily maximum temperature
* Daily minimum temperature
* Temperature at the time of observation
* Precipitation (i.e., rain, melted snow)
* Snowfall
* Snow depth
* Other elements where available
The sections below give a brief overview of the steps that were involved in bringing this dataset into ClickHouse. If you're interested in reading about each step in more detail, we recommend to take a look at our blog post titled ["Exploring massive, real-world data sets: 100+ Years of Weather Records in ClickHouse"](https://clickhouse.com/blog/real-world-data-noaa-climate-data).
## Downloading the data
* A [pre-prepared version](#pre-prepared-data) of the data for ClickHouse, which has been cleansed, re-structured, and enriched. This data covers the years 1900 to 2022.
* [Download the original data](#original-data) and convert to the format required by ClickHouse. Users wanting to add their own columns may wish to explore this approach.
### Pre-prepared data
More specifically, rows have been removed that didn't fail any quality assurance checks by Noaa. The data has also been restructured from a measurement per line to a row per station id and date, i.e.
```csv theme={null}
"station_id","date","tempAvg","tempMax","tempMin","precipitation","snowfall","snowDepth","percentDailySun","averageWindSpeed","maxWindSpeed","weatherType"
"AEM00041194","2022-07-30",347,0,308,0,0,0,0,0,0,0
"AEM00041194","2022-07-31",371,413,329,0,0,0,0,0,0,0
"AEM00041194","2022-08-01",384,427,357,0,0,0,0,0,0,0
"AEM00041194","2022-08-02",381,424,352,0,0,0,0,0,0,0
```
This is simpler to query and ensures the resulting table is less sparse. Finally, the data has also been enriched with latitude and longitude.
This data is available in the following S3 location. Either download the data to your local filesystem (and insert using the ClickHouse client) or insert directly into ClickHouse (see [Inserting from S3](#inserting-from-s3)).
To download:
```bash theme={null}
wget https://datasets-documentation.s3.eu-west-3.amazonaws.com/noaa/noaa_enriched.parquet
```
### Original data
The following details the steps to download and transform the original data in preparation for loading into ClickHouse.
#### Download
To download the original data:
```bash theme={null}
for i in {1900..2023}; do wget https://noaa-ghcn-pds.s3.amazonaws.com/csv.gz/${i}.csv.gz; done
```
#### Sampling the data
```bash theme={null}
$ clickhouse-local --query "SELECT * FROM '2021.csv.gz' LIMIT 10" --format PrettyCompact
┌─c1──────────┬───────c2─┬─c3───┬──c4─┬─c5───┬─c6───┬─c7─┬───c8─┐
│ AE000041196 │ 20210101 │ TMAX │ 278 │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
│ AE000041196 │ 20210101 │ PRCP │ 0 │ D │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
│ AE000041196 │ 20210101 │ TAVG │ 214 │ H │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
│ AEM00041194 │ 20210101 │ TMAX │ 266 │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
│ AEM00041194 │ 20210101 │ TMIN │ 178 │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
│ AEM00041194 │ 20210101 │ PRCP │ 0 │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
│ AEM00041194 │ 20210101 │ TAVG │ 217 │ H │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
│ AEM00041217 │ 20210101 │ TMAX │ 262 │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
│ AEM00041217 │ 20210101 │ TMIN │ 155 │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
│ AEM00041217 │ 20210101 │ TAVG │ 202 │ H │ ᴺᵁᴸᴸ │ S │ ᴺᵁᴸᴸ │
└─────────────┴──────────┴──────┴─────┴──────┴──────┴────┴──────┘
```
Summarizing the [format documentation](https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-ghcn):
Summarizing the format documentation and the columns in order:
* An 11 character station identification code. This itself encodes some useful information
* YEAR/MONTH/DAY = 8 character date in YYYYMMDD format (e.g. 19860529 = May 29, 1986)
* ELEMENT = 4 character indicator of element type. Effectively the measurement type. While there are many measurements available, we select the following:
* PRCP - Precipitation (tenths of mm)
* SNOW - Snowfall (mm)
* SNWD - Snow depth (mm)
* TMAX - Maximum temperature (tenths of degrees C)
* TAVG - Average temperature (tenths of a degree C)
* TMIN - Minimum temperature (tenths of degrees C)
* PSUN - Daily percent of possible sunshine (percent)
* AWND - Average daily wind speed (tenths of meters per second)
* WSFG - Peak gust wind speed (tenths of meters per second)
* WT\*\* = Weather Type where \*\* defines the weather type. Full list of weather types here.
* DATA VALUE = 5 character data value for ELEMENT i.e. the value of the measurement.
* M-FLAG = 1 character Measurement Flag. This has 10 possible values. Some of these values indicate questionable data accuracy. We accept data where this is set to "P" - identified as missing presumed zero, as this is only relevant to the PRCP, SNOW and SNWD measurements.
* Q-FLAG is the measurement quality flag with 14 possible values. We're only interested in data with an empty value i.e. it didn't fail any quality assurance checks.
* S-FLAG is the source flag for the observation. Not useful for our analysis and ignored.
* OBS-TIME = 4-character time of observation in hour-minute format (i.e. 0700 =7:00 am). Typically not present in older data. We ignore this for our purposes.
A measurement per line would result in a sparse table structure in ClickHouse. We should transform to a row per time and station, with measurements as columns. First, we limit the dataset to those rows without issues i.e. where `qFlag` is equal to an empty string.
#### Clean the data
Using [ClickHouse local](https://clickhouse.com/blog/extracting-converting-querying-local-files-with-sql-clickhouse-local) we can filter rows that represent measurements of interest and pass our quality requirements:
```bash theme={null}
clickhouse local --query "SELECT count()
FROM file('*.csv.gz', CSV, 'station_id String, date String, measurement String, value Int64, mFlag String, qFlag String, sFlag String, obsTime String') WHERE qFlag = '' AND (measurement IN ('PRCP', 'SNOW', 'SNWD', 'TMAX', 'TAVG', 'TMIN', 'PSUN', 'AWND', 'WSFG') OR startsWith(measurement, 'WT'))"
2679264563
```
With over 2.6 billion rows, this isn't a fast query since it involves parsing all the files. On our 8 core machine, this takes around 160 seconds.
### Pivot data
While the measurement per line structure can be used with ClickHouse, it will unnecessarily complicate future queries. Ideally, we need a row per station id and date, where each measurement type and associated value are a column i.e.
```csv theme={null}
"station_id","date","tempAvg","tempMax","tempMin","precipitation","snowfall","snowDepth","percentDailySun","averageWindSpeed","maxWindSpeed","weatherType"
"AEM00041194","2022-07-30",347,0,308,0,0,0,0,0,0,0
"AEM00041194","2022-07-31",371,413,329,0,0,0,0,0,0,0
"AEM00041194","2022-08-01",384,427,357,0,0,0,0,0,0,0
"AEM00041194","2022-08-02",381,424,352,0,0,0,0,0,0,0
```
Using ClickHouse local and a simple `GROUP BY`, we can repivot our data to this structure. To limit memory overhead, we do this one file at a time.
```bash theme={null}
for i in {1900..2022}
do
clickhouse-local --query "SELECT station_id,
toDate32(date) as date,
anyIf(value, measurement = 'TAVG') as tempAvg,
anyIf(value, measurement = 'TMAX') as tempMax,
anyIf(value, measurement = 'TMIN') as tempMin,
anyIf(value, measurement = 'PRCP') as precipitation,
anyIf(value, measurement = 'SNOW') as snowfall,
anyIf(value, measurement = 'SNWD') as snowDepth,
anyIf(value, measurement = 'PSUN') as percentDailySun,
anyIf(value, measurement = 'AWND') as averageWindSpeed,
anyIf(value, measurement = 'WSFG') as maxWindSpeed,
toUInt8OrZero(replaceOne(anyIf(measurement, startsWith(measurement, 'WT') AND value = 1), 'WT', '')) as weatherType
FROM file('$i.csv.gz', CSV, 'station_id String, date String, measurement String, value Int64, mFlag String, qFlag String, sFlag String, obsTime String')
WHERE qFlag = '' AND (measurement IN ('PRCP', 'SNOW', 'SNWD', 'TMAX', 'TAVG', 'TMIN', 'PSUN', 'AWND', 'WSFG') OR startsWith(measurement, 'WT'))
GROUP BY station_id, date
ORDER BY station_id, date FORMAT CSV" >> "noaa.csv";
done
```
This query produces a single 50GB file `noaa.csv`.
### Enriching the data
The data has no indication of location aside from a station id, which includes a prefix country code. Ideally, each station would have a latitude and longitude associated with it. To achieve this, NOAA conveniently provides the details of each station as a separate [ghcnd-stations.txt](https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-ghcn#format-of-ghcnd-stationstxt-file). This file has [several columns](https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-ghcn#format-of-ghcnd-stationstxt-file), of which five are useful to our future analysis: id, latitude, longitude, elevation, and name.
```bash theme={null}
wget http://noaa-ghcn-pds.s3.amazonaws.com/ghcnd-stations.txt
```
```bash theme={null}
clickhouse local --query "WITH stations AS (SELECT id, lat, lon, elevation, splitByString(' GSN ',name)[1] as name FROM file('ghcnd-stations.txt', Regexp, 'id String, lat Float64, lon Float64, elevation Float32, name String'))
SELECT station_id,
date,
tempAvg,
tempMax,
tempMin,
precipitation,
snowfall,
snowDepth,
percentDailySun,
averageWindSpeed,
maxWindSpeed,
weatherType,
tuple(lon, lat) as location,
elevation,
name
FROM file('noaa.csv', CSV,
'station_id String, date Date32, tempAvg Int32, tempMax Int32, tempMin Int32, precipitation Int32, snowfall Int32, snowDepth Int32, percentDailySun Int8, averageWindSpeed Int32, maxWindSpeed Int32, weatherType UInt8') as noaa LEFT OUTER
JOIN stations ON noaa.station_id = stations.id INTO OUTFILE 'noaa_enriched.parquet' FORMAT Parquet SETTINGS format_regexp='^(.{11})\s+(\-?\d{1,2}\.\d{4})\s+(\-?\d{1,3}\.\d{1,4})\s+(\-?\d*\.\d*)\s+(.*)\s+(?:[\d]*)'"
```
This query takes a few minutes to run and produces a 6.4 GB file, `noaa_enriched.parquet`.
## Create table
Create a MergeTree table in ClickHouse (from the ClickHouse client).
```sql theme={null}
CREATE TABLE noaa
(
`station_id` LowCardinality(String),
`date` Date32,
`tempAvg` Int32 COMMENT 'Average temperature (tenths of a degrees C)',
`tempMax` Int32 COMMENT 'Maximum temperature (tenths of degrees C)',
`tempMin` Int32 COMMENT 'Minimum temperature (tenths of degrees C)',
`precipitation` UInt32 COMMENT 'Precipitation (tenths of mm)',
`snowfall` UInt32 COMMENT 'Snowfall (mm)',
`snowDepth` UInt32 COMMENT 'Snow depth (mm)',
`percentDailySun` UInt8 COMMENT 'Daily percent of possible sunshine (percent)',
`averageWindSpeed` UInt32 COMMENT 'Average daily wind speed (tenths of meters per second)',
`maxWindSpeed` UInt32 COMMENT 'Peak gust wind speed (tenths of meters per second)',
`weatherType` Enum8('Normal' = 0, 'Fog' = 1, 'Heavy Fog' = 2, 'Thunder' = 3, 'Small Hail' = 4, 'Hail' = 5, 'Glaze' = 6, 'Dust/Ash' = 7, 'Smoke/Haze' = 8, 'Blowing/Drifting Snow' = 9, 'Tornado' = 10, 'High Winds' = 11, 'Blowing Spray' = 12, 'Mist' = 13, 'Drizzle' = 14, 'Freezing Drizzle' = 15, 'Rain' = 16, 'Freezing Rain' = 17, 'Snow' = 18, 'Unknown Precipitation' = 19, 'Ground Fog' = 21, 'Freezing Fog' = 22),
`location` Point,
`elevation` Float32,
`name` LowCardinality(String)
) ENGINE = MergeTree() ORDER BY (station_id, date);
```
## Inserting into ClickHouse
### Inserting from local file
Data can be inserted from a local file as follows (from the ClickHouse client):
```sql theme={null}
INSERT INTO noaa FROM INFILE '/noaa_enriched.parquet'
```
where `` represents the full path to the local file on disk.
See [here](https://clickhouse.com/blog/real-world-data-noaa-climate-data#load-the-data) for how to speed this load up.
### Inserting from S3
```sql theme={null}
INSERT INTO noaa SELECT *
FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/noaa/noaa_enriched.parquet', NOSIGN)
```
For how to speed this up, see our blog post on [tuning large data loads](https://clickhouse.com/blog/supercharge-your-clickhouse-data-loads-part2).
## Sample queries
### Highest temperature ever
```sql theme={null}
SELECT
tempMax / 10 AS maxTemp,
location,
name,
date
FROM blogs.noaa
WHERE tempMax > 500
ORDER BY
tempMax DESC,
date ASC
LIMIT 5
```
```response theme={null}
┌─maxTemp─┬─location──────────┬─name───────────────────────────────────────────┬───────date─┐
│ 56.7 │ (-116.8667,36.45) │ CA GREENLAND RCH │ 1913-07-10 │
│ 56.7 │ (-115.4667,32.55) │ MEXICALI (SMN) │ 1949-08-20 │
│ 56.7 │ (-115.4667,32.55) │ MEXICALI (SMN) │ 1949-09-18 │
│ 56.7 │ (-115.4667,32.55) │ MEXICALI (SMN) │ 1952-07-17 │
│ 56.7 │ (-115.4667,32.55) │ MEXICALI (SMN) │ 1952-09-04 │
└─────────┴───────────────────┴────────────────────────────────────────────────┴────────────┘
5 rows in set. Elapsed: 0.514 sec. Processed 1.06 billion rows, 4.27 GB (2.06 billion rows/s., 8.29 GB/s.)
```
Reassuringly consistent with the [documented record](https://en.wikipedia.org/wiki/List_of_weather_records#Highest_temperatures_ever_recorded) at [Furnace Creek](https://www.google.com/maps/place/36%C2%B027'00.0%22N+116%C2%B052'00.1%22W/@36.1329666,-116.1104099,8.95z/data=!4m5!3m4!1s0x0:0xf2ed901b860f4446!8m2!3d36.45!4d-116.8667) as of 2023.
### Best ski resorts
Using a [list of ski resorts](https://gist.githubusercontent.com/gingerwizard/dd022f754fd128fdaf270e58fa052e35/raw/622e03c37460f17ef72907afe554cb1c07f91f23/ski_resort_stats.csv) in the united states and their respective locations, we join these against the top 1000 weather stations with the most in any month in the last 5 yrs. Sorting this join by [geoDistance](/docs/reference/functions/regular-functions/geo/coordinates#geodistance) and restricting the results to those where the distance is less than 20km, we select the top result per resort and sort this by total snow. Note we also restrict resorts to those above 1800m, as a broad indicator of good skiing conditions.
```sql theme={null}
SELECT
resort_name,
total_snow / 1000 AS total_snow_m,
resort_location,
month_year
FROM
(
WITH resorts AS
(
SELECT
resort_name,
state,
(lon, lat) AS resort_location,
'US' AS code
FROM url('https://gist.githubusercontent.com/gingerwizard/dd022f754fd128fdaf270e58fa052e35/raw/622e03c37460f17ef72907afe554cb1c07f91f23/ski_resort_stats.csv', CSVWithNames)
)
SELECT
resort_name,
highest_snow.station_id,
geoDistance(resort_location.1, resort_location.2, station_location.1, station_location.2) / 1000 AS distance_km,
highest_snow.total_snow,
resort_location,
station_location,
month_year
FROM
(
SELECT
sum(snowfall) AS total_snow,
station_id,
any(location) AS station_location,
month_year,
substring(station_id, 1, 2) AS code
FROM noaa
WHERE (date > '2017-01-01') AND (code = 'US') AND (elevation > 1800)
GROUP BY
station_id,
toYYYYMM(date) AS month_year
ORDER BY total_snow DESC
LIMIT 1000
) AS highest_snow
INNER JOIN resorts ON highest_snow.code = resorts.code
WHERE distance_km < 20
ORDER BY
resort_name ASC,
total_snow DESC
LIMIT 1 BY
resort_name,
station_id
)
ORDER BY total_snow DESC
LIMIT 5
```
```response theme={null}
┌─resort_name──────────┬─total_snow_m─┬─resort_location─┬─month_year─┐
│ Sugar Bowl, CA │ 7.799 │ (-120.3,39.27) │ 201902 │
│ Donner Ski Ranch, CA │ 7.799 │ (-120.34,39.31) │ 201902 │
│ Boreal, CA │ 7.799 │ (-120.35,39.33) │ 201902 │
│ Homewood, CA │ 4.926 │ (-120.17,39.08) │ 201902 │
│ Alpine Meadows, CA │ 4.926 │ (-120.22,39.17) │ 201902 │
└──────────────────────┴──────────────┴─────────────────┴────────────┘
5 rows in set. Elapsed: 0.750 sec. Processed 689.10 million rows, 3.20 GB (918.20 million rows/s., 4.26 GB/s.)
Peak memory usage: 67.66 MiB.
```
## Credits
We would like to acknowledge the efforts of the Global Historical Climatology Network for preparing, cleansing, and distributing this data. We appreciate your efforts.
Menne, M.J., I. Durre, B. Korzeniewski, S. McNeal, K. Thomas, X. Yin, S. Anthony, R. Ray, R.S. Vose, B.E.Gleason, and T.G. Houston, 2012: Global Historical Climatology Network - Daily (GHCN-Daily), Version 3. \[indicate subset used following decimal, e.g. Version 3.25]. NOAA National Centers for Environmental Information. [http://doi.org/10.7289/V5D21VHZ](http://doi.org/10.7289/V5D21VHZ) \[17/08/2020]
# Geospatial analysis with the New York taxi dataset
Source: https://clickhouse.com/docs/get-started/sample-datasets/nyc-taxi
Explore ClickHouse using a dataset for billions of taxi and for-hire vehicle (Uber, Lyft, etc.) trips originating in New York City since 2009
**Cloud**
In this tutorial you'll explore how ClickHouse can be used to run analytical queries over large amounts of data.
You'll also see how you can use a dictionary to enrich the data and write join queries.
## Prerequisites
For this tutorial, you'll need:
* A [ClickHouse Cloud account](https://clickhouse.cloud/signUp?loc=docs-sample-datasets-nyc-taxi) (\$300 in free credits when signing up)
* [A ClickHouse Cloud service](/docs/get-started/setup/cloud#1-create-a-clickhouse-service)
## 1. Create the table
The dataset used in this tutorial is the New York City taxi dataset, which contains details about millions of taxi rides, with columns including tip amount, tolls, payment type, and more.
1. Select **SQL console** from the the left hand menu
2. Click the **+** tab next to the home icon to create a new query
3. In the SQL editor type the following query, then click **Run**:
```sql Expandable theme={null}
CREATE TABLE trips
(
`trip_id` UInt32,
`vendor_id` Enum8('1' = 1, '2' = 2, '3' = 3, '4' = 4, 'CMT' = 5, 'VTS' = 6, 'DDS' = 7, 'B02512' = 10, 'B02598' = 11, 'B02617' = 12, 'B02682' = 13, 'B02764' = 14, '' = 15),
`pickup_date` Date,
`pickup_datetime` DateTime,
`dropoff_date` Date,
`dropoff_datetime` DateTime,
`store_and_fwd_flag` UInt8,
`rate_code_id` UInt8,
`pickup_longitude` Float64,
`pickup_latitude` Float64,
`dropoff_longitude` Float64,
`dropoff_latitude` Float64,
`passenger_count` UInt8,
`trip_distance` Float64,
`fare_amount` Float32,
`extra` Float32,
`mta_tax` Float32,
`tip_amount` Float32,
`tolls_amount` Float32,
`ehail_fee` Float32,
`improvement_surcharge` Float32,
`total_amount` Float32,
`payment_type` Enum8('UNK' = 0, 'CSH' = 1, 'CRE' = 2, 'NOC' = 3, 'DIS' = 4),
`trip_type` UInt8,
`pickup` FixedString(25),
`dropoff` FixedString(25),
`cab_type` Enum8('yellow' = 1, 'green' = 2, 'uber' = 3),
`pickup_nyct2010_gid` Int8,
`pickup_ctlabel` Float32,
`pickup_borocode` Int8,
`pickup_ct2010` String,
`pickup_boroct2010` String,
`pickup_cdeligibil` String,
`pickup_ntacode` FixedString(4),
`pickup_ntaname` String,
`pickup_puma` UInt16,
`dropoff_nyct2010_gid` UInt8,
`dropoff_ctlabel` Float32,
`dropoff_borocode` UInt8,
`dropoff_ct2010` String,
`dropoff_boroct2010` String,
`dropoff_cdeligibil` String,
`dropoff_ntacode` FixedString(4),
`dropoff_ntaname` String,
`dropoff_puma` UInt16
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(pickup_date)
ORDER BY pickup_datetime;
```
## 2. Insert the data
Now that you’ve created a table, add the New York City taxi data from CSV files in S3.
The following command inserts \~2,000,000 rows into your trips table from two different files in S3: `trips_1.tsv.gz` and `trips_2.tsv.gz`:
```sql Expandable theme={null}
INSERT INTO trips
SELECT * FROM s3(
'https://datasets-documentation.s3.eu-west-3.amazonaws.com/nyc-taxi/trips_{1..2}.gz',
'TabSeparatedWithNames', "
`trip_id` UInt32,
`vendor_id` Enum8('1' = 1, '2' = 2, '3' = 3, '4' = 4, 'CMT' = 5, 'VTS' = 6, 'DDS' = 7, 'B02512' = 10, 'B02598' = 11, 'B02617' = 12, 'B02682' = 13, 'B02764' = 14, '' = 15),
`pickup_date` Date,
`pickup_datetime` DateTime,
`dropoff_date` Date,
`dropoff_datetime` DateTime,
`store_and_fwd_flag` UInt8,
`rate_code_id` UInt8,
`pickup_longitude` Float64,
`pickup_latitude` Float64,
`dropoff_longitude` Float64,
`dropoff_latitude` Float64,
`passenger_count` UInt8,
`trip_distance` Float64,
`fare_amount` Float32,
`extra` Float32,
`mta_tax` Float32,
`tip_amount` Float32,
`tolls_amount` Float32,
`ehail_fee` Float32,
`improvement_surcharge` Float32,
`total_amount` Float32,
`payment_type` Enum8('UNK' = 0, 'CSH' = 1, 'CRE' = 2, 'NOC' = 3, 'DIS' = 4),
`trip_type` UInt8,
`pickup` FixedString(25),
`dropoff` FixedString(25),
`cab_type` Enum8('yellow' = 1, 'green' = 2, 'uber' = 3),
`pickup_nyct2010_gid` Int8,
`pickup_ctlabel` Float32,
`pickup_borocode` Int8,
`pickup_ct2010` String,
`pickup_boroct2010` String,
`pickup_cdeligibil` String,
`pickup_ntacode` FixedString(4),
`pickup_ntaname` String,
`pickup_puma` UInt16,
`dropoff_nyct2010_gid` UInt8,
`dropoff_ctlabel` Float32,
`dropoff_borocode` UInt8,
`dropoff_ct2010` String,
`dropoff_boroct2010` String,
`dropoff_cdeligibil` String,
`dropoff_ntacode` FixedString(4),
`dropoff_ntaname` String,
`dropoff_puma` UInt16
") SETTINGS input_format_try_infer_datetimes = 0
```
Wait for the data to finish inserting. Around 150MB of data will be downloaded.
When the data is finished inserting, check the number of rows in the `trips` table:
```sql theme={null}
SELECT count() FROM trips
```
You should get a result of 1,999,657 rows
## 3. Analyze the data
With the data loaded, you can run some queries to analyze the data.
* Calculate the average tip amount:
```sql theme={null}
SELECT round(avg(tip_amount), 2) FROM trips
```
* Calculate the average cost based on the number of passengers:
```sql theme={null}
SELECT
passenger_count,
ceil(avg(total_amount),2) AS average_total_amount
FROM trips
GROUP BY passenger_count
```
* Calculate the daily number of pickups per neighborhood:
```sql theme={null}
SELECT
pickup_date,
pickup_ntaname,
SUM(1) AS number_of_trips
FROM trips
GROUP BY pickup_date, pickup_ntaname
ORDER BY pickup_date ASC
```
* Calculate the length of each trip in minutes, then group the results by trip length:
```sql theme={null}
SELECT
avg(tip_amount) AS avg_tip,
avg(fare_amount) AS avg_fare,
avg(passenger_count) AS avg_passenger,
count() AS count,
truncate(date_diff('second', pickup_datetime, dropoff_datetime)/60) as trip_minutes
FROM trips
WHERE trip_minutes > 0
GROUP BY trip_minutes
ORDER BY trip_minutes DESC
```
* Show the number of pickups in each neighborhood broken down by hour of the day:
```sql theme={null}
SELECT
pickup_ntaname,
toHour(pickup_datetime) as pickup_hour,
SUM(1) AS pickups
FROM trips
WHERE pickup_ntaname != ''
GROUP BY pickup_ntaname, pickup_hour
ORDER BY pickup_ntaname, pickup_hour
```
## 4. Create a dictionary
Next you'll create a dictionary (a mapping of key-value pairs stored in memory) called `taxi_zone_dictionary` to provide a mapping between location IDs and NYC borough names, sourced from a CSV file containing every New York City neighborhood.
These correspond to the `pickup_nyct2010_gid` and `dropoff_nyct2010_gid` columns in the trips table.
Here's an excerpt from the CSV file you're using in table format. The `LocationID` column in the file maps to the `pickup_nyct2010_gid` and `dropoff_nyct2010_gid` columns in your `trips` table:
| LocationID | Borough | Zone | service\_zone |
| - | - | - | - |
| 1 | EWR | Newark Airport | EWR |
| 2 | Queens | Jamaica Bay | Boro Zone |
| 3 | Bronx | Allerton/Pelham Gardens | Boro Zone |
| 4 | Manhattan | Alphabet City | Yellow Zone |
| 5 | Staten Island | Arden Heights | Boro Zone |
Run the following SQL command, which creates a dictionary named `taxi_zone_dictionary` and populates the dictionary from the CSV file in S3. The URL for the file is `https://datasets-documentation.s3.eu-west-3.amazonaws.com/nyc-taxi/taxi_zone_lookup.csv`.
```sql theme={null}
CREATE DICTIONARY taxi_zone_dictionary
(
`LocationID` UInt16 DEFAULT 0,
`Borough` String,
`Zone` String,
`service_zone` String
)
PRIMARY KEY LocationID
SOURCE(HTTP(URL 'https://datasets-documentation.s3.eu-west-3.amazonaws.com/nyc-taxi/taxi_zone_lookup.csv' FORMAT 'CSVWithNames'))
LIFETIME(MIN 0 MAX 0)
LAYOUT(HASHED_ARRAY())
```
> [!NOTE]
>
> Setting `LIFETIME` to 0 disables automatic updates to avoid unnecessary traffic to our S3 bucket. In other cases, you might configure it differently. For details, see [Refreshing dictionary data using LIFETIME](/docs/reference/statements/create/dictionary/lifetime).
Verify it worked. The following query should return 265 rows, or one row for each neighborhood:
```sql theme={null}
SELECT * FROM taxi_zone_dictionary
```
## 5. Run queries using the dictionary
You can use the `dictGet` function ([or its variations](/docs/reference/functions/regular-functions/ext-dict-functions)) to retrieve a value from a dictionary.
You pass in the name of the dictionary, the value you want, and the key (which in our example is the `LocationID` column of `taxi_zone_dictionary`), and get back the matching value.
For example, the following query returns the `Borough` whose `LocationID` is 132 (corresponding to JFK airport):
```sql theme={null}
SELECT dictGet('taxi_zone_dictionary', 'Borough', 132)
```
JFK is in Queens. Notice the time to retrieve the value is essentially 0:
```response theme={null}
┌─dictGet('taxi_zone_dictionary', 'Borough', 132)─┐
│ Queens │
└─────────────────────────────────────────────────┘
1 rows in set. Elapsed: 0.004 sec.
```
Use the `dictHas` function to see if a key is present in the dictionary. For example, the following query returns `1` (which is "true" in ClickHouse):
```sql theme={null}
SELECT dictHas('taxi_zone_dictionary', 132)
```
The following query returns 0 because 4567 isn't a value of `LocationID` in the dictionary:
```sql theme={null}
SELECT dictHas('taxi_zone_dictionary', 4567)
```
Use the `dictGet` function to retrieve a borough's name in a query. For example:
```sql theme={null}
SELECT
count(1) AS total,
dictGetOrDefault('taxi_zone_dictionary','Borough', toUInt64(pickup_nyct2010_gid), 'Unknown') AS borough_name
FROM trips
WHERE dropoff_nyct2010_gid = 132 OR dropoff_nyct2010_gid = 138
GROUP BY borough_name
ORDER BY total DESC
```
This query sums up the number of taxi rides per borough that end at either the LaGuardia or JFK airport. The result looks like the following, and notice there are quite a few trips where the pickup neighborhood is unknown:
```response theme={null}
┌─total─┬─borough_name──┐
│ 23683 │ Unknown │
│ 7053 │ Manhattan │
│ 6828 │ Brooklyn │
│ 4458 │ Queens │
│ 2670 │ Bronx │
│ 554 │ Staten Island │
│ 53 │ EWR │
└───────┴───────────────┘
7 rows in set. Elapsed: 0.019 sec. Processed 2.00 million rows, 4.00 MB (105.70 million rows/s., 211.40 MB/s.)
```
## 6. Perform a join
Finally, write some queries that join the `taxi_zone_dictionary` with your `trips` table.
Start with a simple `JOIN` that acts similarly to the previous airport query above:
```sql theme={null}
SELECT
count(1) AS total,
Borough
FROM trips
JOIN taxi_zone_dictionary ON toUInt64(trips.pickup_nyct2010_gid) = taxi_zone_dictionary.LocationID
WHERE dropoff_nyct2010_gid = 132 OR dropoff_nyct2010_gid = 138
GROUP BY Borough
ORDER BY total DESC
```
The response looks identical to the `dictGet` query:
```response theme={null}
┌─total─┬─Borough───────┐
│ 7053 │ Manhattan │
│ 6828 │ Brooklyn │
│ 4458 │ Queens │
│ 2670 │ Bronx │
│ 554 │ Staten Island │
│ 53 │ EWR │
└───────┴───────────────┘
6 rows in set. Elapsed: 0.034 sec. Processed 2.00 million rows, 4.00 MB (59.14 million rows/s., 118.29 MB/s.)
```
> [!NOTE]
>
> Notice the output of the above `JOIN` query is the same as the query before it that used `dictGetOrDefault` (except that the `Unknown` values aren't included).
> Behind the scenes, ClickHouse is actually calling the `dictGet` function for the `taxi_zone_dictionary` dictionary, but the `JOIN` syntax is more familiar for SQL developers.
This query returns rows for the 1000 trips with the highest tip amount, then performs an inner join of each row with the dictionary:
```sql theme={null}
SELECT *
FROM trips
JOIN taxi_zone_dictionary
ON trips.dropoff_nyct2010_gid = taxi_zone_dictionary.LocationID
WHERE tip_amount > 0
ORDER BY tip_amount DESC
LIMIT 1000
```
> [!TIP]
>
> Prefer listing only the columns your query needs rather than using SELECT \*. Because ClickHouse stores data by column, selecting fewer columns directly reduces the data read, decompressed, and processed.
## Next steps
Learn more about ClickHouse with the following documentation:
* [Introduction to primary indexes in ClickHouse](/docs/guides/clickhouse/data-modelling/sparse-primary-indexes): Learn how ClickHouse uses sparse primary indexes to efficiently locate relevant data during queries.
* [Integrate an external data source](/docs/integrations/home): Review data source integration options, including files, Kafka, PostgreSQL, data pipelines, and many others.
* [Visualize data in ClickHouse](/docs/integrations/connectors/data-visualization/index): Connect your favorite UI/BI tool to ClickHouse.
* [SQL Reference](/docs/reference/home): Browse the SQL functions available in ClickHouse for transforming, processing and analyzing data.
**Open Source**
The New York taxi data sample consists of 3+ billion taxi and for-hire vehicle (Uber, Lyft, etc.) trips originating in New York City since 2009. This getting started guide uses a 3m row sample.
The full dataset can be obtained in a couple of ways:
* insert the data directly into ClickHouse Cloud from S3 or GCS
* download prepared partitions
* Alternatively you can query the full dataset in our demo environment at [sql.clickhouse.com](https://sql.clickhouse.com/?query=U0VMRUNUIGNvdW50KCkgRlJPTSBueWNfdGF4aS50cmlwcw\&chart=eyJ0eXBlIjoibGluZSIsImNvbmZpZyI6eyJ0aXRsZSI6IlRlbXBlcmF0dXJlIGJ5IGNvdW50cnkgYW5kIHllYXIiLCJ4YXhpcyI6InllYXIiLCJ5YXhpcyI6ImNvdW50KCkiLCJzZXJpZXMiOiJDQVNUKHBhc3Nlbmdlcl9jb3VudCwgJ1N0cmluZycpIn19).
> [!NOTE]
>
> The example queries below were executed on a **Production** instance of ClickHouse Cloud. For more information see
> ["Playground specifications"](/docs/get-started/sample-datasets/playground#specifications).
## Create the table trips
Start by creating a table for the taxi rides:
```sql theme={null}
CREATE DATABASE nyc_taxi;
CREATE TABLE nyc_taxi.trips_small (
trip_id UInt32,
pickup_datetime DateTime,
dropoff_datetime DateTime,
pickup_longitude Nullable(Float64),
pickup_latitude Nullable(Float64),
dropoff_longitude Nullable(Float64),
dropoff_latitude Nullable(Float64),
passenger_count UInt8,
trip_distance Float32,
fare_amount Float32,
extra Float32,
tip_amount Float32,
tolls_amount Float32,
total_amount Float32,
payment_type Enum('CSH' = 1, 'CRE' = 2, 'NOC' = 3, 'DIS' = 4, 'UNK' = 5),
pickup_ntaname LowCardinality(String),
dropoff_ntaname LowCardinality(String)
)
ENGINE = MergeTree
PRIMARY KEY (pickup_datetime, dropoff_datetime);
```
## Load the data directly from object storage
Users' can grab a small subset of the data (3 million rows) for getting familiar with it. The data is in TSV files in object storage, which is easily streamed into
ClickHouse Cloud using the `s3` table function.
The same data is stored in both S3 and GCS; choose either tab.
> [tab] S3
>
> The following command streams three files from an S3 bucket into the `trips_small` table (the `{0..2}` syntax is a wildcard for the values 0, 1, and 2):
>
> ```sql theme={null}
> INSERT INTO nyc_taxi.trips_small
> SELECT
> trip_id,
> pickup_datetime,
> dropoff_datetime,
> pickup_longitude,
> pickup_latitude,
> dropoff_longitude,
> dropoff_latitude,
> passenger_count,
> trip_distance,
> fare_amount,
> extra,
> tip_amount,
> tolls_amount,
> total_amount,
> payment_type,
> pickup_ntaname,
> dropoff_ntaname
> FROM s3(
> 'https://datasets-documentation.s3.eu-west-3.amazonaws.com/nyc-taxi/trips_{0..2}.gz',
> 'TabSeparatedWithNames'
> );
> ```
> [tab] GCS
>
> The following command streams three files from a GCS bucket into the `trips` table (the `{0..2}` syntax is a wildcard for the values 0, 1, and 2):
>
> ```sql theme={null}
> INSERT INTO nyc_taxi.trips_small
> SELECT
> trip_id,
> pickup_datetime,
> dropoff_datetime,
> pickup_longitude,
> pickup_latitude,
> dropoff_longitude,
> dropoff_latitude,
> passenger_count,
> trip_distance,
> fare_amount,
> extra,
> tip_amount,
> tolls_amount,
> total_amount,
> payment_type,
> pickup_ntaname,
> dropoff_ntaname
> FROM gcs(
> 'https://storage.googleapis.com/clickhouse-public-datasets/nyc-taxi/trips_{0..2}.gz',
> 'TabSeparatedWithNames'
> );
> ```
## Sample queries
The following queries are executed on the sample described above. You can run the sample queries on the full dataset in [sql.clickhouse.com](https://sql.clickhouse.com/?query=U0VMRUNUIGNvdW50KCkgRlJPTSBueWNfdGF4aS50cmlwcw\&chart=eyJ0eXBlIjoibGluZSIsImNvbmZpZyI6eyJ0aXRsZSI6IlRlbXBlcmF0dXJlIGJ5IGNvdW50cnkgYW5kIHllYXIiLCJ4YXhpcyI6InllYXIiLCJ5YXhpcyI6ImNvdW50KCkiLCJzZXJpZXMiOiJDQVNUKHBhc3Nlbmdlcl9jb3VudCwgJ1N0cmluZycpIn19), modifying the queries below to use the table `nyc_taxi.trips`.
Let's see how many rows were inserted:
```sql theme={null}
SELECT count()
FROM nyc_taxi.trips_small;
```
Each TSV file has about 1M rows, and the three files have 3,000,317 rows. Let's look at a few rows:
```sql theme={null}
SELECT *
FROM nyc_taxi.trips_small
LIMIT 10;
```
Notice there are columns for the pickup and dropoff dates, geo coordinates, fare details, New York neighborhoods, and more.
Let's run a few queries. This query shows us the top 10 neighborhoods that have the most frequent pickups:
```sql theme={null}
SELECT
pickup_ntaname,
count(*) AS count
FROM nyc_taxi.trips_small WHERE pickup_ntaname != ''
GROUP BY pickup_ntaname
ORDER BY count DESC
LIMIT 10;
```
This query shows the average fare based on the number of passengers:
```sql runnable view='chart' chart_config='eyJ0eXBlIjoiYmFyIiwiY29uZmlnIjp7InhheGlzIjoicGFzc2VuZ2VyX2NvdW50IiwieWF4aXMiOiJhdmcodG90YWxfYW1vdW50KSIsInRpdGxlIjoiQXZlcmFnZSBmYXJlIGJ5IHBhc3NlbmdlciBjb3VudCJ9fQ' theme={null}
SELECT
passenger_count,
avg(total_amount)
FROM nyc_taxi.trips_small
WHERE passenger_count < 10
GROUP BY passenger_count;
```
Here's a correlation between the number of passengers and the distance of the trip:
```sql runnable chart_config='eyJ0eXBlIjoiaG9yaXpvbnRhbCBiYXIiLCJjb25maWciOnsieGF4aXMiOiJwYXNzZW5nZXJfY291bnQiLCJ5YXhpcyI6ImRpc3RhbmNlIiwic2VyaWVzIjoiY291bnRyeSIsInRpdGxlIjoiQXZnIGZhcmUgYnkgcGFzc2VuZ2VyIGNvdW50In19' theme={null}
SELECT
passenger_count,
avg(trip_distance) AS distance,
count() AS c
FROM nyc_taxi.trips_small
GROUP BY passenger_count
ORDER BY passenger_count ASC
```
## Download of prepared partitions
> [!NOTE]
>
> The following steps provide information about the original dataset, and a method for loading prepared partitions into a self-managed ClickHouse server environment.
See [https://github.com/toddwschneider/nyc-taxi-data](https://github.com/toddwschneider/nyc-taxi-data) and [http://tech.marksblogg.com/billion-nyc-taxi-rides-redshift.html](http://tech.marksblogg.com/billion-nyc-taxi-rides-redshift.html) for the description of a dataset and instructions for downloading.
Downloading will result in about 227 GB of uncompressed data in CSV files. The download takes about an hour over a 1 Gbit connection (parallel downloading from s3.amazonaws.com recovers at least half of a 1 Gbit channel).
Some of the files might not download fully. Check the file sizes and re-download any that seem doubtful.
```bash theme={null}
$ curl -O https://datasets.clickhouse.com/trips_mergetree/partitions/trips_mergetree.tar
# Validate the checksum
$ md5sum trips_mergetree.tar
# Checksum should be equal to: f3b8d469b41d9a82da064ded7245d12c
$ tar xvf trips_mergetree.tar -C /var/lib/clickhouse # path to ClickHouse data directory
$ # check permissions of unpacked data, fix if required
$ sudo service clickhouse-server restart
$ clickhouse-client --query "select count(*) from datasets.trips_mergetree"
```
> [!NOTE]
>
> If you will run the queries described below, you have to use the full table name, `datasets.trips_mergetree`.
## Results on single server
Q1:
```sql theme={null}
SELECT cab_type, count(*) FROM trips_mergetree GROUP BY cab_type;
```
0.490 seconds.
Q2:
```sql theme={null}
SELECT passenger_count, avg(total_amount) FROM trips_mergetree GROUP BY passenger_count;
```
1.224 seconds.
Q3:
```sql theme={null}
SELECT passenger_count, toYear(pickup_date) AS year, count(*) FROM trips_mergetree GROUP BY passenger_count, year;
```
2.104 seconds.
Q4:
```sql theme={null}
SELECT passenger_count, toYear(pickup_date) AS year, round(trip_distance) AS distance, count(*)
FROM trips_mergetree
GROUP BY passenger_count, year, distance
ORDER BY year, count(*) DESC;
```
3.593 seconds.
The following server was used:
Two Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz, 16 physical cores total, 128 GiB RAM, 8x6 TB HD on hardware RAID-5
Execution time is the best of three runs. But starting from the second run, queries read data from the file system cache. No further caching occurs: the data is read out and processed in each run.
Creating a table on three servers:
On each server:
```sql theme={null}
CREATE TABLE default.trips_mergetree_third ( trip_id UInt32, vendor_id Enum8('1' = 1, '2' = 2, 'CMT' = 3, 'VTS' = 4, 'DDS' = 5, 'B02512' = 10, 'B02598' = 11, 'B02617' = 12, 'B02682' = 13, 'B02764' = 14), pickup_date Date, pickup_datetime DateTime, dropoff_date Date, dropoff_datetime DateTime, store_and_fwd_flag UInt8, rate_code_id UInt8, pickup_longitude Float64, pickup_latitude Float64, dropoff_longitude Float64, dropoff_latitude Float64, passenger_count UInt8, trip_distance Float64, fare_amount Float32, extra Float32, mta_tax Float32, tip_amount Float32, tolls_amount Float32, ehail_fee Float32, improvement_surcharge Float32, total_amount Float32, payment_type_ Enum8('UNK' = 0, 'CSH' = 1, 'CRE' = 2, 'NOC' = 3, 'DIS' = 4), trip_type UInt8, pickup FixedString(25), dropoff FixedString(25), cab_type Enum8('yellow' = 1, 'green' = 2, 'uber' = 3), pickup_nyct2010_gid UInt8, pickup_ctlabel Float32, pickup_borocode UInt8, pickup_boroname Enum8('' = 0, 'Manhattan' = 1, 'Bronx' = 2, 'Brooklyn' = 3, 'Queens' = 4, 'Staten Island' = 5), pickup_ct2010 FixedString(6), pickup_boroct2010 FixedString(7), pickup_cdeligibil Enum8(' ' = 0, 'E' = 1, 'I' = 2), pickup_ntacode FixedString(4), pickup_ntaname Enum16('' = 0, 'Airport' = 1, 'Allerton-Pelham Gardens' = 2, 'Annadale-Huguenot-Prince\'s Bay-Eltingville' = 3, 'Arden Heights' = 4, 'Astoria' = 5, 'Auburndale' = 6, 'Baisley Park' = 7, 'Bath Beach' = 8, 'Battery Park City-Lower Manhattan' = 9, 'Bay Ridge' = 10, 'Bayside-Bayside Hills' = 11, 'Bedford' = 12, 'Bedford Park-Fordham North' = 13, 'Bellerose' = 14, 'Belmont' = 15, 'Bensonhurst East' = 16, 'Bensonhurst West' = 17, 'Borough Park' = 18, 'Breezy Point-Belle Harbor-Rockaway Park-Broad Channel' = 19, 'Briarwood-Jamaica Hills' = 20, 'Brighton Beach' = 21, 'Bronxdale' = 22, 'Brooklyn Heights-Cobble Hill' = 23, 'Brownsville' = 24, 'Bushwick North' = 25, 'Bushwick South' = 26, 'Cambria Heights' = 27, 'Canarsie' = 28, 'Carroll Gardens-Columbia Street-Red Hook' = 29, 'Central Harlem North-Polo Grounds' = 30, 'Central Harlem South' = 31, 'Charleston-Richmond Valley-Tottenville' = 32, 'Chinatown' = 33, 'Claremont-Bathgate' = 34, 'Clinton' = 35, 'Clinton Hill' = 36, 'Co-op City' = 37, 'College Point' = 38, 'Corona' = 39, 'Crotona Park East' = 40, 'Crown Heights North' = 41, 'Crown Heights South' = 42, 'Cypress Hills-City Line' = 43, 'DUMBO-Vinegar Hill-Downtown Brooklyn-Boerum Hill' = 44, 'Douglas Manor-Douglaston-Little Neck' = 45, 'Dyker Heights' = 46, 'East Concourse-Concourse Village' = 47, 'East Elmhurst' = 48, 'East Flatbush-Farragut' = 49, 'East Flushing' = 50, 'East Harlem North' = 51, 'East Harlem South' = 52, 'East New York' = 53, 'East New York (Pennsylvania Ave)' = 54, 'East Tremont' = 55, 'East Village' = 56, 'East Williamsburg' = 57, 'Eastchester-Edenwald-Baychester' = 58, 'Elmhurst' = 59, 'Elmhurst-Maspeth' = 60, 'Erasmus' = 61, 'Far Rockaway-Bayswater' = 62, 'Flatbush' = 63, 'Flatlands' = 64, 'Flushing' = 65, 'Fordham South' = 66, 'Forest Hills' = 67, 'Fort Greene' = 68, 'Fresh Meadows-Utopia' = 69, 'Ft. Totten-Bay Terrace-Clearview' = 70, 'Georgetown-Marine Park-Bergen Beach-Mill Basin' = 71, 'Glen Oaks-Floral Park-New Hyde Park' = 72, 'Glendale' = 73, 'Gramercy' = 74, 'Grasmere-Arrochar-Ft. Wadsworth' = 75, 'Gravesend' = 76, 'Great Kills' = 77, 'Greenpoint' = 78, 'Grymes Hill-Clifton-Fox Hills' = 79, 'Hamilton Heights' = 80, 'Hammels-Arverne-Edgemere' = 81, 'Highbridge' = 82, 'Hollis' = 83, 'Homecrest' = 84, 'Hudson Yards-Chelsea-Flatiron-Union Square' = 85, 'Hunters Point-Sunnyside-West Maspeth' = 86, 'Hunts Point' = 87, 'Jackson Heights' = 88, 'Jamaica' = 89, 'Jamaica Estates-Holliswood' = 90, 'Kensington-Ocean Parkway' = 91, 'Kew Gardens' = 92, 'Kew Gardens Hills' = 93, 'Kingsbridge Heights' = 94, 'Laurelton' = 95, 'Lenox Hill-Roosevelt Island' = 96, 'Lincoln Square' = 97, 'Lindenwood-Howard Beach' = 98, 'Longwood' = 99, 'Lower East Side' = 100, 'Madison' = 101, 'Manhattanville' = 102, 'Marble Hill-Inwood' = 103, 'Mariner\'s Harbor-Arlington-Port Ivory-Graniteville' = 104, 'Maspeth' = 105, 'Melrose South-Mott Haven North' = 106, 'Middle Village' = 107, 'Midtown-Midtown South' = 108, 'Midwood' = 109, 'Morningside Heights' = 110, 'Morrisania-Melrose' = 111, 'Mott Haven-Port Morris' = 112, 'Mount Hope' = 113, 'Murray Hill' = 114, 'Murray Hill-Kips Bay' = 115, 'New Brighton-Silver Lake' = 116, 'New Dorp-Midland Beach' = 117, 'New Springville-Bloomfield-Travis' = 118, 'North Corona' = 119, 'North Riverdale-Fieldston-Riverdale' = 120, 'North Side-South Side' = 121, 'Norwood' = 122, 'Oakland Gardens' = 123, 'Oakwood-Oakwood Beach' = 124, 'Ocean Hill' = 125, 'Ocean Parkway South' = 126, 'Old Astoria' = 127, 'Old Town-Dongan Hills-South Beach' = 128, 'Ozone Park' = 129, 'Park Slope-Gowanus' = 130, 'Parkchester' = 131, 'Pelham Bay-Country Club-City Island' = 132, 'Pelham Parkway' = 133, 'Pomonok-Flushing Heights-Hillcrest' = 134, 'Port Richmond' = 135, 'Prospect Heights' = 136, 'Prospect Lefferts Gardens-Wingate' = 137, 'Queens Village' = 138, 'Queensboro Hill' = 139, 'Queensbridge-Ravenswood-Long Island City' = 140, 'Rego Park' = 141, 'Richmond Hill' = 142, 'Ridgewood' = 143, 'Rikers Island' = 144, 'Rosedale' = 145, 'Rossville-Woodrow' = 146, 'Rugby-Remsen Village' = 147, 'Schuylerville-Throgs Neck-Edgewater Park' = 148, 'Seagate-Coney Island' = 149, 'Sheepshead Bay-Gerritsen Beach-Manhattan Beach' = 150, 'SoHo-TriBeCa-Civic Center-Little Italy' = 151, 'Soundview-Bruckner' = 152, 'Soundview-Castle Hill-Clason Point-Harding Park' = 153, 'South Jamaica' = 154, 'South Ozone Park' = 155, 'Springfield Gardens North' = 156, 'Springfield Gardens South-Brookville' = 157, 'Spuyten Duyvil-Kingsbridge' = 158, 'St. Albans' = 159, 'Stapleton-Rosebank' = 160, 'Starrett City' = 161, 'Steinway' = 162, 'Stuyvesant Heights' = 163, 'Stuyvesant Town-Cooper Village' = 164, 'Sunset Park East' = 165, 'Sunset Park West' = 166, 'Todt Hill-Emerson Hill-Heartland Village-Lighthouse Hill' = 167, 'Turtle Bay-East Midtown' = 168, 'University Heights-Morris Heights' = 169, 'Upper East Side-Carnegie Hill' = 170, 'Upper West Side' = 171, 'Van Cortlandt Village' = 172, 'Van Nest-Morris Park-Westchester Square' = 173, 'Washington Heights North' = 174, 'Washington Heights South' = 175, 'West Brighton' = 176, 'West Concourse' = 177, 'West Farms-Bronx River' = 178, 'West New Brighton-New Brighton-St. George' = 179, 'West Village' = 180, 'Westchester-Unionport' = 181, 'Westerleigh' = 182, 'Whitestone' = 183, 'Williamsbridge-Olinville' = 184, 'Williamsburg' = 185, 'Windsor Terrace' = 186, 'Woodhaven' = 187, 'Woodlawn-Wakefield' = 188, 'Woodside' = 189, 'Yorkville' = 190, 'park-cemetery-etc-Bronx' = 191, 'park-cemetery-etc-Brooklyn' = 192, 'park-cemetery-etc-Manhattan' = 193, 'park-cemetery-etc-Queens' = 194, 'park-cemetery-etc-Staten Island' = 195), pickup_puma UInt16, dropoff_nyct2010_gid UInt8, dropoff_ctlabel Float32, dropoff_borocode UInt8, dropoff_boroname Enum8('' = 0, 'Manhattan' = 1, 'Bronx' = 2, 'Brooklyn' = 3, 'Queens' = 4, 'Staten Island' = 5), dropoff_ct2010 FixedString(6), dropoff_boroct2010 FixedString(7), dropoff_cdeligibil Enum8(' ' = 0, 'E' = 1, 'I' = 2), dropoff_ntacode FixedString(4), dropoff_ntaname Enum16('' = 0, 'Airport' = 1, 'Allerton-Pelham Gardens' = 2, 'Annadale-Huguenot-Prince\'s Bay-Eltingville' = 3, 'Arden Heights' = 4, 'Astoria' = 5, 'Auburndale' = 6, 'Baisley Park' = 7, 'Bath Beach' = 8, 'Battery Park City-Lower Manhattan' = 9, 'Bay Ridge' = 10, 'Bayside-Bayside Hills' = 11, 'Bedford' = 12, 'Bedford Park-Fordham North' = 13, 'Bellerose' = 14, 'Belmont' = 15, 'Bensonhurst East' = 16, 'Bensonhurst West' = 17, 'Borough Park' = 18, 'Breezy Point-Belle Harbor-Rockaway Park-Broad Channel' = 19, 'Briarwood-Jamaica Hills' = 20, 'Brighton Beach' = 21, 'Bronxdale' = 22, 'Brooklyn Heights-Cobble Hill' = 23, 'Brownsville' = 24, 'Bushwick North' = 25, 'Bushwick South' = 26, 'Cambria Heights' = 27, 'Canarsie' = 28, 'Carroll Gardens-Columbia Street-Red Hook' = 29, 'Central Harlem North-Polo Grounds' = 30, 'Central Harlem South' = 31, 'Charleston-Richmond Valley-Tottenville' = 32, 'Chinatown' = 33, 'Claremont-Bathgate' = 34, 'Clinton' = 35, 'Clinton Hill' = 36, 'Co-op City' = 37, 'College Point' = 38, 'Corona' = 39, 'Crotona Park East' = 40, 'Crown Heights North' = 41, 'Crown Heights South' = 42, 'Cypress Hills-City Line' = 43, 'DUMBO-Vinegar Hill-Downtown Brooklyn-Boerum Hill' = 44, 'Douglas Manor-Douglaston-Little Neck' = 45, 'Dyker Heights' = 46, 'East Concourse-Concourse Village' = 47, 'East Elmhurst' = 48, 'East Flatbush-Farragut' = 49, 'East Flushing' = 50, 'East Harlem North' = 51, 'East Harlem South' = 52, 'East New York' = 53, 'East New York (Pennsylvania Ave)' = 54, 'East Tremont' = 55, 'East Village' = 56, 'East Williamsburg' = 57, 'Eastchester-Edenwald-Baychester' = 58, 'Elmhurst' = 59, 'Elmhurst-Maspeth' = 60, 'Erasmus' = 61, 'Far Rockaway-Bayswater' = 62, 'Flatbush' = 63, 'Flatlands' = 64, 'Flushing' = 65, 'Fordham South' = 66, 'Forest Hills' = 67, 'Fort Greene' = 68, 'Fresh Meadows-Utopia' = 69, 'Ft. Totten-Bay Terrace-Clearview' = 70, 'Georgetown-Marine Park-Bergen Beach-Mill Basin' = 71, 'Glen Oaks-Floral Park-New Hyde Park' = 72, 'Glendale' = 73, 'Gramercy' = 74, 'Grasmere-Arrochar-Ft. Wadsworth' = 75, 'Gravesend' = 76, 'Great Kills' = 77, 'Greenpoint' = 78, 'Grymes Hill-Clifton-Fox Hills' = 79, 'Hamilton Heights' = 80, 'Hammels-Arverne-Edgemere' = 81, 'Highbridge' = 82, 'Hollis' = 83, 'Homecrest' = 84, 'Hudson Yards-Chelsea-Flatiron-Union Square' = 85, 'Hunters Point-Sunnyside-West Maspeth' = 86, 'Hunts Point' = 87, 'Jackson Heights' = 88, 'Jamaica' = 89, 'Jamaica Estates-Holliswood' = 90, 'Kensington-Ocean Parkway' = 91, 'Kew Gardens' = 92, 'Kew Gardens Hills' = 93, 'Kingsbridge Heights' = 94, 'Laurelton' = 95, 'Lenox Hill-Roosevelt Island' = 96, 'Lincoln Square' = 97, 'Lindenwood-Howard Beach' = 98, 'Longwood' = 99, 'Lower East Side' = 100, 'Madison' = 101, 'Manhattanville' = 102, 'Marble Hill-Inwood' = 103, 'Mariner\'s Harbor-Arlington-Port Ivory-Graniteville' = 104, 'Maspeth' = 105, 'Melrose South-Mott Haven North' = 106, 'Middle Village' = 107, 'Midtown-Midtown South' = 108, 'Midwood' = 109, 'Morningside Heights' = 110, 'Morrisania-Melrose' = 111, 'Mott Haven-Port Morris' = 112, 'Mount Hope' = 113, 'Murray Hill' = 114, 'Murray Hill-Kips Bay' = 115, 'New Brighton-Silver Lake' = 116, 'New Dorp-Midland Beach' = 117, 'New Springville-Bloomfield-Travis' = 118, 'North Corona' = 119, 'North Riverdale-Fieldston-Riverdale' = 120, 'North Side-South Side' = 121, 'Norwood' = 122, 'Oakland Gardens' = 123, 'Oakwood-Oakwood Beach' = 124, 'Ocean Hill' = 125, 'Ocean Parkway South' = 126, 'Old Astoria' = 127, 'Old Town-Dongan Hills-South Beach' = 128, 'Ozone Park' = 129, 'Park Slope-Gowanus' = 130, 'Parkchester' = 131, 'Pelham Bay-Country Club-City Island' = 132, 'Pelham Parkway' = 133, 'Pomonok-Flushing Heights-Hillcrest' = 134, 'Port Richmond' = 135, 'Prospect Heights' = 136, 'Prospect Lefferts Gardens-Wingate' = 137, 'Queens Village' = 138, 'Queensboro Hill' = 139, 'Queensbridge-Ravenswood-Long Island City' = 140, 'Rego Park' = 141, 'Richmond Hill' = 142, 'Ridgewood' = 143, 'Rikers Island' = 144, 'Rosedale' = 145, 'Rossville-Woodrow' = 146, 'Rugby-Remsen Village' = 147, 'Schuylerville-Throgs Neck-Edgewater Park' = 148, 'Seagate-Coney Island' = 149, 'Sheepshead Bay-Gerritsen Beach-Manhattan Beach' = 150, 'SoHo-TriBeCa-Civic Center-Little Italy' = 151, 'Soundview-Bruckner' = 152, 'Soundview-Castle Hill-Clason Point-Harding Park' = 153, 'South Jamaica' = 154, 'South Ozone Park' = 155, 'Springfield Gardens North' = 156, 'Springfield Gardens South-Brookville' = 157, 'Spuyten Duyvil-Kingsbridge' = 158, 'St. Albans' = 159, 'Stapleton-Rosebank' = 160, 'Starrett City' = 161, 'Steinway' = 162, 'Stuyvesant Heights' = 163, 'Stuyvesant Town-Cooper Village' = 164, 'Sunset Park East' = 165, 'Sunset Park West' = 166, 'Todt Hill-Emerson Hill-Heartland Village-Lighthouse Hill' = 167, 'Turtle Bay-East Midtown' = 168, 'University Heights-Morris Heights' = 169, 'Upper East Side-Carnegie Hill' = 170, 'Upper West Side' = 171, 'Van Cortlandt Village' = 172, 'Van Nest-Morris Park-Westchester Square' = 173, 'Washington Heights North' = 174, 'Washington Heights South' = 175, 'West Brighton' = 176, 'West Concourse' = 177, 'West Farms-Bronx River' = 178, 'West New Brighton-New Brighton-St. George' = 179, 'West Village' = 180, 'Westchester-Unionport' = 181, 'Westerleigh' = 182, 'Whitestone' = 183, 'Williamsbridge-Olinville' = 184, 'Williamsburg' = 185, 'Windsor Terrace' = 186, 'Woodhaven' = 187, 'Woodlawn-Wakefield' = 188, 'Woodside' = 189, 'Yorkville' = 190, 'park-cemetery-etc-Bronx' = 191, 'park-cemetery-etc-Brooklyn' = 192, 'park-cemetery-etc-Manhattan' = 193, 'park-cemetery-etc-Queens' = 194, 'park-cemetery-etc-Staten Island' = 195), dropoff_puma UInt16) ENGINE = MergeTree(pickup_date, pickup_datetime, 8192);
```
On the source server:
```sql theme={null}
CREATE TABLE trips_mergetree_x3 AS trips_mergetree_third ENGINE = Distributed(perftest, default, trips_mergetree_third, rand());
```
The following query redistributes data:
```sql theme={null}
INSERT INTO trips_mergetree_x3 SELECT * FROM trips_mergetree;
```
This takes 2454 seconds.
On three servers:
Q1: 0.212 seconds.
Q2: 0.438 seconds.
Q3: 0.733 seconds.
Q4: 1.241 seconds.
No surprises here, since the queries are scaled linearly.
We also have the results from a cluster of 140 servers:
Q1: 0.028 sec.
Q2: 0.043 sec.
Q3: 0.051 sec.
Q4: 0.072 sec.
In this case, the query processing time is determined above all by network latency.
We ran queries using a client located in a different datacenter than where the cluster was located, which added about 20 ms of latency.
## Summary
| servers | Q1 | Q2 | Q3 | Q4 |
| - | - | - | - | - |
| 1, E5-2650v2 | 0.490 | 1.224 | 2.104 | 3.593 |
| 3, E5-2650v2 | 0.212 | 0.438 | 0.733 | 1.241 |
| 1, AWS c5n.4xlarge | 0.249 | 1.279 | 1.738 | 3.527 |
| 1, AWS c5n.9xlarge | 0.130 | 0.584 | 0.777 | 1.811 |
| 3, AWS c5n.9xlarge | 0.057 | 0.231 | 0.285 | 0.641 |
| 140, E5-2650v2 | 0.028 | 0.043 | 0.051 | 0.072 |
# NYPD complaint data
Source: https://clickhouse.com/docs/get-started/sample-datasets/nypd-complaint-data
Ingest and query Tab Separated Value data in 5 steps
Tab separated value, or TSV, files are common and may include field headings as the first line of the file. ClickHouse can ingest TSVs, and also can query TSVs without ingesting the files. This guide covers both of these cases. If you need to query or ingest CSV files, the same techniques work, simply substitute `TSV` with `CSV` in your format arguments.
While working through this guide you will:
* **Investigate**: Query the structure and content of the TSV file.
* **Determine the target ClickHouse schema**: Choose proper data types and map the existing data to those types.
* **Create a ClickHouse table**.
* **Preprocess and stream** the data to ClickHouse.
* **Run some queries** against ClickHouse.
The dataset used in this guide comes from the NYC Open Data team, and contains data about "all valid felony, misdemeanor, and violation crimes reported to the New York City Police Department (NYPD)". At the time of writing, the data file is 166MB, but it is updated regularly.
**Source**: [data.cityofnewyork.us](https://data.cityofnewyork.us/Public-Safety/NYPD-Complaint-Data-Current-Year-To-Date-/5uac-w243)
**Terms of use**: [https://www1.nyc.gov/home/terms-of-use.page](https://www1.nyc.gov/home/terms-of-use.page)
## Prerequisites
* Download the dataset by visiting the [NYPD Complaint Data Current (Year To Date)](https://data.cityofnewyork.us/Public-Safety/NYPD-Complaint-Data-Current-Year-To-Date-/5uac-w243) page, clicking the Export button, and choosing **TSV for Excel**.
* Install [ClickHouse server and client](/docs/get-started/setup/install)
### A note about the commands described in this guide
There are two types of commands in this guide:
* Some of the commands are querying the TSV files, these are run at the command prompt.
* The rest of the commands are querying ClickHouse, and these are run in the `clickhouse-client` or Play UI.
> [!NOTE]
>
> The examples in this guide assume that you have saved the TSV file to `${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv`, please adjust the commands if needed.
## Familiarize yourself with the TSV file
Before starting to work with the ClickHouse database familiarize yourself with the data.
### Look at the fields in the source TSV file
This is an example of a command to query a TSV file, but don't run it yet.
```sh title="Query" theme={null}
clickhouse-local --query \
"describe file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')"
```
Sample response
```response theme={null}
CMPLNT_NUM Nullable(Float64)
ADDR_PCT_CD Nullable(Float64)
BORO_NM Nullable(String)
CMPLNT_FR_DT Nullable(String)
CMPLNT_FR_TM Nullable(String)
```
> [!TIP]
>
> Most of the time the above command will let you know which fields in the input data are numeric, and which are strings, and which are tuples. This isn't always the case. Because ClickHouse is routineley used with datasets containing billions of records there is a default number (100) of rows examined to [infer the schema](/docs/guides/clickhouse/data-formats/json/inference) in order to avoid parsing billions of rows to infer the schema. The response below may not match what you see, as the dataset is updated several times each year. Looking at the Data Dictionary you can see that CMPLNT\_NUM is specified as text, and not numeric. By overriding the default of 100 rows for inference with the setting `SETTINGS input_format_max_rows_to_read_for_schema_inference=2000`
> you can get a better idea of the content.
>
> Note: as of version 22.5 the default is now 25,000 rows for inferring the schema, so only change the setting if you're on an older version or if you need more than 25,000 rows to be sampled.
Run this command at your command prompt. You will be using `clickhouse-local` to query the data in the TSV file you downloaded.
```sh title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"describe file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')"
```
```response title="Response" theme={null}
CMPLNT_NUM Nullable(String)
ADDR_PCT_CD Nullable(Float64)
BORO_NM Nullable(String)
CMPLNT_FR_DT Nullable(String)
CMPLNT_FR_TM Nullable(String)
CMPLNT_TO_DT Nullable(String)
CMPLNT_TO_TM Nullable(String)
CRM_ATPT_CPTD_CD Nullable(String)
HADEVELOPT Nullable(String)
HOUSING_PSA Nullable(Float64)
JURISDICTION_CODE Nullable(Float64)
JURIS_DESC Nullable(String)
KY_CD Nullable(Float64)
LAW_CAT_CD Nullable(String)
LOC_OF_OCCUR_DESC Nullable(String)
OFNS_DESC Nullable(String)
PARKS_NM Nullable(String)
PATROL_BORO Nullable(String)
PD_CD Nullable(Float64)
PD_DESC Nullable(String)
PREM_TYP_DESC Nullable(String)
RPT_DT Nullable(String)
STATION_NAME Nullable(String)
SUSP_AGE_GROUP Nullable(String)
SUSP_RACE Nullable(String)
SUSP_SEX Nullable(String)
TRANSIT_DISTRICT Nullable(Float64)
VIC_AGE_GROUP Nullable(String)
VIC_RACE Nullable(String)
VIC_SEX Nullable(String)
X_COORD_CD Nullable(Float64)
Y_COORD_CD Nullable(Float64)
Latitude Nullable(Float64)
Longitude Nullable(Float64)
Lat_Lon Tuple(Nullable(Float64), Nullable(Float64))
New Georeferenced Column Nullable(String)
```
At this point you should check that the columns in the TSV file match the names and types specified in the **Columns in this Dataset** section of the [dataset web page](https://data.cityofnewyork.us/Public-Safety/NYPD-Complaint-Data-Current-Year-To-Date-/5uac-w243). The data types aren't very specific, all numeric fields are set to `Nullable(Float64)`, and all other fields are `Nullable(String)`. When you create a ClickHouse table to store the data you can specify more appropriate and performant types.
### Determine the proper schema
In order to figure out what types should be used for the fields it is necessary to know what the data looks like. For example, the field `JURISDICTION_CODE` is a numeric: should it be a `UInt8`, or an `Enum`, or is `Float64` appropriate?
```sql title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"select JURISDICTION_CODE, count() FROM
file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
GROUP BY JURISDICTION_CODE
ORDER BY JURISDICTION_CODE
FORMAT PrettyCompact"
```
```response title="Response" theme={null}
┌─JURISDICTION_CODE─┬─count()─┐
│ 0 │ 188875 │
│ 1 │ 4799 │
│ 2 │ 13833 │
│ 3 │ 656 │
│ 4 │ 51 │
│ 6 │ 5 │
│ 7 │ 2 │
│ 9 │ 13 │
│ 11 │ 14 │
│ 12 │ 5 │
│ 13 │ 2 │
│ 14 │ 70 │
│ 15 │ 20 │
│ 72 │ 159 │
│ 87 │ 9 │
│ 88 │ 75 │
│ 97 │ 405 │
└───────────────────┴─────────┘
```
The query response shows that the `JURISDICTION_CODE` fits well in a `UInt8`.
Similarly, look at some of the `String` fields and see if they're well suited to being `DateTime` or [`LowCardinality(String)`](/docs/reference/data-types/lowcardinality) fields.
For example, the field `PARKS_NM` is described as "Name of NYC park, playground or greenspace of occurrence, if applicable (state parks aren't included)". The names of parks in New York City may be a good candidate for a `LowCardinality(String)`:
```sh title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"select count(distinct PARKS_NM) FROM
file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
FORMAT PrettyCompact"
```
```response title="Response" theme={null}
┌─uniqExact(PARKS_NM)─┐
│ 319 │
└─────────────────────┘
```
Have a look at some of the park names:
```sql title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"select distinct PARKS_NM FROM
file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
LIMIT 10
FORMAT PrettyCompact"
```
```response title="Response" theme={null}
┌─PARKS_NM───────────────────┐
│ (null) │
│ ASSER LEVY PARK │
│ JAMES J WALKER PARK │
│ BELT PARKWAY/SHORE PARKWAY │
│ PROSPECT PARK │
│ MONTEFIORE SQUARE │
│ SUTTON PLACE PARK │
│ JOYCE KILMER PARK │
│ ALLEY ATHLETIC PLAYGROUND │
│ ASTORIA PARK │
└────────────────────────────┘
```
The dataset in use at the time of writing has only a few hundred distinct parks and playgrounds in the `PARK_NM` column. This is a small number based on the [LowCardinality](/docs/reference/data-types/lowcardinality#description) recommendation to stay below 10,000 distinct strings in a `LowCardinality(String)` field.
### DateTime fields
Based on the **Columns in this Dataset** section of the [dataset web page](https://data.cityofnewyork.us/Public-Safety/NYPD-Complaint-Data-Current-Year-To-Date-/5uac-w243) there are date and time fields for the start and end of the reported event. Looking at the min and max of the `CMPLNT_FR_DT` and `CMPLT_TO_DT` gives an idea of whether or not the fields are always populated:
```sh title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"select min(CMPLNT_FR_DT), max(CMPLNT_FR_DT) FROM
file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
FORMAT PrettyCompact"
```
```response title="Response" theme={null}
┌─min(CMPLNT_FR_DT)─┬─max(CMPLNT_FR_DT)─┐
│ 01/01/1973 │ 12/31/2021 │
└───────────────────┴───────────────────┘
```
```sh title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"select min(CMPLNT_TO_DT), max(CMPLNT_TO_DT) FROM
file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
FORMAT PrettyCompact"
```
```response title="Response" theme={null}
┌─min(CMPLNT_TO_DT)─┬─max(CMPLNT_TO_DT)─┐
│ │ 12/31/2021 │
└───────────────────┴───────────────────┘
```
```sh title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"select min(CMPLNT_FR_TM), max(CMPLNT_FR_TM) FROM
file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
FORMAT PrettyCompact"
```
```response title="Response" theme={null}
┌─min(CMPLNT_FR_TM)─┬─max(CMPLNT_FR_TM)─┐
│ 00:00:00 │ 23:59:00 │
└───────────────────┴───────────────────┘
```
```sh title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"select min(CMPLNT_TO_TM), max(CMPLNT_TO_TM) FROM
file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
FORMAT PrettyCompact"
```
```response title="Response" theme={null}
┌─min(CMPLNT_TO_TM)─┬─max(CMPLNT_TO_TM)─┐
│ (null) │ 23:59:00 │
└───────────────────┴───────────────────┘
```
## Make a plan
Based on the above investigation:
* `JURISDICTION_CODE` should be cast as `UInt8`.
* `PARKS_NM` should be cast to `LowCardinality(String)`
* `CMPLNT_FR_DT` and `CMPLNT_FR_TM` are always populated (possibly with a default time of `00:00:00`)
* `CMPLNT_TO_DT` and `CMPLNT_TO_TM` may be empty
* Dates and times are stored in separate fields in the source
* Dates are `mm/dd/yyyy` format
* Times are `hh:mm:ss` format
* Dates and times can be concatenated into DateTime types
* There are some dates before January 1st 1970, which means we need a 64 bit DateTime
> [!NOTE]
>
> There are many more changes to be made to the types, they all can be determined by following the same investigation steps. Look at the number of distinct strings in a field, the min and max of the numerics, and make your decisions. The table schema that is given later in the guide has many low cardinality strings and unsigned integer fields and very few floating point numerics.
## Concatenate the date and time fields
To concatenate the date and time fields `CMPLNT_FR_DT` and `CMPLNT_FR_TM` into a single `String` that can be cast to a `DateTime`, select the two fields joined by the concatenation operator: `CMPLNT_FR_DT || ' ' || CMPLNT_FR_TM`. The `CMPLNT_TO_DT` and `CMPLNT_TO_TM` fields are handled similarly.
```sh title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"select CMPLNT_FR_DT || ' ' || CMPLNT_FR_TM AS complaint_begin FROM
file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
LIMIT 10
FORMAT PrettyCompact"
```
```response title="Response" theme={null}
┌─complaint_begin─────┐
│ 07/29/2010 00:01:00 │
│ 12/01/2011 12:00:00 │
│ 04/01/2017 15:00:00 │
│ 03/26/2018 17:20:00 │
│ 01/01/2019 00:00:00 │
│ 06/14/2019 00:00:00 │
│ 11/29/2021 20:00:00 │
│ 12/04/2021 00:35:00 │
│ 12/05/2021 12:50:00 │
│ 12/07/2021 20:30:00 │
└─────────────────────┘
```
## Convert the date and time String to a DateTime64 type
Earlier in the guide we discovered that there are dates in the TSV file before January 1st 1970, which means that we need a 64 bit DateTime type for the dates. The dates also need to be converted from `MM/DD/YYYY` to `YYYY/MM/DD` format. Both of these can be done with [`parseDateTime64BestEffort()`](/docs/reference/functions/regular-functions/type-conversion-functions#parseDateTime64BestEffort).
```sh title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"WITH (CMPLNT_FR_DT || ' ' || CMPLNT_FR_TM) AS CMPLNT_START,
(CMPLNT_TO_DT || ' ' || CMPLNT_TO_TM) AS CMPLNT_END
select parseDateTime64BestEffort(CMPLNT_START) AS complaint_begin,
parseDateTime64BestEffortOrNull(CMPLNT_END) AS complaint_end
FROM file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
ORDER BY complaint_begin ASC
LIMIT 25
FORMAT PrettyCompact"
```
Lines 2 and 3 above contain the concatenation from the previous step, and lines 4 and 5 above parse the strings into `DateTime64`. As the complaint end time isn't guaranteed to exist `parseDateTime64BestEffortOrNull` is used.
```response title="Response" theme={null}
┌─────────complaint_begin─┬───────────complaint_end─┐
│ 1925-01-01 10:00:00.000 │ 2021-02-12 09:30:00.000 │
│ 1925-01-01 11:37:00.000 │ 2022-01-16 11:49:00.000 │
│ 1925-01-01 15:00:00.000 │ 2021-12-31 00:00:00.000 │
│ 1925-01-01 15:00:00.000 │ 2022-02-02 22:00:00.000 │
│ 1925-01-01 19:00:00.000 │ 2022-04-14 05:00:00.000 │
│ 1955-09-01 19:55:00.000 │ 2022-08-01 00:45:00.000 │
│ 1972-03-17 11:40:00.000 │ 2022-03-17 11:43:00.000 │
│ 1972-05-23 22:00:00.000 │ 2022-05-24 09:00:00.000 │
│ 1972-05-30 23:37:00.000 │ 2022-05-30 23:50:00.000 │
│ 1972-07-04 02:17:00.000 │ ᴺᵁᴸᴸ │
│ 1973-01-01 00:00:00.000 │ ᴺᵁᴸᴸ │
│ 1975-01-01 00:00:00.000 │ ᴺᵁᴸᴸ │
│ 1976-11-05 00:01:00.000 │ 1988-10-05 23:59:00.000 │
│ 1977-01-01 00:00:00.000 │ 1977-01-01 23:59:00.000 │
│ 1977-12-20 00:01:00.000 │ ᴺᵁᴸᴸ │
│ 1981-01-01 00:01:00.000 │ ᴺᵁᴸᴸ │
│ 1981-08-14 00:00:00.000 │ 1987-08-13 23:59:00.000 │
│ 1983-01-07 00:00:00.000 │ 1990-01-06 00:00:00.000 │
│ 1984-01-01 00:01:00.000 │ 1984-12-31 23:59:00.000 │
│ 1985-01-01 12:00:00.000 │ 1987-12-31 15:00:00.000 │
│ 1985-01-11 09:00:00.000 │ 1985-12-31 12:00:00.000 │
│ 1986-03-16 00:05:00.000 │ 2022-03-16 00:45:00.000 │
│ 1987-01-07 00:00:00.000 │ 1987-01-09 00:00:00.000 │
│ 1988-04-03 18:30:00.000 │ 2022-08-03 09:45:00.000 │
│ 1988-07-29 12:00:00.000 │ 1990-07-27 22:00:00.000 │
└─────────────────────────┴─────────────────────────┘
```
> [!NOTE]
>
> The dates shown as `1925` above are from errors in the data. There are several records in the original data with dates in the years `1019` - `1022` that should be `2019` - `2022`. They're being stored as Jan 1st 1925 as that is the earliest date with a 64 bit DateTime.
## Create a table
The decisions made above on the data types used for the columns are reflected in the table schema
below. We also need to decide on the `ORDER BY` and `PRIMARY KEY` used for the table. At least one
of `ORDER BY` or `PRIMARY KEY` must be specified. Here are some guidelines on deciding on the
columns to includes in `ORDER BY`, and more information is in the *Next Steps* section at the end
of this document.
### `ORDER BY` and `PRIMARY KEY` clauses
* The `ORDER BY` tuple should include fields that are used in query filters
* To maximize compression on disk the `ORDER BY` tuple should be ordered by ascending cardinality
* If it exists, the `PRIMARY KEY` tuple must be a subset of the `ORDER BY` tuple
* If only `ORDER BY` is specified, then the same tuple will be used as `PRIMARY KEY`
* The primary key index is created using the `PRIMARY KEY` tuple if specified, otherwise the `ORDER BY` tuple
* The `PRIMARY KEY` index is kept in main memory
Looking at the dataset and the questions that might be answered by querying it we might
decide that we would look at the types of crimes reported over time in the five boroughs of
New York City. These fields might be then included in the `ORDER BY`:
| Column | Description (from the data dictionary) |
| - | - |
| OFNS\_DESC | Description of offense corresponding with key code |
| RPT\_DT | Date event was reported to police |
| BORO\_NM | The name of the borough in which the incident occurred |
Querying the TSV file for the cardinality of the three candidate columns:
```bash title="Query" theme={null}
clickhouse-local --input_format_max_rows_to_read_for_schema_inference=2000 \
--query \
"select formatReadableQuantity(uniq(OFNS_DESC)) as cardinality_OFNS_DESC,
formatReadableQuantity(uniq(RPT_DT)) as cardinality_RPT_DT,
formatReadableQuantity(uniq(BORO_NM)) as cardinality_BORO_NM
FROM
file('${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv', 'TSVWithNames')
FORMAT PrettyCompact"
```
```response title="Response" theme={null}
┌─cardinality_OFNS_DESC─┬─cardinality_RPT_DT─┬─cardinality_BORO_NM─┐
│ 60.00 │ 306.00 │ 6.00 │
└───────────────────────┴────────────────────┴─────────────────────┘
```
Ordering by cardinality, the `ORDER BY` becomes:
```sql theme={null}
ORDER BY ( BORO_NM, OFNS_DESC, RPT_DT )
```
> [!NOTE]
>
> The table below will use more easily read column names, the above names will be mapped to
>
> ```sql theme={null}
> ORDER BY ( borough, offense_description, date_reported )
> ```
Putting together the changes to data types and the `ORDER BY` tuple gives this table structure:
```sql theme={null}
CREATE TABLE NYPD_Complaint (
complaint_number String,
precinct UInt8,
borough LowCardinality(String),
complaint_begin DateTime64(0,'America/New_York'),
complaint_end DateTime64(0,'America/New_York'),
was_crime_completed String,
housing_authority String,
housing_level_code UInt32,
jurisdiction_code UInt8,
jurisdiction LowCardinality(String),
offense_code UInt8,
offense_level LowCardinality(String),
location_descriptor LowCardinality(String),
offense_description LowCardinality(String),
park_name LowCardinality(String),
patrol_borough LowCardinality(String),
PD_CD UInt16,
PD_DESC String,
location_type LowCardinality(String),
date_reported Date,
transit_station LowCardinality(String),
suspect_age_group LowCardinality(String),
suspect_race LowCardinality(String),
suspect_sex LowCardinality(String),
transit_district UInt8,
victim_age_group LowCardinality(String),
victim_race LowCardinality(String),
victim_sex LowCardinality(String),
NY_x_coordinate UInt32,
NY_y_coordinate UInt32,
Latitude Float64,
Longitude Float64
) ENGINE = MergeTree
ORDER BY ( borough, offense_description, date_reported )
```
### Finding the primary key of a table
The ClickHouse `system` database, specifically `system.table` has all of the information about the table you
just created. This query shows the `ORDER BY` (sorting key), and the `PRIMARY KEY`:
```sql theme={null}
SELECT
partition_key,
sorting_key,
primary_key,
table
FROM system.tables
WHERE table = 'NYPD_Complaint'
FORMAT Vertical
```
Response
```response theme={null}
Query id: 6a5b10bf-9333-4090-b36e-c7f08b1d9e01
Row 1:
──────
partition_key:
sorting_key: borough, offense_description, date_reported
primary_key: borough, offense_description, date_reported
table: NYPD_Complaint
1 row in set. Elapsed: 0.001 sec.
```
## Preprocess and import data
We will use `clickhouse-local` tool for data preprocessing and `clickhouse-client` to upload it.
### `clickhouse-local` arguments used
> [!TIP]
>
> `table='input'` appears in the arguments to clickhouse-local below. clickhouse-local takes the provided input (`cat ${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv`) and inserts the input into a table. By default the table is named `table`. In this guide the name of the table is set to `input` to make the data flow clearer. The final argument to clickhouse-local is a query that selects from the table (`FROM input`) which is then piped to `clickhouse-client` to populate the table `NYPD_Complaint`.
```sql theme={null}
cat ${HOME}/NYPD_Complaint_Data_Current__Year_To_Date_.tsv \
| clickhouse-local --table='input' --input-format='TSVWithNames' \
--input_format_max_rows_to_read_for_schema_inference=2000 \
--query "
WITH (CMPLNT_FR_DT || ' ' || CMPLNT_FR_TM) AS CMPLNT_START,
(CMPLNT_TO_DT || ' ' || CMPLNT_TO_TM) AS CMPLNT_END
SELECT
CMPLNT_NUM AS complaint_number,
ADDR_PCT_CD AS precinct,
BORO_NM AS borough,
parseDateTime64BestEffort(CMPLNT_START) AS complaint_begin,
parseDateTime64BestEffortOrNull(CMPLNT_END) AS complaint_end,
CRM_ATPT_CPTD_CD AS was_crime_completed,
HADEVELOPT AS housing_authority_development,
HOUSING_PSA AS housing_level_code,
JURISDICTION_CODE AS jurisdiction_code,
JURIS_DESC AS jurisdiction,
KY_CD AS offense_code,
LAW_CAT_CD AS offense_level,
LOC_OF_OCCUR_DESC AS location_descriptor,
OFNS_DESC AS offense_description,
PARKS_NM AS park_name,
PATROL_BORO AS patrol_borough,
PD_CD,
PD_DESC,
PREM_TYP_DESC AS location_type,
toDate(parseDateTimeBestEffort(RPT_DT)) AS date_reported,
STATION_NAME AS transit_station,
SUSP_AGE_GROUP AS suspect_age_group,
SUSP_RACE AS suspect_race,
SUSP_SEX AS suspect_sex,
TRANSIT_DISTRICT AS transit_district,
VIC_AGE_GROUP AS victim_age_group,
VIC_RACE AS victim_race,
VIC_SEX AS victim_sex,
X_COORD_CD AS NY_x_coordinate,
Y_COORD_CD AS NY_y_coordinate,
Latitude,
Longitude
FROM input" \
| clickhouse-client --query='INSERT INTO NYPD_Complaint FORMAT TSV'
```
## Validate the data
> [!NOTE]
>
> The dataset changes once or more per year, your counts may not match what is in this document.
```sql title="Query" theme={null}
SELECT count()
FROM NYPD_Complaint
```
```text title="Response" theme={null}
┌─count()─┐
│ 208993 │
└─────────┘
1 row in set. Elapsed: 0.001 sec.
```
The size of the dataset in ClickHouse is just 12% of the original TSV file, compare the size of the original TSV file with the size of the table:
```sql title="Query" theme={null}
SELECT formatReadableSize(total_bytes)
FROM system.tables
WHERE name = 'NYPD_Complaint'
```
```text title="Response" theme={null}
┌─formatReadableSize(total_bytes)─┐
│ 8.63 MiB │
└─────────────────────────────────┘
```
## Run some queries
### Query 1. Compare the number of complaints by month
```sql title="Query" theme={null}
SELECT
dateName('month', date_reported) AS month,
count() AS complaints,
bar(complaints, 0, 50000, 80)
FROM NYPD_Complaint
GROUP BY month
ORDER BY complaints DESC
```
```response title="Response" theme={null}
Query id: 7fbd4244-b32a-4acf-b1f3-c3aa198e74d9
┌─month─────┬─complaints─┬─bar(count(), 0, 50000, 80)───────────────────────────────┐
│ March │ 34536 │ ███████████████████████████████████████████████████████▎ │
│ May │ 34250 │ ██████████████████████████████████████████████████████▋ │
│ April │ 32541 │ ████████████████████████████████████████████████████ │
│ January │ 30806 │ █████████████████████████████████████████████████▎ │
│ February │ 28118 │ ████████████████████████████████████████████▊ │
│ November │ 7474 │ ███████████▊ │
│ December │ 7223 │ ███████████▌ │
│ October │ 7070 │ ███████████▎ │
│ September │ 6910 │ ███████████ │
│ August │ 6801 │ ██████████▊ │
│ June │ 6779 │ ██████████▋ │
│ July │ 6485 │ ██████████▍ │
└───────────┴────────────┴──────────────────────────────────────────────────────────┘
12 rows in set. Elapsed: 0.006 sec. Processed 208.99 thousand rows, 417.99 KB (37.48 million rows/s., 74.96 MB/s.)
```
### Query 2. Compare total number of complaints by borough
```sql title="Query" theme={null}
SELECT
borough,
count() AS complaints,
bar(complaints, 0, 125000, 60)
FROM NYPD_Complaint
GROUP BY borough
ORDER BY complaints DESC
```
```response title="Response" theme={null}
Query id: 8cdcdfd4-908f-4be0-99e3-265722a2ab8d
┌─borough───────┬─complaints─┬─bar(count(), 0, 125000, 60)──┐
│ BROOKLYN │ 57947 │ ███████████████████████████▋ │
│ MANHATTAN │ 53025 │ █████████████████████████▍ │
│ QUEENS │ 44875 │ █████████████████████▌ │
│ BRONX │ 44260 │ █████████████████████▏ │
│ STATEN ISLAND │ 8503 │ ████ │
│ (null) │ 383 │ ▏ │
└───────────────┴────────────┴──────────────────────────────┘
6 rows in set. Elapsed: 0.008 sec. Processed 208.99 thousand rows, 209.43 KB (27.14 million rows/s., 27.20 MB/s.)
```
## Next steps
[A Practical Introduction to Sparse Primary Indexes in ClickHouse](/docs/guides/clickhouse/data-modelling/sparse-primary-indexes) discusses the differences in ClickHouse indexing compared to traditional relational databases, how ClickHouse builds and uses a sparse primary index, and indexing best practices.
# OnTime
Source: https://clickhouse.com/docs/get-started/sample-datasets/ontime
Dataset containing the on-time performance of airline flights
This dataset contains data from Bureau of Transportation Statistics.
## Creating a table
```sql theme={null}
CREATE TABLE `ontime`
(
`Year` UInt16,
`Quarter` UInt8,
`Month` UInt8,
`DayofMonth` UInt8,
`DayOfWeek` UInt8,
`FlightDate` Date,
`Reporting_Airline` LowCardinality(String),
`DOT_ID_Reporting_Airline` Int32,
`IATA_CODE_Reporting_Airline` LowCardinality(String),
`Tail_Number` LowCardinality(String),
`Flight_Number_Reporting_Airline` LowCardinality(String),
`OriginAirportID` Int32,
`OriginAirportSeqID` Int32,
`OriginCityMarketID` Int32,
`Origin` FixedString(5),
`OriginCityName` LowCardinality(String),
`OriginState` FixedString(2),
`OriginStateFips` FixedString(2),
`OriginStateName` LowCardinality(String),
`OriginWac` Int32,
`DestAirportID` Int32,
`DestAirportSeqID` Int32,
`DestCityMarketID` Int32,
`Dest` FixedString(5),
`DestCityName` LowCardinality(String),
`DestState` FixedString(2),
`DestStateFips` FixedString(2),
`DestStateName` LowCardinality(String),
`DestWac` Int32,
`CRSDepTime` Int32,
`DepTime` Int32,
`DepDelay` Int32,
`DepDelayMinutes` Int32,
`DepDel15` Int32,
`DepartureDelayGroups` LowCardinality(String),
`DepTimeBlk` LowCardinality(String),
`TaxiOut` Int32,
`WheelsOff` LowCardinality(String),
`WheelsOn` LowCardinality(String),
`TaxiIn` Int32,
`CRSArrTime` Int32,
`ArrTime` Int32,
`ArrDelay` Int32,
`ArrDelayMinutes` Int32,
`ArrDel15` Int32,
`ArrivalDelayGroups` LowCardinality(String),
`ArrTimeBlk` LowCardinality(String),
`Cancelled` Int8,
`CancellationCode` FixedString(1),
`Diverted` Int8,
`CRSElapsedTime` Int32,
`ActualElapsedTime` Int32,
`AirTime` Int32,
`Flights` Int32,
`Distance` Int32,
`DistanceGroup` Int8,
`CarrierDelay` Int32,
`WeatherDelay` Int32,
`NASDelay` Int32,
`SecurityDelay` Int32,
`LateAircraftDelay` Int32,
`FirstDepTime` Int16,
`TotalAddGTime` Int16,
`LongestAddGTime` Int16,
`DivAirportLandings` Int8,
`DivReachedDest` Int8,
`DivActualElapsedTime` Int16,
`DivArrDelay` Int16,
`DivDistance` Int16,
`Div1Airport` LowCardinality(String),
`Div1AirportID` Int32,
`Div1AirportSeqID` Int32,
`Div1WheelsOn` Int16,
`Div1TotalGTime` Int16,
`Div1LongestGTime` Int16,
`Div1WheelsOff` Int16,
`Div1TailNum` LowCardinality(String),
`Div2Airport` LowCardinality(String),
`Div2AirportID` Int32,
`Div2AirportSeqID` Int32,
`Div2WheelsOn` Int16,
`Div2TotalGTime` Int16,
`Div2LongestGTime` Int16,
`Div2WheelsOff` Int16,
`Div2TailNum` LowCardinality(String),
`Div3Airport` LowCardinality(String),
`Div3AirportID` Int32,
`Div3AirportSeqID` Int32,
`Div3WheelsOn` Int16,
`Div3TotalGTime` Int16,
`Div3LongestGTime` Int16,
`Div3WheelsOff` Int16,
`Div3TailNum` LowCardinality(String),
`Div4Airport` LowCardinality(String),
`Div4AirportID` Int32,
`Div4AirportSeqID` Int32,
`Div4WheelsOn` Int16,
`Div4TotalGTime` Int16,
`Div4LongestGTime` Int16,
`Div4WheelsOff` Int16,
`Div4TailNum` LowCardinality(String),
`Div5Airport` LowCardinality(String),
`Div5AirportID` Int32,
`Div5AirportSeqID` Int32,
`Div5WheelsOn` Int16,
`Div5TotalGTime` Int16,
`Div5LongestGTime` Int16,
`Div5WheelsOff` Int16,
`Div5TailNum` LowCardinality(String)
) ENGINE = MergeTree
ORDER BY (Year, Quarter, Month, DayofMonth, FlightDate, IATA_CODE_Reporting_Airline);
```
## Import from raw data
Downloading data:
```bash theme={null}
wget --no-check-certificate --continue https://transtats.bts.gov/PREZIP/On_Time_Reporting_Carrier_On_Time_Performance_1987_present_{1987..2022}_{1..12}.zip
```
Loading data with multiple threads:
```bash theme={null}
ls -1 *.zip | xargs -I{} -P $(nproc) bash -c "echo {}; unzip -cq {} '*.csv' | sed 's/\.00//g' | clickhouse-client --input_format_csv_empty_as_default 1 --query='INSERT INTO ontime FORMAT CSVWithNames'"
```
(if you will have memory shortage or other issues on your server, remove the `-P $(nproc)` part)
## Import from a saved copy
Alternatively, you can import data from a saved copy by the following query:
```sql theme={null}
INSERT INTO ontime SELECT * FROM s3('https://clickhouse-public-datasets.s3.amazonaws.com/ontime/csv_by_year/*.csv.gz', NOSIGN, CSVWithNames) SETTINGS max_insert_threads = 40;
```
The snapshot was created on 2022-05-29.
## Queries
Q0.
```sql theme={null}
SELECT avg(c1)
FROM
(
SELECT Year, Month, count(*) AS c1
FROM ontime
GROUP BY Year, Month
);
```
Q1. The number of flights per day from the year 2000 to 2008
```sql theme={null}
SELECT DayOfWeek, count(*) AS c
FROM ontime
WHERE Year>=2000 AND Year<=2008
GROUP BY DayOfWeek
ORDER BY c DESC;
```
Q2. The number of flights delayed by more than 10 minutes, grouped by the day of the week, for 2000-2008
```sql theme={null}
SELECT DayOfWeek, count(*) AS c
FROM ontime
WHERE DepDelay>10 AND Year>=2000 AND Year<=2008
GROUP BY DayOfWeek
ORDER BY c DESC;
```
Q3. The number of delays by the airport for 2000-2008
```sql theme={null}
SELECT Origin, count(*) AS c
FROM ontime
WHERE DepDelay>10 AND Year>=2000 AND Year<=2008
GROUP BY Origin
ORDER BY c DESC
LIMIT 10;
```
Q4. The number of delays by carrier for 2007
```sql theme={null}
SELECT IATA_CODE_Reporting_Airline AS Carrier, count(*)
FROM ontime
WHERE DepDelay>10 AND Year=2007
GROUP BY Carrier
ORDER BY count(*) DESC;
```
Q5. The percentage of delays by carrier for 2007
```sql theme={null}
SELECT Carrier, c, c2, c*100/c2 AS c3
FROM
(
SELECT
IATA_CODE_Reporting_Airline AS Carrier,
count(*) AS c
FROM ontime
WHERE DepDelay>10
AND Year=2007
GROUP BY Carrier
) q
JOIN
(
SELECT
IATA_CODE_Reporting_Airline AS Carrier,
count(*) AS c2
FROM ontime
WHERE Year=2007
GROUP BY Carrier
) qq USING Carrier
ORDER BY c3 DESC;
```
Better version of the same query:
```sql theme={null}
SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(DepDelay>10)*100 AS c3
FROM ontime
WHERE Year=2007
GROUP BY Carrier
ORDER BY c3 DESC
```
Q6. The previous request for a broader range of years, 2000-2008
```sql theme={null}
SELECT Carrier, c, c2, c*100/c2 AS c3
FROM
(
SELECT
IATA_CODE_Reporting_Airline AS Carrier,
count(*) AS c
FROM ontime
WHERE DepDelay>10
AND Year>=2000 AND Year<=2008
GROUP BY Carrier
) q
JOIN
(
SELECT
IATA_CODE_Reporting_Airline AS Carrier,
count(*) AS c2
FROM ontime
WHERE Year>=2000 AND Year<=2008
GROUP BY Carrier
) qq USING Carrier
ORDER BY c3 DESC;
```
Better version of the same query:
```sql theme={null}
SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(DepDelay>10)*100 AS c3
FROM ontime
WHERE Year>=2000 AND Year<=2008
GROUP BY Carrier
ORDER BY c3 DESC;
```
Q7. Percentage of flights delayed for more than 10 minutes, by year
```sql theme={null}
SELECT Year, c1/c2
FROM
(
SELECT
Year,
count(*)*100 AS c1
FROM ontime
WHERE DepDelay>10
GROUP BY Year
) q
JOIN
(
SELECT
Year,
count(*) AS c2
FROM ontime
GROUP BY Year
) qq USING (Year)
ORDER BY Year;
```
Better version of the same query:
```sql theme={null}
SELECT Year, avg(DepDelay>10)*100
FROM ontime
GROUP BY Year
ORDER BY Year;
```
Q8. The most popular destinations by the number of directly connected cities for various year ranges
```sql theme={null}
SELECT DestCityName, uniqExact(OriginCityName) AS u
FROM ontime
WHERE Year >= 2000 AND Year <= 2010
GROUP BY DestCityName
ORDER BY u DESC LIMIT 10;
```
Q9.
```sql theme={null}
SELECT Year, count(*) AS c1
FROM ontime
GROUP BY Year;
```
Q10.
```sql theme={null}
SELECT
min(Year), max(Year), IATA_CODE_Reporting_Airline AS Carrier, count(*) AS cnt,
sum(ArrDelayMinutes>30) AS flights_delayed,
round(sum(ArrDelayMinutes>30)/count(*),2) AS rate
FROM ontime
WHERE
DayOfWeek NOT IN (6,7) AND OriginState NOT IN ('AK', 'HI', 'PR', 'VI')
AND DestState NOT IN ('AK', 'HI', 'PR', 'VI')
AND FlightDate < '2010-01-01'
GROUP BY Carrier
HAVING cnt>100000 AND max(Year)>1990
ORDER BY rate DESC
LIMIT 1000;
```
Bonus:
```sql theme={null}
SELECT avg(cnt)
FROM
(
SELECT Year,Month,count(*) AS cnt
FROM ontime
WHERE DepDel15=1
GROUP BY Year,Month
);
SELECT avg(c1) FROM
(
SELECT Year,Month,count(*) AS c1
FROM ontime
GROUP BY Year,Month
);
SELECT DestCityName, uniqExact(OriginCityName) AS u
FROM ontime
GROUP BY DestCityName
ORDER BY u DESC
LIMIT 10;
SELECT OriginCityName, DestCityName, count() AS c
FROM ontime
GROUP BY OriginCityName, DestCityName
ORDER BY c DESC
LIMIT 10;
SELECT OriginCityName, count() AS c
FROM ontime
GROUP BY OriginCityName
ORDER BY c DESC
LIMIT 10;
```
You can also play with the data in Playground, [example](https://sql.clickhouse.com?query_id=M4FSVBVMSHY98NKCQP8N4K).
This performance test was created by Vadim Tkachenko. See:
* [https://www.percona.com/blog/2009/10/02/analyzing-air-traffic-performance-with-infobright-and-monetdb/](https://www.percona.com/blog/2009/10/02/analyzing-air-traffic-performance-with-infobright-and-monetdb/)
* [https://www.percona.com/blog/2009/10/26/air-traffic-queries-in-luciddb/](https://www.percona.com/blog/2009/10/26/air-traffic-queries-in-luciddb/)
* [https://www.percona.com/blog/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/](https://www.percona.com/blog/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/)
* [https://www.percona.com/blog/2014/04/21/using-apache-hadoop-and-impala-together-with-mysql-for-data-analysis/](https://www.percona.com/blog/2014/04/21/using-apache-hadoop-and-impala-together-with-mysql-for-data-analysis/)
* [https://www.percona.com/blog/2016/01/07/apache-spark-with-air-ontime-performance-data/](https://www.percona.com/blog/2016/01/07/apache-spark-with-air-ontime-performance-data/)
* [http://nickmakos.blogspot.ru/2012/08/analyzing-air-traffic-performance-with.html](http://nickmakos.blogspot.ru/2012/08/analyzing-air-traffic-performance-with.html)
# ClickHouse playground
Source: https://clickhouse.com/docs/get-started/sample-datasets/playground
The ClickHouse Playground allows people to experiment with ClickHouse by running queries instantly, without setting up their server or cluster.
[ClickHouse Playground](https://sql.clickhouse.com) allows people to experiment with ClickHouse by running queries instantly, without setting up their server or cluster.
Several example datasets are available in Playground.
You can make queries to Playground using any HTTP client, for example [curl](https://curl.haxx.se) or [wget](https://www.gnu.org/software/wget/), or set up a connection using [JDBC](/docs/integrations/language-clients/java/jdbc) or [ODBC](/docs/concepts/features/interfaces/odbc) drivers. More information about software products that support ClickHouse is available [here](/docs/integrations/home).
## Credentials
| Parameter | Value |
| :- | :- |
| HTTPS endpoint | `https://play.clickhouse.com:443/` |
| Native TCP endpoint | `play.clickhouse.com:9440` |
| User | `explorer` or `play` |
| Password | (empty) |
## Limitations
The queries are executed as a read-only user, so the following limitations apply:
* DDL queries aren't allowed
* INSERT queries aren't allowed
The service also has quotas on its usage.
To avoid the read-only limitations and experiment with loading your own data, you can sign up for a free [ClickHouse Cloud trial](https://clickhouse.cloud/signUp?loc=docs-get-started-sample-datasets-playground) or [install ClickHouse](/docs/get-started/setup/install) locally.
## Examples
HTTPS endpoint example with `curl`:
```bash theme={null}
curl "https://play.clickhouse.com/?user=explorer" --data-binary "SELECT 'Play ClickHouse'"
```
TCP endpoint example with [ClickHouse client](/docs/concepts/features/interfaces/client):
```bash theme={null}
clickhouse client --secure --host play.clickhouse.com --user explorer
```
## Playground specifications
Our ClickHouse Playground is running with the following specifications:
* Hosted on Google Cloud (GCE) in the US Central region (US-Central-1)
* 3-replica setup
* 256 GiB of storage and 59 virtual CPUs each.
# Analyzing Stack Overflow data with ClickHouse
Source: https://clickhouse.com/docs/get-started/sample-datasets/stackoverflow
Analyzing Stack Overflow data with ClickHouse
This dataset contains every `Posts`, `Users`, `Votes`, `Comments`, `Badges`, `PostHistory`, and `PostLinks` that has occurred on Stack Overflow.
You can either download pre-prepared Parquet versions of the data, containing every post up to April 2024, or download the latest data in XML format and load this. Stack Overflow provide updates to this data periodically - historically every 3 months.
The following diagram shows the schema for the available tables assuming Parquet format.
A description of the schema of this data can be found [here](https://meta.stackexchange.com/questions/2677/database-schema-documentation-for-the-public-data-dump-and-sede).
## Pre-prepared data
We provide a copy of this data in Parquet format, up to date as of April 2024. While small for ClickHouse with respect to the number of rows (60 million posts), this dataset contains significant volumes of text and large String columns.
```sql theme={null}
CREATE DATABASE stackoverflow
```
The following timings are for a 96 GiB, 24 vCPU ClickHouse Cloud cluster located in `eu-west-2`. The dataset is located in `eu-west-3`.
### Posts
```sql theme={null}
CREATE TABLE stackoverflow.posts
(
`Id` Int32 CODEC(Delta, Default),
`PostTypeId` Enum8('Question' = 1, 'Answer' = 2, 'Wiki' = 3, 'TagWikiExcerpt' = 4, 'TagWiki' = 5, 'ModeratorNomination' = 6, 'WikiPlaceholder' = 7, 'PrivilegeWiki' = 8),
`AcceptedAnswerId` UInt32,
`CreationDate` DateTime64(3, 'UTC'),
`Score` Int32,
`ViewCount` UInt32 CODEC(Delta, Default),
`Body` String,
`OwnerUserId` Int32,
`OwnerDisplayName` String,
`LastEditorUserId` Int32,
`LastEditorDisplayName` String,
`LastEditDate` DateTime64(3, 'UTC') CODEC(Delta, Default),
`LastActivityDate` DateTime64(3, 'UTC'),
`Title` String,
`Tags` String,
`AnswerCount` UInt16 CODEC(Delta, Default),
`CommentCount` UInt8,
`FavoriteCount` UInt8,
`ContentLicense` LowCardinality(String),
`ParentId` String,
`CommunityOwnedDate` DateTime64(3, 'UTC'),
`ClosedDate` DateTime64(3, 'UTC')
)
ENGINE = MergeTree
PARTITION BY toYear(CreationDate)
ORDER BY (PostTypeId, toDate(CreationDate), CreationDate);
INSERT INTO stackoverflow.posts SELECT * FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/posts/*.parquet', NOSIGN)
```
```response theme={null}
0 rows in set. Elapsed: 265.466 sec. Processed 59.82 million rows, 38.07 GB (225.34 thousand rows/s., 143.42 MB/s.)
```
Posts are also available by year e.g. [https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/posts/2020.parquet](https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/posts/2020.parquet)
### Votes
```sql theme={null}
CREATE TABLE stackoverflow.votes
(
`Id` UInt32,
`PostId` Int32,
`VoteTypeId` UInt8,
`CreationDate` DateTime64(3, 'UTC'),
`UserId` Int32,
`BountyAmount` UInt8
)
ENGINE = MergeTree
ORDER BY (VoteTypeId, CreationDate, PostId, UserId);
INSERT INTO stackoverflow.votes SELECT * FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/votes/*.parquet', NOSIGN)
```
```response theme={null}
0 rows in set. Elapsed: 21.605 sec. Processed 238.98 million rows, 2.13 GB (11.06 million rows/s., 98.46 MB/s.)
```
Votes are also available by year e.g. [https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/votes/2020.parquet](https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/votes/2020.parquet)
### Comments
```sql theme={null}
CREATE TABLE stackoverflow.comments
(
`Id` UInt32,
`PostId` UInt32,
`Score` UInt16,
`Text` String,
`CreationDate` DateTime64(3, 'UTC'),
`UserId` Int32,
`UserDisplayName` LowCardinality(String)
)
ENGINE = MergeTree
ORDER BY CreationDate;
INSERT INTO stackoverflow.comments SELECT * FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/comments/*.parquet', NOSIGN)
```
```response theme={null}
0 rows in set. Elapsed: 56.593 sec. Processed 90.38 million rows, 11.14 GB (1.60 million rows/s., 196.78 MB/s.)
```
Comments are also available by year e.g. [https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/comments/2020.parquet](https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/comments/2020.parquet)
### Users
```sql theme={null}
CREATE TABLE stackoverflow.users
(
`Id` Int32,
`Reputation` LowCardinality(String),
`CreationDate` DateTime64(3, 'UTC') CODEC(Delta, Default),
`DisplayName` String,
`LastAccessDate` DateTime64(3, 'UTC'),
`AboutMe` String,
`Views` UInt32,
`UpVotes` UInt32,
`DownVotes` UInt32,
`WebsiteUrl` String,
`Location` LowCardinality(String),
`AccountId` Int32
)
ENGINE = MergeTree
ORDER BY (Id, CreationDate);
INSERT INTO stackoverflow.users SELECT * FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/users.parquet', NOSIGN)
```
```response theme={null}
0 rows in set. Elapsed: 10.988 sec. Processed 22.48 million rows, 1.36 GB (2.05 million rows/s., 124.10 MB/s.)
```
### Badges
```sql theme={null}
CREATE TABLE stackoverflow.badges
(
`Id` UInt32,
`UserId` Int32,
`Name` LowCardinality(String),
`Date` DateTime64(3, 'UTC'),
`Class` Enum8('Gold' = 1, 'Silver' = 2, 'Bronze' = 3),
`TagBased` Bool
)
ENGINE = MergeTree
ORDER BY UserId;
INSERT INTO stackoverflow.badges SELECT * FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/badges.parquet', NOSIGN)
```
```response theme={null}
0 rows in set. Elapsed: 6.635 sec. Processed 51.29 million rows, 797.05 MB (7.73 million rows/s., 120.13 MB/s.)
```
### PostLinks
```sql theme={null}
CREATE TABLE stackoverflow.postlinks
(
`Id` UInt64,
`CreationDate` DateTime64(3, 'UTC'),
`PostId` Int32,
`RelatedPostId` Int32,
`LinkTypeId` Enum8('Linked' = 1, 'Duplicate' = 3)
)
ENGINE = MergeTree
ORDER BY (PostId, RelatedPostId);
INSERT INTO stackoverflow.postlinks SELECT * FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/postlinks.parquet', NOSIGN)
```
```response theme={null}
0 rows in set. Elapsed: 1.534 sec. Processed 6.55 million rows, 129.70 MB (4.27 million rows/s., 84.57 MB/s.)
```
### PostHistory
```sql theme={null}
CREATE TABLE stackoverflow.posthistory
(
`Id` UInt64,
`PostHistoryTypeId` UInt8,
`PostId` Int32,
`RevisionGUID` String,
`CreationDate` DateTime64(3, 'UTC'),
`UserId` Int32,
`Text` String,
`ContentLicense` LowCardinality(String),
`Comment` String,
`UserDisplayName` String
)
ENGINE = MergeTree
ORDER BY (CreationDate, PostId);
INSERT INTO stackoverflow.posthistory SELECT * FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/posthistory/*.parquet', NOSIGN)
```
```response theme={null}
0 rows in set. Elapsed: 422.795 sec. Processed 160.79 million rows, 67.08 GB (380.30 thousand rows/s., 158.67 MB/s.)
```
## Original dataset
The original dataset is available in compressed (7zip) XML format at [https://archive.org/download/stackexchange](https://archive.org/download/stackexchange) - files with prefix `stackoverflow.com*`.
### Download
```bash theme={null}
wget https://archive.org/download/stackexchange/stackoverflow.com-Badges.7z
wget https://archive.org/download/stackexchange/stackoverflow.com-Comments.7z
wget https://archive.org/download/stackexchange/stackoverflow.com-PostHistory.7z
wget https://archive.org/download/stackexchange/stackoverflow.com-PostLinks.7z
wget https://archive.org/download/stackexchange/stackoverflow.com-Posts.7z
wget https://archive.org/download/stackexchange/stackoverflow.com-Users.7z
wget https://archive.org/download/stackexchange/stackoverflow.com-Votes.7z
```
These files are up to 35GB and can take around 30 mins to download depending on internet connection - the download server throttles at around 20MB/sec.
### Convert to JSON
At the time of writing, ClickHouse doesn't have native support for XML as an input format. To load the data into ClickHouse we first convert to NDJSON.
To convert XML to JSON we recommend the [`xq`](https://github.com/kislyuk/yq) linux tool, a simple `jq` wrapper for XML documents.
Install xq and jq:
```bash theme={null}
sudo apt install jq
pip install yq
```
The following steps apply to any of the above files. We use the `stackoverflow.com-Posts.7z` file as an example. Modify as required.
Extract the file using [p7zip](https://p7zip.sourceforge.net/). This will produce a single xml file - in this case `Posts.xml`.
> Files are compressed approximately 4.5x. At 22GB compressed, the posts file requires around 97G uncompressed.
```bash theme={null}
p7zip -d stackoverflow.com-Posts.7z
```
The following splits the xml file into files, each containing 10000 rows.
```bash theme={null}
mkdir posts
cd posts
# the following splits the input xml file into sub files of 10000 rows
tail +3 ../Posts.xml | head -n -1 | split -l 10000 --filter='{ printf "\n"; cat - ; printf "\n"; } > $FILE' -
```
After running the above you will have a set of files, each with 10000 lines. This ensures the memory overhead of the next command isn't excessive (xml to JSON conversion is done in memory).
```bash theme={null}
find . -maxdepth 1 -type f -exec xq -c '.rows.row[]' {} \; | sed -e 's:"@:":g' > posts_v2.json
```
The above command will produce a single `posts.json` file.
Load into ClickHouse with the following command. Note the schema is specified for the `posts.json` file. This will need to be adjusted per data type to align with the target table.
```bash theme={null}
clickhouse local --query "SELECT * FROM file('posts.json', JSONEachRow, 'Id Int32, PostTypeId UInt8, AcceptedAnswerId UInt32, CreationDate DateTime64(3, \'UTC\'), Score Int32, ViewCount UInt32, Body String, OwnerUserId Int32, OwnerDisplayName String, LastEditorUserId Int32, LastEditorDisplayName String, LastEditDate DateTime64(3, \'UTC\'), LastActivityDate DateTime64(3, \'UTC\'), Title String, Tags String, AnswerCount UInt16, CommentCount UInt8, FavoriteCount UInt8, ContentLicense String, ParentId String, CommunityOwnedDate DateTime64(3, \'UTC\'), ClosedDate DateTime64(3, \'UTC\')') FORMAT Native" | clickhouse client --host --secure --password --query "INSERT INTO stackoverflow.posts_v2 FORMAT Native"
```
## Example queries
A few simple questions to you get started.
### Most popular tags on Stack Overflow
```sql theme={null}
SELECT
arrayJoin(arrayFilter(t -> (t != ''), splitByChar('|', Tags))) AS Tags,
count() AS c
FROM stackoverflow.posts
GROUP BY Tags
ORDER BY c DESC
LIMIT 10
```
```response theme={null}
┌─Tags───────┬───────c─┐
│ javascript │ 2527130 │
│ python │ 2189638 │
│ java │ 1916156 │
│ c# │ 1614236 │
│ php │ 1463901 │
│ android │ 1416442 │
│ html │ 1186567 │
│ jquery │ 1034621 │
│ c++ │ 806202 │
│ css │ 803755 │
└────────────┴─────────┘
10 rows in set. Elapsed: 1.013 sec. Processed 59.82 million rows, 1.21 GB (59.07 million rows/s., 1.19 GB/s.)
Peak memory usage: 224.03 MiB.
```
### User with the most answers (active accounts)
Account requires a `UserId`.
```sql theme={null}
SELECT
any(OwnerUserId) UserId,
OwnerDisplayName,
count() AS c
FROM stackoverflow.posts WHERE OwnerDisplayName != '' AND PostTypeId='Answer' AND OwnerUserId != 0
GROUP BY OwnerDisplayName
ORDER BY c DESC
LIMIT 5
```
```response theme={null}
┌─UserId─┬─OwnerDisplayName─┬────c─┐
│ 22656 │ Jon Skeet │ 2727 │
│ 23354 │ Marc Gravell │ 2150 │
│ 12950 │ tvanfosson │ 1530 │
│ 3043 │ Joel Coehoorn │ 1438 │
│ 10661 │ S.Lott │ 1087 │
└────────┴──────────────────┴──────┘
5 rows in set. Elapsed: 0.154 sec. Processed 35.83 million rows, 193.39 MB (232.33 million rows/s., 1.25 GB/s.)
Peak memory usage: 206.45 MiB.
```
### ClickHouse related posts with the most views
```sql theme={null}
SELECT
Id,
Title,
ViewCount,
AnswerCount
FROM stackoverflow.posts
WHERE Title ILIKE '%ClickHouse%'
ORDER BY ViewCount DESC
LIMIT 10
```
```response theme={null}
┌───────Id─┬─Title────────────────────────────────────────────────────────────────────────────┬─ViewCount─┬─AnswerCount─┐
│ 52355143 │ Is it possible to delete old records from clickhouse table? │ 41462 │ 3 │
│ 37954203 │ Clickhouse Data Import │ 38735 │ 3 │
│ 37901642 │ Updating data in Clickhouse │ 36236 │ 6 │
│ 58422110 │ Pandas: How to insert dataframe into Clickhouse │ 29731 │ 4 │
│ 63621318 │ DBeaver - Clickhouse - SQL Error [159] .. Read timed out │ 27350 │ 1 │
│ 47591813 │ How to filter clickhouse table by array column contents? │ 27078 │ 2 │
│ 58728436 │ How to search the string in query with case insensitive on Clickhouse database? │ 26567 │ 3 │
│ 65316905 │ Clickhouse: DB::Exception: Memory limit (for query) exceeded │ 24899 │ 2 │
│ 49944865 │ How to add a column in clickhouse │ 24424 │ 1 │
│ 59712399 │ How to cast date Strings to DateTime format with extended parsing in ClickHouse? │ 22620 │ 1 │
└──────────┴──────────────────────────────────────────────────────────────────────────────────┴───────────┴─────────────┘
10 rows in set. Elapsed: 0.472 sec. Processed 59.82 million rows, 1.91 GB (126.63 million rows/s., 4.03 GB/s.)
Peak memory usage: 240.01 MiB.
```
### Most controversial posts
```sql theme={null}
SELECT
Id,
Title,
UpVotes,
DownVotes,
abs(UpVotes - DownVotes) AS Controversial_ratio
FROM stackoverflow.posts
INNER JOIN
(
SELECT
PostId,
countIf(VoteTypeId = 2) AS UpVotes,
countIf(VoteTypeId = 3) AS DownVotes
FROM stackoverflow.votes
GROUP BY PostId
HAVING (UpVotes > 10) AND (DownVotes > 10)
) AS votes ON posts.Id = votes.PostId
WHERE Title != ''
ORDER BY Controversial_ratio ASC
LIMIT 3
```
```response theme={null}
┌───────Id─┬─Title─────────────────────────────────────────────┬─UpVotes─┬─DownVotes─┬─Controversial_ratio─┐
│ 583177 │ VB.NET Infinite For Loop │ 12 │ 12 │ 0 │
│ 9756797 │ Read console input as enumerable - one statement? │ 16 │ 16 │ 0 │
│ 13329132 │ What's the point of ARGV in Ruby? │ 22 │ 22 │ 0 │
└──────────┴───────────────────────────────────────────────────┴─────────┴───────────┴─────────────────────┘
3 rows in set. Elapsed: 4.779 sec. Processed 298.80 million rows, 3.16 GB (62.52 million rows/s., 661.05 MB/s.)
Peak memory usage: 6.05 GiB.
```
## Attribution
We thank Stack Overflow for providing this data under the `cc-by-sa 4.0` license, acknowledging their efforts and the original source of the data at [https://archive.org/details/stackexchange](https://archive.org/details/stackexchange).
# Star Schema Benchmark (SSB, 2009)
Source: https://clickhouse.com/docs/get-started/sample-datasets/star-schema
The Star Schema Benchmark (SSB) data set and queries
The Star Schema Benchmark is roughly based on the [TPC-H](/docs/get-started/sample-datasets/tpch)'s tables and queries but unlike TPC-H, it uses a star schema layout.
The bulk of the data sits in a gigantic fact table which is surrounded by multiple small dimension tables.
The queries joined the fact table with one or more dimension tables to apply filter criteria, e.g. `MONTH = 'JANUARY'`.
References:
* [Star Schema Benchmark](https://cs.umb.edu/~poneil/StarSchemaB.pdf) (O'Neil et. al), 2009
* [Variations of the Star Schema Benchmark to Test the Effects of Data Skew on Query Performance](https://doi.org/10.1145/2479871.2479927) (Rabl. et. al.), 2013
## Data generation
First, checkout the star schema benchmark repository and compile the data generator:
```bash theme={null}
git clone https://github.com/vadimtk/ssb-dbgen.git
cd ssb-dbgen
make
```
Then, generate the data. Parameter `-s` specifies the scale factor. For example, with `-s 100`, 600 million rows are generated.
```bash theme={null}
./dbgen -s 1000 -T c
./dbgen -s 1000 -T l
./dbgen -s 1000 -T p
./dbgen -s 1000 -T s
./dbgen -s 1000 -T d
```
## Create tables
Now create tables in ClickHouse:
```sql theme={null}
CREATE TABLE customer
(
C_CUSTKEY UInt32,
C_NAME String,
C_ADDRESS String,
C_CITY LowCardinality(String),
C_NATION LowCardinality(String),
C_REGION LowCardinality(String),
C_PHONE String,
C_MKTSEGMENT LowCardinality(String)
)
ENGINE = MergeTree ORDER BY (C_CUSTKEY);
CREATE TABLE lineorder
(
LO_ORDERKEY UInt32,
LO_LINENUMBER UInt8,
LO_CUSTKEY UInt32,
LO_PARTKEY UInt32,
LO_SUPPKEY UInt32,
LO_ORDERDATE Date,
LO_ORDERPRIORITY LowCardinality(String),
LO_SHIPPRIORITY UInt8,
LO_QUANTITY UInt8,
LO_EXTENDEDPRICE UInt32,
LO_ORDTOTALPRICE UInt32,
LO_DISCOUNT UInt8,
LO_REVENUE UInt32,
LO_SUPPLYCOST UInt32,
LO_TAX UInt8,
LO_COMMITDATE Date,
LO_SHIPMODE LowCardinality(String)
)
ENGINE = MergeTree PARTITION BY toYear(LO_ORDERDATE) ORDER BY (LO_ORDERDATE, LO_ORDERKEY);
CREATE TABLE part
(
P_PARTKEY UInt32,
P_NAME String,
P_MFGR LowCardinality(String),
P_CATEGORY LowCardinality(String),
P_BRAND LowCardinality(String),
P_COLOR LowCardinality(String),
P_TYPE LowCardinality(String),
P_SIZE UInt8,
P_CONTAINER LowCardinality(String)
)
ENGINE = MergeTree ORDER BY P_PARTKEY;
CREATE TABLE supplier
(
S_SUPPKEY UInt32,
S_NAME String,
S_ADDRESS String,
S_CITY LowCardinality(String),
S_NATION LowCardinality(String),
S_REGION LowCardinality(String),
S_PHONE String
)
ENGINE = MergeTree ORDER BY S_SUPPKEY;
CREATE TABLE date
(
D_DATEKEY Date,
D_DATE FixedString(18),
D_DAYOFWEEK LowCardinality(String),
D_MONTH LowCardinality(String),
D_YEAR UInt16,
D_YEARMONTHNUM UInt32,
D_YEARMONTH LowCardinality(FixedString(7)),
D_DAYNUMINWEEK UInt8,
D_DAYNUMINMONTH UInt8,
D_DAYNUMINYEAR UInt16,
D_MONTHNUMINYEAR UInt8,
D_WEEKNUMINYEAR UInt8,
D_SELLINGSEASON String,
D_LASTDAYINWEEKFL UInt8,
D_LASTDAYINMONTHFL UInt8,
D_HOLIDAYFL UInt8,
D_WEEKDAYFL UInt8
)
ENGINE = MergeTree ORDER BY D_DATEKEY;
```
## Import data
The data can be imported as follows:
```bash theme={null}
clickhouse-client --query "INSERT INTO customer FORMAT CSV" < customer.tbl
clickhouse-client --query "INSERT INTO part FORMAT CSV" < part.tbl
clickhouse-client --query "INSERT INTO supplier FORMAT CSV" < supplier.tbl
clickhouse-client --query "INSERT INTO lineorder FORMAT CSV" < lineorder.tbl
clickhouse-client --query "INSERT INTO date FORMAT CSV" < date.tbl
```
## Denormalized table
In many use cases of ClickHouse, multiple tables are converted into a single denormalized flat table.
This step is optional — below, queries are listed in their original form and in a format rewritten for the denormalized table.
```sql theme={null}
SET max_memory_usage = 20000000000;
CREATE TABLE lineorder_flat
ENGINE = MergeTree ORDER BY (LO_ORDERDATE, LO_ORDERKEY)
AS SELECT
l.LO_ORDERKEY AS LO_ORDERKEY,
l.LO_LINENUMBER AS LO_LINENUMBER,
l.LO_CUSTKEY AS LO_CUSTKEY,
l.LO_PARTKEY AS LO_PARTKEY,
l.LO_SUPPKEY AS LO_SUPPKEY,
l.LO_ORDERDATE AS LO_ORDERDATE,
l.LO_ORDERPRIORITY AS LO_ORDERPRIORITY,
l.LO_SHIPPRIORITY AS LO_SHIPPRIORITY,
l.LO_QUANTITY AS LO_QUANTITY,
l.LO_EXTENDEDPRICE AS LO_EXTENDEDPRICE,
l.LO_ORDTOTALPRICE AS LO_ORDTOTALPRICE,
l.LO_DISCOUNT AS LO_DISCOUNT,
l.LO_REVENUE AS LO_REVENUE,
l.LO_SUPPLYCOST AS LO_SUPPLYCOST,
l.LO_TAX AS LO_TAX,
l.LO_COMMITDATE AS LO_COMMITDATE,
l.LO_SHIPMODE AS LO_SHIPMODE,
c.C_NAME AS C_NAME,
c.C_ADDRESS AS C_ADDRESS,
c.C_CITY AS C_CITY,
c.C_NATION AS C_NATION,
c.C_REGION AS C_REGION,
c.C_PHONE AS C_PHONE,
c.C_MKTSEGMENT AS C_MKTSEGMENT,
s.S_NAME AS S_NAME,
s.S_ADDRESS AS S_ADDRESS,
s.S_CITY AS S_CITY,
s.S_NATION AS S_NATION,
s.S_REGION AS S_REGION,
s.S_PHONE AS S_PHONE,
p.P_NAME AS P_NAME,
p.P_MFGR AS P_MFGR,
p.P_CATEGORY AS P_CATEGORY,
p.P_BRAND AS P_BRAND,
p.P_COLOR AS P_COLOR,
p.P_TYPE AS P_TYPE,
p.P_SIZE AS P_SIZE,
p.P_CONTAINER AS P_CONTAINER,
d.D_YEAR AS D_YEAR,
d.D_YEARMONTHNUM AS D_YEARMONTHNUM,
d.D_YEARMONTH AS D_YEARMONTH,
d.D_WEEKNUMINYEAR AS D_WEEKNUMINYEAR
FROM lineorder AS l
INNER JOIN customer AS c ON c.C_CUSTKEY = l.LO_CUSTKEY
INNER JOIN supplier AS s ON s.S_SUPPKEY = l.LO_SUPPKEY
INNER JOIN part AS p ON p.P_PARTKEY = l.LO_PARTKEY
INNER JOIN date AS d ON d.D_DATEKEY = l.LO_ORDERDATE;
```
## Queries
The queries are generated by `./qgen -s `. Example queries for `s = 100`:
### Q1.1
```sql theme={null}
SELECT
sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS REVENUE
FROM
lineorder,
date
WHERE
LO_ORDERDATE = D_DATEKEY
AND D_YEAR = 1993
AND LO_DISCOUNT BETWEEN 1 AND 3
AND LO_QUANTITY < 25;
```
Denormalized table:
```sql theme={null}
SELECT
sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
FROM
lineorder_flat
WHERE
toYear(LO_ORDERDATE) = 1993
AND LO_DISCOUNT BETWEEN 1 AND 3
AND LO_QUANTITY < 25;
```
### Q1.2
```sql theme={null}
SELECT
sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS REVENUE
FROM
lineorder,
date
WHERE
LO_ORDERDATE = D_DATEKEY
AND D_YEARMONTHNUM = 199401
AND LO_DISCOUNT BETWEEN 4 AND 6
AND LO_QUANTITY BETWEEN 26 AND 35;
```
Denormalized table:
```sql theme={null}
SELECT
sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
FROM
lineorder_flat
WHERE
toYYYYMM(LO_ORDERDATE) = 199401
AND LO_DISCOUNT BETWEEN 4 AND 6
AND LO_QUANTITY BETWEEN 26 AND 35;
```
### Q1.3
```sql theme={null}
SELECT
sum(LO_EXTENDEDPRICE*LO_DISCOUNT) AS REVENUE
FROM
lineorder,
date
WHERE
LO_ORDERDATE = D_DATEKEY
AND D_WEEKNUMINYEAR = 6
AND D_YEAR = 1994
AND LO_DISCOUNT BETWEEN 5 AND 7
AND LO_QUANTITY BETWEEN 26 AND 35;
```
Denormalized table:
```sql theme={null}
SELECT
sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
FROM
lineorder_flat
WHERE
D_WEEKNUMINYEAR = 6
AND D_YEAR = 1994
AND LO_DISCOUNT BETWEEN 5 AND 7
AND LO_QUANTITY BETWEEN 26 AND 35;
```
### Q2.1
```sql theme={null}
SELECT
sum(LO_REVENUE),
D_YEAR,
P_BRAND
FROM
lineorder,
date,
part,
supplier
WHERE
LO_ORDERDATE = D_DATEKEY
AND LO_PARTKEY = P_PARTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND P_CATEGORY = 'MFGR#12'
AND S_REGION = 'AMERICA'
GROUP BY
D_YEAR,
P_BRAND
ORDER BY
D_YEAR,
P_BRAND;
```
Denormalized table:
```sql theme={null}
SELECT
sum(LO_REVENUE),
toYear(LO_ORDERDATE) AS year,
P_BRAND
FROM lineorder_flat
WHERE
P_CATEGORY = 'MFGR#12'
AND S_REGION = 'AMERICA'
GROUP BY
year,
P_BRAND
ORDER BY
year,
P_BRAND;
```
### Q2.2
```sql theme={null}
SELECT
sum(LO_REVENUE),
D_YEAR,
P_BRAND
FROM
lineorder,
date,
part,
supplier
WHERE
LO_ORDERDATE = D_DATEKEY
AND LO_PARTKEY = P_PARTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND P_BRAND BETWEEN
'MFGR#2221' AND 'MFGR#2228'
AND S_REGION = 'ASIA'
GROUP BY
D_YEAR,
P_BRAND
ORDER BY
D_YEAR,
P_BRAND;
```
Denormalized table:
```sql theme={null}
SELECT
sum(LO_REVENUE),
toYear(LO_ORDERDATE) AS year,
P_BRAND
FROM lineorder_flat
WHERE P_BRAND >= 'MFGR#2221' AND P_BRAND <= 'MFGR#2228' AND S_REGION = 'ASIA'
GROUP BY
year,
P_BRAND
ORDER BY
year,
P_BRAND;
```
### Q2.3
```sql theme={null}
SELECT
sum(LO_REVENUE),
D_YEAR,
P_BRAND
FROM
lineorder,
date,
part,
supplier
WHERE
LO_ORDERDATE = D_DATEKEY
AND LO_PARTKEY = P_PARTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND P_BRAND = 'MFGR#2221'
AND S_REGION = 'EUROPE'
GROUP BY
D_YEAR,
P_BRAND
ORDER BY
D_YEAR,
P_BRAND;
```
Denormalized table:
```sql theme={null}
SELECT
sum(LO_REVENUE),
toYear(LO_ORDERDATE) AS year,
P_BRAND
FROM lineorder_flat
WHERE P_BRAND = 'MFGR#2221' AND S_REGION = 'EUROPE'
GROUP BY
year,
P_BRAND
ORDER BY
year,
P_BRAND;
```
### Q3.1
```sql theme={null}
SELECT
C_NATION,
S_NATION,
D_YEAR,
sum(LO_REVENUE) AS REVENUE
FROM
customer,
lineorder,
supplier,
date
WHERE
LO_CUSTKEY = C_CUSTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND LO_ORDERDATE = D_DATEKEY
AND C_REGION = 'ASIA' AND S_REGION = 'ASIA'
AND D_YEAR >= 1992 AND D_YEAR <= 1997
GROUP BY
C_NATION,
S_NATION,
D_YEAR
ORDER BY
D_YEAR ASC,
REVENUE DESC;
```
Denormalized table:
```sql theme={null}
SELECT
C_NATION,
S_NATION,
toYear(LO_ORDERDATE) AS year,
sum(LO_REVENUE) AS revenue
FROM lineorder_flat
WHERE
C_REGION = 'ASIA'
AND S_REGION = 'ASIA'
AND year >= 1992
AND year <= 1997
GROUP BY
C_NATION,
S_NATION,
year
ORDER BY
year ASC,
revenue DESC;
```
### Q3.2
```sql theme={null}
SELECT
C_CITY,
S_CITY,
D_YEAR,
sum(LO_REVENUE) AS REVENUE
FROM
customer,
lineorder,
supplier,
date
WHERE
LO_CUSTKEY = C_CUSTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND LO_ORDERDATE = D_DATEKEY
AND C_NATION = 'UNITED STATES'
AND S_NATION = 'UNITED STATES'
AND D_YEAR >= 1992 AND D_YEAR <= 1997
GROUP BY
C_CITY,
S_CITY,
D_YEAR
ORDER BY
D_YEAR ASC,
REVENUE DESC;
```
Denormalized table:
```sql theme={null}
SELECT
C_CITY,
S_CITY,
toYear(LO_ORDERDATE) AS year,
sum(LO_REVENUE) AS revenue
FROM lineorder_flat
WHERE
C_NATION = 'UNITED STATES'
AND S_NATION = 'UNITED STATES'
AND year >= 1992
AND year <= 1997
GROUP BY
C_CITY,
S_CITY,
year
ORDER BY
year ASC,
revenue DESC;
```
### Q3.3
```sql theme={null}
SELECT
C_CITY,
S_CITY,
D_YEAR,
sum(LO_REVENUE) AS revenue
FROM
customer,
lineorder,
supplier,
date
WHERE
LO_CUSTKEY = C_CUSTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND LO_ORDERDATE = D_DATEKEY
AND (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5')
AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5')
AND D_YEAR >= 1992
AND D_YEAR <= 1997
GROUP BY
C_CITY,
S_CITY,
D_YEAR
ORDER BY
D_YEAR ASC,
revenue DESC;
```
Denormalized table:
```sql theme={null}
SELECT
C_CITY,
S_CITY,
toYear(LO_ORDERDATE) AS year,
sum(LO_REVENUE) AS revenue
FROM lineorder_flat
WHERE
(C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5')
AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5')
AND year >= 1992
AND year <= 1997
GROUP BY
C_CITY,
S_CITY,
year
ORDER BY
year ASC,
revenue DESC;
```
### Q3.4
```sql theme={null}
SELECT
C_CITY,
S_CITY,
D_YEAR,
sum(LO_REVENUE) AS revenue
FROM
customer,
lineorder,
supplier,
date
WHERE
LO_CUSTKEY = C_CUSTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND LO_ORDERDATE = D_DATEKEY
AND (C_CITY='UNITED KI1' OR C_CITY='UNITED KI5')
AND (S_CITY='UNITED KI1' OR S_CITY='UNITED KI5')
AND D_YEARMONTH = 'Dec1997'
GROUP BY
C_CITY,
S_CITY,
D_YEAR
ORDER BY
D_YEAR ASC,
revenue DESC;
```
Denormalized table:
```sql theme={null}
SELECT
C_CITY,
S_CITY,
toYear(LO_ORDERDATE) AS year,
sum(LO_REVENUE) AS revenue
FROM lineorder_flat
WHERE
(C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5')
AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5')
AND toYYYYMM(LO_ORDERDATE) = 199712
GROUP BY
C_CITY,
S_CITY,
year
ORDER BY
year ASC,
revenue DESC;
```
### Q4.1
```sql theme={null}
SELECT
D_YEAR,
C_NATION,
sum(LO_REVENUE - LO_SUPPLYCOST) AS PROFIT
FROM
date,
customer,
supplier,
part,
lineorder
WHERE
LO_CUSTKEY = C_CUSTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND LO_PARTKEY = P_PARTKEY
AND LO_ORDERDATE = D_DATEKEY
AND C_REGION = 'AMERICA'
AND S_REGION = 'AMERICA'
AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2')
GROUP BY
D_YEAR,
C_NATION
ORDER BY
D_YEAR,
C_NATION
```
Denormalized table:
```sql theme={null}
SELECT
toYear(LO_ORDERDATE) AS year,
C_NATION,
sum(LO_REVENUE - LO_SUPPLYCOST) AS profit
FROM lineorder_flat
WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2')
GROUP BY
year,
C_NATION
ORDER BY
year ASC,
C_NATION ASC;
```
### Q4.2
```sql theme={null}
SELECT
D_YEAR,
S_NATION,
P_CATEGORY,
sum(LO_REVENUE - LO_SUPPLYCOST) AS profit
FROM
date,
customer,
supplier,
part,
lineorder
WHERE
LO_CUSTKEY = C_CUSTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND LO_PARTKEY = P_PARTKEY
AND LO_ORDERDATE = D_DATEKEY
AND C_REGION = 'AMERICA'
AND S_REGION = 'AMERICA'
AND (D_YEAR = 1997 OR D_YEAR = 1998)
AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2')
GROUP BY
D_YEAR,
S_NATION,
P_CATEGORY
ORDER BY
D_YEAR,
S_NATION,
P_CATEGORY
```
Denormalized table:
```sql theme={null}
SELECT
toYear(LO_ORDERDATE) AS year,
S_NATION,
P_CATEGORY,
sum(LO_REVENUE - LO_SUPPLYCOST) AS profit
FROM lineorder_flat
WHERE
C_REGION = 'AMERICA'
AND S_REGION = 'AMERICA'
AND (year = 1997 OR year = 1998)
AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2')
GROUP BY
year,
S_NATION,
P_CATEGORY
ORDER BY
year ASC,
S_NATION ASC,
P_CATEGORY ASC;
```
### Q4.3
```sql theme={null}
SELECT
D_YEAR,
S_CITY,
P_BRAND,
sum(LO_REVENUE - LO_SUPPLYCOST) AS profit
FROM
date,
customer,
supplier,
part,
lineorder
WHERE
LO_CUSTKEY = C_CUSTKEY
AND LO_SUPPKEY = S_SUPPKEY
AND LO_PARTKEY = P_PARTKEY
AND LO_ORDERDATE = D_DATEKEY
AND C_REGION = 'AMERICA'
AND S_NATION = 'UNITED STATES'
AND (D_YEAR = 1997 OR D_YEAR = 1998)
AND P_CATEGORY = 'MFGR#14'
GROUP BY
D_YEAR,
S_CITY,
P_BRAND
ORDER BY
D_YEAR,
S_CITY,
P_BRAND
```
Denormalized table:
```sql theme={null}
SELECT
toYear(LO_ORDERDATE) AS year,
S_CITY,
P_BRAND,
sum(LO_REVENUE - LO_SUPPLYCOST) AS profit
FROM
lineorder_flat
WHERE
C_REGION = 'AMERICA'
AND S_NATION = 'UNITED STATES'
AND (year = 1997 OR year = 1998)
AND P_CATEGORY = 'MFGR#14'
GROUP BY
year,
S_CITY,
P_BRAND
ORDER BY
year ASC,
S_CITY ASC,
P_BRAND ASC;
```
# TPC-DS (2012)
Source: https://clickhouse.com/docs/get-started/sample-datasets/tpcds
The TPC-DS benchmark data set and queries.
Similar to the [Star Schema Benchmark (SSB)](/docs/get-started/sample-datasets/star-schema), TPC-DS is based on [TPC-H](/docs/get-started/sample-datasets/tpch), but it took the opposite route, i.e. it expanded the number of joins needed by storing the data in a complex snowflake schema (24 instead of 8 tables). The data distribution is skewed (e.g. normal and Poisson distributions). It includes 99 reporting and ad-hoc queries with random substitutions.
**References**
* [The Making of TPC-DS](https://dl.acm.org/doi/10.5555/1182635.1164217) (Nambiar), 2006
## Data Generation and Import
First, checkout the TPC-DS repository and compile the data generator:
```bash theme={null}
git clone https://github.com/gregrahn/tpcds-kit.git
cd tpcds-kit/tools
make
```
Then, generate the data. Parameter `-scale` specifies the scale factor.
```bash theme={null}
./dsdgen -scale 1
```
Now create tables in ClickHouse. The table definitions are available in [`init.sql`](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/tpc-ds/init.sql) in the ClickHouse repository.
The data can be imported as follows:
```bash theme={null}
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO call_center FORMAT CSV" < call_center.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO catalog_page FORMAT CSV" < catalog_page.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO catalog_returns FORMAT CSV" < catalog_returns.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO catalog_sales FORMAT CSV" < catalog_sales.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO customer FORMAT CSV" < customer.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO customer_address FORMAT CSV" < customer_address.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO customer_demographics FORMAT CSV" < customer_demographics.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO date_dim FORMAT CSV" < date_dim.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO household_demographics FORMAT CSV" < household_demographics.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO income_band FORMAT CSV" < income_band.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO inventory FORMAT CSV" < inventory.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO item FORMAT CSV" < item.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO promotion FORMAT CSV" < promotion.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO reason FORMAT CSV" < reason.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO ship_mode FORMAT CSV" < ship_mode.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO store FORMAT CSV" < store.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO store_returns FORMAT CSV" < store_returns.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO store_sales FORMAT CSV" < store_sales.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO time_dim FORMAT CSV" < time_dim.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO warehouse FORMAT CSV" < warehouse.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO web_page FORMAT CSV" < web_page.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO web_returns FORMAT CSV" < web_returns.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO web_sales FORMAT CSV" < web_sales.dat
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO web_site FORMAT CSV" < web_site.dat
```
Then run the generated queries.
## Queries
The 99 TPC-DS queries can be found [here](https://github.com/ClickHouse/ClickHouse/tree/master/tests/benchmarks/tpc-ds/queries) in the ClickHouse repository.
To get SQL standard compatible behavior and expected results, apply the settings from [`settings.json`](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/tpc-ds/settings.json). See the [README](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/tpc-ds/README.md) for known issues and notes on specific queries.
**Correctness**
The results of the queries agree with the official results unless otherwise noted. There may be minor precision differences, which are permitted by the TPC-DS specification.
## Performance benchmark
ClickHouse tracks TPC-DS query performance across every released version. You can explore run times for all 99 TPC-DS queries (scale factor 32) on the [ClickHouse versions benchmark](https://benchmark.clickhouse.com/versions/#version=+s\&dataset=+ps\&metric=hot\&queries=-) page to see how performance has evolved over time.
# TPC-H (1999)
Source: https://clickhouse.com/docs/get-started/sample-datasets/tpch
The TPC-H benchmark data set and queries.
A popular benchmark which models the internal data warehouse of a wholesale supplier. The data is stored into a 3rd normal form representation, requiring lots of joins at query runtime. Despite its age and its unrealistic assumption that the data is uniformly and independently distributed, TPC-H remains the most popular OLAP benchmark to date.
**References**
* [TPC-H](https://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp)
* [New TPC Benchmarks for Decision Support and Web Commerce](https://doi.org/10.1145/369275.369291) (Poess et. al., 2000)
* [TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark](https://doi.org/10.1007/978-3-319-04936-6_5) (Boncz et. al.), 2013
* [Quantifying TPC-H Choke Points and Their Optimizations](https://doi.org/10.14778/3389133.3389138) (Dresseler et. al.), 2020
## Data Generation and Import
First, checkout the TPC-H repository and compile the data generator:
```bash theme={null}
git clone https://github.com/gregrahn/tpch-kit.git
cd tpch-kit/dbgen
make
```
Then, generate the data. Parameter `-s` specifies the scale factor. For example, with `-s 100`, 600 million rows are generated for table 'lineitem'.
```bash theme={null}
./dbgen -s 100
```
To speed things up, you can use "chunked" generation (in multiple processes):
```bash theme={null}
for i in $(seq 1 8); do
./dbgen -s 100 -C 8 -S $i &
done
wait
```
Detailed table sizes with scale factor 100:
| Table | size (in rows) | size (compressed in ClickHouse) |
| - | - | - |
| nation | 25 | 2 kB |
| region | 5 | 1 kB |
| part | 20.000.000 | 895 MB |
| supplier | 1.000.000 | 75 MB |
| partsupp | 80.000.000 | 4.37 GB |
| customer | 15.000.000 | 1.19 GB |
| orders | 150.000.000 | 6.15 GB |
| lineitem | 600.000.000 | 26.69 GB |
(Compressed sizes in ClickHouse are taken from `system.tables.total_bytes` and based on below table definitions.)
Now create tables in ClickHouse. The table definitions are available in [`init.sql`](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/tpc-h/init.sql) in the ClickHouse repository.
The data can be imported as follows:
```bash theme={null}
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO nation FORMAT CSV" < nation.tbl
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO region FORMAT CSV" < region.tbl
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO part FORMAT CSV" < part.tbl
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO supplier FORMAT CSV" < supplier.tbl
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO partsupp FORMAT CSV" < partsupp.tbl
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO customer FORMAT CSV" < customer.tbl
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO orders FORMAT CSV" < orders.tbl
clickhouse-client --format_csv_delimiter '|' --query "INSERT INTO lineitem FORMAT CSV" < lineitem.tbl
```
> [!NOTE]
>
> Instead of using tpch-kit and generating the tables by yourself, you can alternatively import the data from a public S3 bucket. Make sure to create empty tables first using [`init.sql`](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/tpc-h/init.sql).
>
> ```sql theme={null}
> -- Scale factor 1
> INSERT INTO nation SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/1/nation.tbl', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO region SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/1/region.tbl', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO part SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/1/part.tbl', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO supplier SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/1/supplier.tbl', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO partsupp SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/1/partsupp.tbl', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO customer SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/1/customer.tbl', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO orders SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/1/orders.tbl', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO lineitem SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/1/lineitem.tbl', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
>
> -- Scale factor 10
> INSERT INTO nation SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/10/nation.tbl.zst', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO region SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/10/region.tbl.zst', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO part SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/10/part.tbl.zst', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO supplier SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/10/supplier.tbl.zst', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO partsupp SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/10/partsupp.tbl.zst', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO customer SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/10/customer.tbl.zst', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO orders SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/10/orders.tbl.zst', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO lineitem SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/10/lineitem.tbl.zst', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
>
> -- Scale factor 100
> INSERT INTO nation SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/100/nation.tbl.gz', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO region SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/100/region.tbl.gz', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO part SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/100/part.tbl.gz', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO supplier SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/100/supplier.tbl.gz', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO partsupp SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/100/partsupp.tbl.gz', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO customer SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/100/customer.tbl.gz', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO orders SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/100/orders.tbl.gz', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> INSERT INTO lineitem SELECT * FROM s3('https://clickhouse-datasets.s3.amazonaws.com/h/100/lineitem.tbl.gz', NOSIGN, CSV) SETTINGS format_csv_delimiter = '|', input_format_defaults_for_omitted_fields = 1, input_format_csv_empty_as_default = 1;
> ```
## Queries
The 22 TPC-H queries can be found [here](https://github.com/ClickHouse/ClickHouse/tree/master/tests/benchmarks/tpc-h/queries) in the ClickHouse repository.
To get SQL standard compatible behavior and expected results, apply the settings from [`settings.json`](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/tpc-h/settings.json). See the [README](https://github.com/ClickHouse/ClickHouse/blob/master/tests/benchmarks/tpc-h/README.md) for known issues and notes on specific queries.
**Correctness**
The result of the queries agrees with the official results unless mentioned otherwise. To verify, generate a TPC-H database with scale factor = 1 (`dbgen`, see above) and compare with the [expected results in tpch-kit](https://github.com/gregrahn/tpch-kit/tree/master/dbgen/answers).
## Performance benchmark
ClickHouse tracks TPC-H query performance across every released version. You can explore run times for all 22 TPC-H queries (scale factor 40) on the [ClickHouse versions benchmark](https://benchmark.clickhouse.com/versions/#version=+s\&dataset=+ph\&metric=hot\&queries=-) page to see how performance has evolved over time.
# Taiwan historical weather datasets
Source: https://clickhouse.com/docs/get-started/sample-datasets/tw-weather
131 million rows of weather observation data for the last 128 yrs
This dataset contains historical meteorological observations measurements for the last 128 years. Each row is a measurement for a point in date time and weather station.
The origin of this dataset is available [here](https://github.com/Raingel/historical_weather) and the list of weather station numbers can be found [here](https://github.com/Raingel/weather_station_list).
> The sources of meteorological datasets include the meteorological stations that are established by the Central Weather Administration (station code is beginning with C0, C1, and 4) and the agricultural meteorological stations belonging to the Council of Agriculture (station code other than those mentioned above):
* StationId
* MeasuredDate, the observation time
* StnPres, the station air pressure
* SeaPres, the sea level pressure
* Td, the dew point temperature
* RH, the relative humidity
* Other elements where available
## Downloading the data
* A [pre-processed version](#pre-processed-data) of the data for the ClickHouse, which has been cleaned, re-structured, and enriched. This dataset covers the years from 1896 to 2023.
* [Download the original raw data](#original-raw-data) and convert to the format required by ClickHouse. Users wanting to add their own columns may wish to explore or complete their approaches.
### Pre-processed data
The dataset has also been re-structured from a measurement per line to a row per weather station id and measured date, i.e.
```csv theme={null}
StationId,MeasuredDate,StnPres,Tx,RH,WS,WD,WSGust,WDGust,Precp,GloblRad,TxSoil0cm,TxSoil5cm,TxSoil20cm,TxSoil50cm,TxSoil100cm,SeaPres,Td,PrecpHour,SunShine,TxSoil10cm,EvapA,Visb,UVI,Cloud Amount,TxSoil30cm,TxSoil200cm,TxSoil300cm,TxSoil500cm,VaporPressure
C0X100,2016-01-01 01:00:00,1022.1,16.1,72,1.1,8.0,,,,,,,,,,,,,,,,,,,,,,,
C0X100,2016-01-01 02:00:00,1021.6,16.0,73,1.2,358.0,,,,,,,,,,,,,,,,,,,,,,,
C0X100,2016-01-01 03:00:00,1021.3,15.8,74,1.5,353.0,,,,,,,,,,,,,,,,,,,,,,,
C0X100,2016-01-01 04:00:00,1021.2,15.8,74,1.7,8.0,,,,,,,,,,,,,,,,,,,,,,,
```
It is easy to query and ensure that the resulting table has less sparse and some elements are null because they're not available to be measured in this weather station.
This dataset is available in the following Google CloudStorage location. Either download the dataset to your local filesystem (and insert them with the ClickHouse client) or insert them directly into the ClickHouse (see [Inserting from URL](#inserting-from-url)).
To download:
```bash theme={null}
wget https://storage.googleapis.com/taiwan-weather-observaiton-datasets/preprocessed_weather_daily_1896_2023.tar.gz
# Option: Validate the checksum
md5sum preprocessed_weather_daily_1896_2023.tar.gz
# Checksum should be equal to: 11b484f5bd9ddafec5cfb131eb2dd008
tar -xzvf preprocessed_weather_daily_1896_2023.tar.gz
daily_weather_preprocessed_1896_2023.csv
# Option: Validate the checksum
md5sum daily_weather_preprocessed_1896_2023.csv
# Checksum should be equal to: 1132248c78195c43d93f843753881754
```
### Original raw data
The following details are about the steps to download the original raw data to transform and convert as you want.
#### Download
To download the original raw data:
```bash theme={null}
mkdir tw_raw_weather_data && cd tw_raw_weather_data
wget https://storage.googleapis.com/taiwan-weather-observaiton-datasets/raw_data_weather_daily_1896_2023.tar.gz
# Option: Validate the checksum
md5sum raw_data_weather_daily_1896_2023.tar.gz
# Checksum should be equal to: b66b9f137217454d655e3004d7d1b51a
tar -xzvf raw_data_weather_daily_1896_2023.tar.gz
466920_1928.csv
466920_1929.csv
466920_1930.csv
466920_1931.csv
...
# Option: Validate the checksum
cat *.csv | md5sum
# Checksum should be equal to: b26db404bf84d4063fac42e576464ce1
```
#### Retrieve the Taiwan weather stations
```bash theme={null}
wget -O weather_sta_list.csv https://github.com/Raingel/weather_station_list/raw/main/data/weather_sta_list.csv
# Option: Convert the UTF-8-BOM to UTF-8 encoding
sed -i '1s/^\xEF\xBB\xBF//' weather_sta_list.csv
```
## Create table schema
Create the MergeTree table in ClickHouse (from the ClickHouse client).
```bash theme={null}
CREATE TABLE tw_weather_data (
StationId String null,
MeasuredDate DateTime64,
StnPres Float64 null,
SeaPres Float64 null,
Tx Float64 null,
Td Float64 null,
RH Float64 null,
WS Float64 null,
WD Float64 null,
WSGust Float64 null,
WDGust Float64 null,
Precp Float64 null,
PrecpHour Float64 null,
SunShine Float64 null,
GloblRad Float64 null,
TxSoil0cm Float64 null,
TxSoil5cm Float64 null,
TxSoil10cm Float64 null,
TxSoil20cm Float64 null,
TxSoil50cm Float64 null,
TxSoil100cm Float64 null,
TxSoil30cm Float64 null,
TxSoil200cm Float64 null,
TxSoil300cm Float64 null,
TxSoil500cm Float64 null,
VaporPressure Float64 null,
UVI Float64 null,
"Cloud Amount" Float64 null,
EvapA Float64 null,
Visb Float64 null
)
ENGINE = MergeTree
ORDER BY (MeasuredDate);
```
## Inserting into ClickHouse
### Inserting from local file
Data can be inserted from a local file as follows (from the ClickHouse client):
```sql theme={null}
INSERT INTO tw_weather_data FROM INFILE '/path/to/daily_weather_preprocessed_1896_2023.csv'
```
where `/path/to` represents the specific user path to the local file on the disk.
And the sample response output is as follows after inserting data into the ClickHouse:
```response theme={null}
Query id: 90e4b524-6e14-4855-817c-7e6f98fbeabb
Ok.
131985329 rows in set. Elapsed: 71.770 sec. Processed 131.99 million rows, 10.06 GB (1.84 million rows/s., 140.14 MB/s.)
Peak memory usage: 583.23 MiB.
```
### Inserting from URL
```sql theme={null}
INSERT INTO tw_weather_data SELECT *
FROM url('https://storage.googleapis.com/taiwan-weather-observaiton-datasets/daily_weather_preprocessed_1896_2023.csv', 'CSVWithNames')
```
To know how to speed this up, please see our blog post on [tuning large data loads](https://clickhouse.com/blog/supercharge-your-clickhouse-data-loads-part2).
## Check data rows and sizes
1. Let's see how many rows are inserted:
```sql theme={null}
SELECT formatReadableQuantity(count())
FROM tw_weather_data;
```
```response theme={null}
┌─formatReadableQuantity(count())─┐
│ 131.99 million │
└─────────────────────────────────┘
```
2. Let's see how much disk space are used for this table:
```sql theme={null}
SELECT
formatReadableSize(sum(bytes)) AS disk_size,
formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed_size
FROM system.parts
WHERE (`table` = 'tw_weather_data') AND active
```
```response theme={null}
┌─disk_size─┬─uncompressed_size─┐
│ 2.13 GiB │ 32.94 GiB │
└───────────┴───────────────────┘
```
## Sample queries
### Q1: Retrieve the highest dew point temperature for each weather station in the specific year
```sql theme={null}
SELECT
StationId,
max(Td) AS max_td
FROM tw_weather_data
WHERE (year(MeasuredDate) = 2023) AND (Td IS NOT NULL)
GROUP BY StationId
```
```response theme={null}
┌─StationId─┬─max_td─┐
│ 466940 │ 1 │
│ 467300 │ 1 │
│ 467540 │ 1 │
│ 467490 │ 1 │
│ 467080 │ 1 │
│ 466910 │ 1 │
│ 467660 │ 1 │
│ 467270 │ 1 │
│ 467350 │ 1 │
│ 467571 │ 1 │
│ 466920 │ 1 │
│ 467650 │ 1 │
│ 467550 │ 1 │
│ 467480 │ 1 │
│ 467610 │ 1 │
│ 467050 │ 1 │
│ 467590 │ 1 │
│ 466990 │ 1 │
│ 467060 │ 1 │
│ 466950 │ 1 │
│ 467620 │ 1 │
│ 467990 │ 1 │
│ 466930 │ 1 │
│ 467110 │ 1 │
│ 466881 │ 1 │
│ 467410 │ 1 │
│ 467441 │ 1 │
│ 467420 │ 1 │
│ 467530 │ 1 │
│ 466900 │ 1 │
└───────────┴────────┘
30 rows in set. Elapsed: 0.045 sec. Processed 6.41 million rows, 187.33 MB (143.92 million rows/s., 4.21 GB/s.)
```
### Q2: Raw data fetching with the specific duration time range, fields and weather station
```sql theme={null}
SELECT
StnPres,
SeaPres,
Tx,
Td,
RH,
WS,
WD,
WSGust,
WDGust,
Precp,
PrecpHour
FROM tw_weather_data
WHERE (StationId = 'C0UB10') AND (MeasuredDate >= '2023-12-23') AND (MeasuredDate < '2023-12-24')
ORDER BY MeasuredDate ASC
LIMIT 10
```
```response theme={null}
┌─StnPres─┬─SeaPres─┬───Tx─┬───Td─┬─RH─┬──WS─┬──WD─┬─WSGust─┬─WDGust─┬─Precp─┬─PrecpHour─┐
│ 1029.5 │ ᴺᵁᴸᴸ │ 11.8 │ ᴺᵁᴸᴸ │ 78 │ 2.7 │ 271 │ 5.5 │ 275 │ -99.8 │ -99.8 │
│ 1029.8 │ ᴺᵁᴸᴸ │ 12.3 │ ᴺᵁᴸᴸ │ 78 │ 2.7 │ 289 │ 5.5 │ 308 │ -99.8 │ -99.8 │
│ 1028.6 │ ᴺᵁᴸᴸ │ 12.3 │ ᴺᵁᴸᴸ │ 79 │ 2.3 │ 251 │ 6.1 │ 289 │ -99.8 │ -99.8 │
│ 1028.2 │ ᴺᵁᴸᴸ │ 13 │ ᴺᵁᴸᴸ │ 75 │ 4.3 │ 312 │ 7.5 │ 316 │ -99.8 │ -99.8 │
│ 1027.8 │ ᴺᵁᴸᴸ │ 11.1 │ ᴺᵁᴸᴸ │ 89 │ 7.1 │ 310 │ 11.6 │ 322 │ -99.8 │ -99.8 │
│ 1027.8 │ ᴺᵁᴸᴸ │ 11.6 │ ᴺᵁᴸᴸ │ 90 │ 3.1 │ 269 │ 10.7 │ 295 │ -99.8 │ -99.8 │
│ 1027.9 │ ᴺᵁᴸᴸ │ 12.3 │ ᴺᵁᴸᴸ │ 89 │ 4.7 │ 296 │ 8.1 │ 310 │ -99.8 │ -99.8 │
│ 1028.2 │ ᴺᵁᴸᴸ │ 12.2 │ ᴺᵁᴸᴸ │ 94 │ 2.5 │ 246 │ 7.1 │ 283 │ -99.8 │ -99.8 │
│ 1028.4 │ ᴺᵁᴸᴸ │ 12.5 │ ᴺᵁᴸᴸ │ 94 │ 3.1 │ 265 │ 4.8 │ 297 │ -99.8 │ -99.8 │
│ 1028.3 │ ᴺᵁᴸᴸ │ 13.6 │ ᴺᵁᴸᴸ │ 91 │ 1.2 │ 273 │ 4.4 │ 256 │ -99.8 │ -99.8 │
└─────────┴─────────┴──────┴──────┴────┴─────┴─────┴────────┴────────┴───────┴───────────┘
10 rows in set. Elapsed: 0.009 sec. Processed 91.70 thousand rows, 2.33 MB (9.67 million rows/s., 245.31 MB/s.)
```
## Credits
We would like to acknowledge the efforts of the Central Weather Administration and Agricultural Meteorological Observation Network (Station) of the Council of Agriculture for preparing, cleaning, and distributing this dataset. We appreciate your efforts.
Ou, J.-H., Kuo, C.-H., Wu, Y.-F., Lin, G.-C., Lee, M.-H., Chen, R.-K., Chou, H.-P., Wu, H.-Y., Chu, S.-C., Lai, Q.-J., Tsai, Y.-C., Lin, C.-C., Kuo, C.-C., Liao, C.-T., Chen, Y.-N., Chu, Y.-W., Chen, C.-Y., 2023. Application-oriented deep learning model for early warning of rice blast in Taiwan. Ecological Informatics 73, 101950. [https://doi.org/10.1016/j.ecoinf.2022.101950](https://doi.org/10.1016/j.ecoinf.2022.101950) \[13/12/2022]
# Analytical queries on UK property price data
Source: https://clickhouse.com/docs/get-started/sample-datasets/uk-price-paid
Explore ClickHouse through UK property price data
In this tutorial you'll explore ClickHouse using the UK Property Price dataset (Contains HM Land Registry data © Crown copyright and database right 2021. This data is licensed under the Open Government Licence v3.0.) which contains data of prices paid for real-estate property in England and Wales since 1995.
## Prerequisites
For this tutorial, you'll need:
* A [ClickHouse Cloud account](https://clickhouse.cloud/signUp?loc=docs-sample-datasets-uk-property-price) (\$300 in free credits when signing up)
* [A ClickHouse Cloud service](/docs/get-started/setup/cloud#1-create-a-clickhouse-service)
## 1. Create the table
1. Select **SQL console** from the the left hand menu
2. Click the **+** tab next to the home icon to create a new query
3. In the SQL editor type the following query, then click **Run**:
```sql theme={null}
CREATE DATABASE uk;
CREATE TABLE uk.uk_price_paid
(
price UInt32,
date Date,
postcode1 LowCardinality(String),
postcode2 LowCardinality(String),
type Enum8('terraced' = 1, 'semi-detached' = 2, 'detached' = 3, 'flat' = 4, 'other' = 0),
is_new UInt8,
duration Enum8('freehold' = 1, 'leasehold' = 2, 'unknown' = 0),
addr1 String,
addr2 String,
street LowCardinality(String),
locality LowCardinality(String),
town LowCardinality(String),
district LowCardinality(String),
county LowCardinality(String)
)
ENGINE = MergeTree
ORDER BY (postcode1, postcode2, addr1, addr2);
```
> [!TIP]
>
> Take note of `ORDER BY (postcode1, postcode2, addr1, addr2)` which defines how ClickHouse sorts
> data on disk. Choosing an effective primary key in ClickHouse which matches your access patterns is crucial for query performance and storage efficiency.
> See ["Choosing a primary key"](/docs/concepts/best-practices/choosing-a-primary-key) for more details.
See [https://www.gov.uk](https://www.gov.uk/guidance/about-the-price-paid-data) for a description of the fields.
## 2. Preprocess and insert the data
You can use the `url` function to stream data into ClickHouse. Some preprocess is required first.
The query below inserts 25+ million rows into the `uk_price_paid` table and performs the following preprocessing steps:
* splits the `postcode` to two different columns - `postcode1` and `postcode2`, which is better for storage and queries
* converts the `time` field to date as it only contains `00:00` time
* ignores the [UUID](/docs/reference/data-types/uuid) field because it isn't needed for analysis
* transforms `type` and `duration` to more readable `Enum` fields using the [transform](/docs/reference/functions/regular-functions/other-functions#transform) function
* transforms the `is_new` field from a single-character string (`Y`/`N`) to a [UInt8](/docs/reference/data-types/int-uint) field with 0 or 1
* drops the last two columns since they all have the same value (which is 0)
```sql theme={null}
INSERT INTO uk.uk_price_paid
SELECT
toUInt32(price_string) AS price,
parseDateTimeBestEffortUS(time) AS date,
splitByChar(' ', postcode)[1] AS postcode1,
splitByChar(' ', postcode)[2] AS postcode2,
transform(a, ['T', 'S', 'D', 'F', 'O'], ['terraced', 'semi-detached', 'detached', 'flat', 'other']) AS type,
b = 'Y' AS is_new,
transform(c, ['F', 'L', 'U'], ['freehold', 'leasehold', 'unknown']) AS duration,
addr1,
addr2,
street,
locality,
town,
district,
county
FROM url(
'http://prod1.publicdata.landregistry.gov.uk.s3-website-eu-west-1.amazonaws.com/pp-complete.csv',
'CSV',
'uuid_string String,
price_string String,
time String,
postcode String,
a String,
b String,
c String,
addr1 String,
addr2 String,
street String,
locality String,
town String,
district String,
county String,
d String,
e String'
) SETTINGS max_http_get_redirects=10;
```
Wait for the data to insert - it will take a minute or two depending on the network speed.
## 3. Validate the data
Let's verify it worked by seeing how many rows were inserted:
```sql theme={null}
SELECT count()
FROM uk.uk_price_paid
```
At the time this query was run, the dataset had 27,450,499 rows. Let's see what the storage size is of the table in ClickHouse:
```sql theme={null}
SELECT formatReadableSize(total_bytes)
FROM system.tables
WHERE name = 'uk_price_paid'
```
Notice the size of the table is just 221.43 MiB, while the original dataset in uncompressed form is about 4 GiB.
ClickHouse offers excellent data compression out of the box, but also allows you to [further tune compression per column](/docs/reference/statements/create/table#column_compression_codec) if you need.
## 4. Run some queries
With the data loaded, try out the following queries to get a sense of how fast analytical queries return results.
The query below finds the average price per year across all of the data:
```sql theme={null}
SELECT
toYear(date) AS year,
round(avg(price)) AS price,
bar(price, 0, 1000000, 80
)
FROM uk.uk_price_paid
GROUP BY year
ORDER BY year
```
The query below applies a filter to find the average price per year in London:
```sql theme={null}
SELECT
toYear(date) AS year,
round(avg(price)) AS price,
bar(price, 0, 2000000, 100
)
FROM uk.uk_price_paid
WHERE town = 'LONDON'
GROUP BY year
ORDER BY year
```
It looks like something happened to home prices in 2020! But that is probably not a surprise...
The query below finds the most expensive neighborhoods:
```sql theme={null}
SELECT
town,
district,
count() AS c,
round(avg(price)) AS price,
bar(price, 0, 5000000, 100)
FROM uk.uk_price_paid
WHERE date >= '2020-01-01'
GROUP BY
town,
district
HAVING c >= 100
ORDER BY price DESC
LIMIT 100
```
## Next steps
In this tutorial you created a table, preprocessed and loaded UK property price data into ClickHouse, and then
ran some analytical queries on that data.
Next you can:
* Learn how to speed up these queries with projections. See ["Projections"](/docs/concepts/features/projections/projections) for examples which use the same dataset.
* Learn more about ClickHouse [core concepts](/docs/concepts/core-concepts)
* Explore ClickHouse [best practices](/docs/concepts/best-practices)
* Explore other [sample datasets](/docs/get-started/sample-datasets)
# WikiStat
Source: https://clickhouse.com/docs/get-started/sample-datasets/wikistat
Explore the WikiStat dataset containing 0.5 trillion records.
The dataset contains 0.5 trillion records.
See the video from FOSDEM 2023: [https://www.youtube.com/watch?v=JlcI2Vfz\_uk](https://www.youtube.com/watch?v=JlcI2Vfz_uk)
And the presentation: [https://presentations.clickhouse.com/fosdem2023/](https://presentations.clickhouse.com/fosdem2023/)
Data source: [https://dumps.wikimedia.org/other/pageviews/](https://dumps.wikimedia.org/other/pageviews/)
Getting the list of links:
```shell theme={null}
for i in {2015..2023}; do
for j in {01..12}; do
echo "${i}-${j}" >&2
curl -sSL "https://dumps.wikimedia.org/other/pageviews/$i/$i-$j/" \
| grep -oE 'pageviews-[0-9]+-[0-9]+\.gz'
done
done | sort | uniq | tee links.txt
```
Downloading the data:
```shell theme={null}
sed -r 's!pageviews-([0-9]{4})([0-9]{2})[0-9]{2}-[0-9]+\.gz!https://dumps.wikimedia.org/other/pageviews/\1/\1-\2/\0!' \
links.txt | xargs -P3 wget --continue
```
(it will take about 3 days)
Creating a table:
```sql theme={null}
CREATE TABLE wikistat
(
time DateTime CODEC(Delta, ZSTD(3)),
project LowCardinality(String),
subproject LowCardinality(String),
path String CODEC(ZSTD(3)),
hits UInt64 CODEC(ZSTD(3))
)
ENGINE = MergeTree
ORDER BY (path, time);
```
Loading the data:
```shell theme={null}
clickhouse-local --query "
WITH replaceRegexpOne(_path, '^.+pageviews-(\\d{4})(\\d{2})(\\d{2})-(\\d{2})(\\d{2})(\\d{2}).gz$', '\1-\2-\3 \4-\5-\6')::DateTime AS time,
extractGroups(line, '^([^ \\.]+)(\\.[^ ]+)? +([^ ]+) +(\\d+) +(\\d+)$') AS values
SELECT
time,
values[1] AS project,
values[2] AS subproject,
values[3] AS path,
(values[4])::UInt64 AS hits
FROM file('pageviews*.gz', LineAsString)
WHERE length(values) = 5 FORMAT Native
" | clickhouse-client --query "INSERT INTO wikistat FORMAT Native"
```
Or loading the cleaning data:
```sql theme={null}
INSERT INTO wikistat WITH
parseDateTimeBestEffort(extract(_file, '^pageviews-([\\d\\-]+)\\.gz$')) AS time,
splitByChar(' ', line) AS values,
splitByChar('.', values[1]) AS projects
SELECT
time,
projects[1] AS project,
projects[2] AS subproject,
decodeURLComponent(values[2]) AS path,
CAST(values[3], 'UInt64') AS hits
FROM s3(
'https://clickhouse-public-datasets.s3.amazonaws.com/wikistat/original/pageviews*.gz',
LineAsString)
WHERE length(values) >= 3
```
# YouTube dataset of dislikes
Source: https://clickhouse.com/docs/get-started/sample-datasets/youtube-dislikes
A collection of dislikes of YouTube videos.
In November of 2021, YouTube removed the public ***dislike*** count from all of its videos. While creators can still see the number of dislikes, viewers can only see how many ***likes*** a video has received.
> [!WARNING]
>
> The dataset has over 4.55 billion records, so be careful just copying-and-pasting the commands below unless your resources can handle that type of volume. These commands were run on a suitably sized [ClickHouse Cloud](/docs/products/cloud/getting-started/intro) service.
> [!TIP]
>
> To try these commands yourself, you can sign up for a free [ClickHouse Cloud trial](https://clickhouse.cloud/signUp?loc=docs-get-started-sample-datasets-youtube-dislikes) or set up [self-hosted ClickHouse](/docs/get-started/setup/install).
The data is in a JSON format and can be downloaded from [archive.org](https://archive.org/download/dislikes_youtube_2021_12_video_json_files). We have made this same data available in S3 so that it can be downloaded more efficiently into a ClickHouse Cloud instance.
Here are the steps to create a table in ClickHouse Cloud and insert the data.
> [!NOTE]
>
> The steps below work on a local install of ClickHouse too.
## Step-by-step instructions
**1. Data exploration**
Let's see what the data looks like. The `s3` table function returns a table, so we can `DESCRIBE` the result:
```sql theme={null}
DESCRIBE s3(
'https://clickhouse-public-datasets.s3.amazonaws.com/youtube/original/files/*.zst',
'JSONLines'
);
```
ClickHouse infers the following schema from the JSON file:
```response theme={null}
┌─name────────────────┬─type───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐
│ id │ Nullable(String) │ │ │ │ │ │
│ fetch_date │ Nullable(String) │ │ │ │ │ │
│ upload_date │ Nullable(String) │ │ │ │ │ │
│ title │ Nullable(String) │ │ │ │ │ │
│ uploader_id │ Nullable(String) │ │ │ │ │ │
│ uploader │ Nullable(String) │ │ │ │ │ │
│ uploader_sub_count │ Nullable(Int64) │ │ │ │ │ │
│ is_age_limit │ Nullable(Bool) │ │ │ │ │ │
│ view_count │ Nullable(Int64) │ │ │ │ │ │
│ like_count │ Nullable(Int64) │ │ │ │ │ │
│ dislike_count │ Nullable(Int64) │ │ │ │ │ │
│ is_crawlable │ Nullable(Bool) │ │ │ │ │ │
│ is_live_content │ Nullable(Bool) │ │ │ │ │ │
│ has_subtitles │ Nullable(Bool) │ │ │ │ │ │
│ is_ads_enabled │ Nullable(Bool) │ │ │ │ │ │
│ is_comments_enabled │ Nullable(Bool) │ │ │ │ │ │
│ description │ Nullable(String) │ │ │ │ │ │
│ rich_metadata │ Array(Tuple(call Nullable(String), content Nullable(String), subtitle Nullable(String), title Nullable(String), url Nullable(String))) │ │ │ │ │ │
│ super_titles │ Array(Tuple(text Nullable(String), url Nullable(String))) │ │ │ │ │ │
│ uploader_badges │ Nullable(String) │ │ │ │ │ │
│ video_badges │ Nullable(String) │ │ │ │ │ │
└─────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘
```
**2. Create the table**
Based on the inferred schema, we cleaned up the data types and added a primary key.
Define the following table:
```sql theme={null}
CREATE TABLE youtube
(
`id` String,
`fetch_date` DateTime,
`upload_date_str` String,
`upload_date` Date,
`title` String,
`uploader_id` String,
`uploader` String,
`uploader_sub_count` Int64,
`is_age_limit` Bool,
`view_count` Int64,
`like_count` Int64,
`dislike_count` Int64,
`is_crawlable` Bool,
`has_subtitles` Bool,
`is_ads_enabled` Bool,
`is_comments_enabled` Bool,
`description` String,
`rich_metadata` Array(Tuple(call String, content String, subtitle String, title String, url String)),
`super_titles` Array(Tuple(text String, url String)),
`uploader_badges` String,
`video_badges` String
)
ENGINE = MergeTree
ORDER BY (uploader, upload_date)
```
**3. Insert data**
The following command streams the records from the S3 files into the `youtube` table.
> [!WARNING]
>
> This inserts a lot of data - 4.65 billion rows. If you don't want the entire dataset, simply add a `LIMIT` clause with the desired number of rows.
```sql theme={null}
INSERT INTO youtube
SETTINGS input_format_null_as_default = 1
SELECT
id,
parseDateTimeBestEffortUSOrZero(toString(fetch_date)) AS fetch_date,
upload_date AS upload_date_str,
toDate(parseDateTimeBestEffortUSOrZero(upload_date::String)) AS upload_date,
ifNull(title, '') AS title,
uploader_id,
ifNull(uploader, '') AS uploader,
uploader_sub_count,
is_age_limit,
view_count,
like_count,
dislike_count,
is_crawlable,
has_subtitles,
is_ads_enabled,
is_comments_enabled,
ifNull(description, '') AS description,
rich_metadata,
super_titles,
ifNull(uploader_badges, '') AS uploader_badges,
ifNull(video_badges, '') AS video_badges
FROM s3(
'https://clickhouse-public-datasets.s3.amazonaws.com/youtube/original/files/*.zst',
'JSONLines'
)
```
Some comments about our `INSERT` command:
* The `parseDateTimeBestEffortUSOrZero` function is handy when the incoming date fields may not be in the proper format. If `fetch_date` doesn't get parsed properly, it will be set to `0`
* The `upload_date` column contains valid dates, but it also contains strings like "4 hours ago" - which is certainly not a valid date. We decided to store the original value in `upload_date_str` and attempt to parse it with `toDate(parseDateTimeBestEffortUSOrZero(upload_date::String))`. If the parsing fails we just get `0`
* We used `ifNull` to avoid getting `NULL` values in our table. If an incoming value is `NULL`, the `ifNull` function is setting the value to an empty string
**4. Count the number of rows**
Open a new tab in the SQL Console of ClickHouse Cloud (or a new `clickhouse-client` window) and watch the count increase.
It will take a while to insert 4.56B rows, depending on your server resources. (Without any tweaking of settings, it takes about 4.5 hours.)
```sql theme={null}
SELECT formatReadableQuantity(count())
FROM youtube
```
```response theme={null}
┌─formatReadableQuantity(count())─┐
│ 4.56 billion │
└─────────────────────────────────┘
```
**5. Explore the data**
Once the data is inserted, go ahead and count the number of dislikes of your favorite videos or channels. Let's see how many videos were uploaded by ClickHouse:
```sql theme={null}
SELECT count()
FROM youtube
WHERE uploader = 'ClickHouse';
```
```response theme={null}
┌─count()─┐
│ 84 │
└─────────┘
1 row in set. Elapsed: 0.570 sec. Processed 237.57 thousand rows, 5.77 MB (416.54 thousand rows/s., 10.12 MB/s.)
```
> [!NOTE]
>
> The query above runs so quickly because we chose `uploader` as the first column of the primary key - so it only had to process 237k rows.
Let's look and likes and dislikes of ClickHouse videos:
```sql theme={null}
SELECT
title,
like_count,
dislike_count
FROM youtube
WHERE uploader = 'ClickHouse'
ORDER BY dislike_count DESC;
```
The response looks like:
```response theme={null}
┌─title────────────────────────────────────────────────────────────────────────────────────────────────┬─like_count─┬─dislike_count─┐
│ ClickHouse v21.11 Release Webinar │ 52 │ 3 │
│ ClickHouse Introduction │ 97 │ 3 │
│ Casa Modelo Algarve │ 180 │ 3 │
│ Профайлер запросов: трудный путь │ 33 │ 3 │
│ ClickHouse в Курсометре │ 4 │ 2 │
│ 10 Good Reasons to Use ClickHouse │ 27 │ 2 │
...
84 rows in set. Elapsed: 0.013 sec. Processed 155.65 thousand rows, 16.94 MB (11.96 million rows/s., 1.30 GB/s.)
```
Here is a search for videos with **ClickHouse** in the `title` or `description` fields:
```sql theme={null}
SELECT
view_count,
like_count,
dislike_count,
concat('https://youtu.be/', id) AS url,
title
FROM youtube
WHERE (title ILIKE '%ClickHouse%') OR (description ILIKE '%ClickHouse%')
ORDER BY
like_count DESC,
view_count DESC;
```
This query has to process every row, and also parse through two columns of strings. Even then, we get decent performance at 4.15M rows/second:
```response theme={null}
1174 rows in set. Elapsed: 1099.368 sec. Processed 4.56 billion rows, 1.98 TB (4.15 million rows/s., 1.80 GB/s.)
```
The results look like:
```response theme={null}
┌─view_count─┬─like_count─┬─dislike_count─┬─url──────────────────────────┬─title──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1919 │ 63 │ 1 │ https://youtu.be/b9MeoOtAivQ │ ClickHouse v21.10 Release Webinar │
│ 8710 │ 62 │ 4 │ https://youtu.be/PeV1mC2z--M │ What is JDBC DriverManager? | JDBC │
│ 3534 │ 62 │ 1 │ https://youtu.be/8nWRhK9gw10 │ CLICKHOUSE - Arquitetura Modular │
```
## Questions
### If someone disables comments does it lower the chance someone will actually click like or dislike?
When commenting is disabled, are people more likely to like or dislike to express their feelings about a video?
```sql theme={null}
SELECT
concat('< ', formatReadableQuantity(view_range)) AS views,
is_comments_enabled,
total_clicks / num_views AS prob_like_dislike
FROM
(
SELECT
is_comments_enabled,
power(10, CEILING(log10(view_count + 1))) AS view_range,
sum(like_count + dislike_count) AS total_clicks,
sum(view_count) AS num_views
FROM youtube
GROUP BY
view_range,
is_comments_enabled
) WHERE view_range > 1
ORDER BY
is_comments_enabled ASC,
num_views ASC;
```
```response theme={null}
┌─views─────────────┬─is_comments_enabled─┬────prob_like_dislike─┐
│ < 10.00 │ false │ 0.08224180712685371 │
│ < 100.00 │ false │ 0.06346337759167248 │
│ < 1.00 thousand │ false │ 0.03201883652987105 │
│ < 10.00 thousand │ false │ 0.01716073540410903 │
│ < 10.00 billion │ false │ 0.004555639481829971 │
│ < 100.00 thousand │ false │ 0.01293351460515323 │
│ < 1.00 billion │ false │ 0.004761811192464957 │
│ < 1.00 million │ false │ 0.010472604018980551 │
│ < 10.00 million │ false │ 0.00788902538420125 │
│ < 100.00 million │ false │ 0.00579152804250582 │
│ < 10.00 │ true │ 0.09819517478134059 │
│ < 100.00 │ true │ 0.07403784478585775 │
│ < 1.00 thousand │ true │ 0.03846294910067627 │
│ < 10.00 billion │ true │ 0.005615217329358215 │
│ < 10.00 thousand │ true │ 0.02505881391701455 │
│ < 1.00 billion │ true │ 0.007434998802482997 │
│ < 100.00 thousand │ true │ 0.022694648130822004 │
│ < 100.00 million │ true │ 0.011761563746575625 │
│ < 1.00 million │ true │ 0.020776022304589435 │
│ < 10.00 million │ true │ 0.016917095718089584 │
└───────────────────┴─────────────────────┴──────────────────────┘
22 rows in set. Elapsed: 8.460 sec. Processed 4.56 billion rows, 77.48 GB (538.73 million rows/s., 9.16 GB/s.)
```
Enabling comments seems to be correlated with a higher rate of engagement.
### How does the number of videos change over time - notable events?
```sql theme={null}
SELECT
toStartOfMonth(toDateTime(upload_date)) AS month,
uniq(uploader_id) AS uploaders,
count() AS num_videos,
sum(view_count) AS view_count
FROM youtube
GROUP BY month
ORDER BY month ASC;
```
```response theme={null}
┌──────month─┬─uploaders─┬─num_videos─┬───view_count─┐
│ 2005-04-01 │ 5 │ 6 │ 213597737 │
│ 2005-05-01 │ 6 │ 9 │ 2944005 │
│ 2005-06-01 │ 165 │ 351 │ 18624981 │
│ 2005-07-01 │ 395 │ 1168 │ 94164872 │
│ 2005-08-01 │ 1171 │ 3128 │ 124540774 │
│ 2005-09-01 │ 2418 │ 5206 │ 475536249 │
│ 2005-10-01 │ 6750 │ 13747 │ 737593613 │
│ 2005-11-01 │ 13706 │ 28078 │ 1896116976 │
│ 2005-12-01 │ 24756 │ 49885 │ 2478418930 │
│ 2006-01-01 │ 49992 │ 100447 │ 4532656581 │
│ 2006-02-01 │ 67882 │ 138485 │ 5677516317 │
│ 2006-03-01 │ 103358 │ 212237 │ 8430301366 │
│ 2006-04-01 │ 114615 │ 234174 │ 9980760440 │
│ 2006-05-01 │ 152682 │ 332076 │ 14129117212 │
│ 2006-06-01 │ 193962 │ 429538 │ 17014143263 │
│ 2006-07-01 │ 234401 │ 530311 │ 18721143410 │
│ 2006-08-01 │ 281280 │ 614128 │ 20473502342 │
│ 2006-09-01 │ 312434 │ 679906 │ 23158422265 │
│ 2006-10-01 │ 404873 │ 897590 │ 27357846117 │
```
A spike of uploaders [around covid is noticeable](https://www.theverge.com/2020/3/27/21197642/youtube-with-me-style-videos-views-coronavirus-cook-workout-study-home-beauty).
### More subtitles over time and when
With advances in speech recognition, it's easier than ever to create subtitles for video with youtube adding auto-captioning in late 2009 - was the jump then?
```sql theme={null}
SELECT
toStartOfMonth(upload_date) AS month,
countIf(has_subtitles) / count() AS percent_subtitles,
percent_subtitles - any(percent_subtitles) OVER (
ORDER BY month ASC ROWS BETWEEN 1 PRECEDING AND 1 PRECEDING
) AS previous
FROM youtube
GROUP BY month
ORDER BY month ASC;
```
```response theme={null}
┌──────month─┬───percent_subtitles─┬────────────────previous─┐
│ 2015-01-01 │ 0.2652653881082824 │ 0.2652653881082824 │
│ 2015-02-01 │ 0.3147556050309162 │ 0.049490216922633834 │
│ 2015-03-01 │ 0.32460464492371877 │ 0.009849039892802558 │
│ 2015-04-01 │ 0.33471963051468445 │ 0.010114985590965686 │
│ 2015-05-01 │ 0.3168087575501062 │ -0.017910872964578273 │
│ 2015-06-01 │ 0.3162609788438222 │ -0.0005477787062839745 │
│ 2015-07-01 │ 0.31828767677518033 │ 0.0020266979313581235 │
│ 2015-08-01 │ 0.3045551564286859 │ -0.013732520346494415 │
│ 2015-09-01 │ 0.311221133995152 │ 0.006665977566466086 │
│ 2015-10-01 │ 0.30574870926812175 │ -0.005472424727030245 │
│ 2015-11-01 │ 0.31125409712077234 │ 0.0055053878526505895 │
│ 2015-12-01 │ 0.3190967954651779 │ 0.007842698344405541 │
│ 2016-01-01 │ 0.32636021432496176 │ 0.007263418859783877 │
```
The data results show a spike in 2009. Apparently at that, time YouTube was removing their community captions feature, which allowed you to upload captions for other people's video.
This prompted a very successful campaign to have creators add captions to their videos for hard of hearing and deaf viewers.
### Top uploaders over time
```sql theme={null}
WITH uploaders AS
(
SELECT uploader
FROM youtube
GROUP BY uploader
ORDER BY sum(view_count) DESC
LIMIT 10
)
SELECT
month,
uploader,
sum(view_count) AS total_views,
avg(dislike_count / like_count) AS like_to_dislike_ratio
FROM youtube
WHERE uploader IN (uploaders)
GROUP BY
toStartOfMonth(upload_date) AS month,
uploader
ORDER BY
month ASC,
total_views DESC;
```
```response theme={null}
┌──────month─┬─uploader───────────────────┬─total_views─┬─like_to_dislike_ratio─┐
│ 1970-01-01 │ T-Series │ 10957099 │ 0.022784656361208206 │
│ 1970-01-01 │ Ryan's World │ 0 │ 0.003035559410234172 │
│ 1970-01-01 │ SET India │ 0 │ nan │
│ 2006-09-01 │ Cocomelon - Nursery Rhymes │ 256406497 │ 0.7005566715978622 │
│ 2007-06-01 │ Cocomelon - Nursery Rhymes │ 33641320 │ 0.7088650914344298 │
│ 2008-02-01 │ WWE │ 43733469 │ 0.07198856488734842 │
│ 2008-03-01 │ WWE │ 16514541 │ 0.1230603715431997 │
│ 2008-04-01 │ WWE │ 5907295 │ 0.2089399470159618 │
│ 2008-05-01 │ WWE │ 7779627 │ 0.09101676560436774 │
│ 2008-06-01 │ WWE │ 7018780 │ 0.0974184753155297 │
│ 2008-07-01 │ WWE │ 4686447 │ 0.1263845422065158 │
│ 2008-08-01 │ WWE │ 4514312 │ 0.08384574274791441 │
│ 2008-09-01 │ WWE │ 3717092 │ 0.07872802579349912 │
```
### How do like ratio changes as views go up?
```sql theme={null}
SELECT
concat('< ', formatReadableQuantity(view_range)) AS view_range,
is_comments_enabled,
round(like_ratio, 2) AS like_ratio
FROM
(
SELECT
power(10, CEILING(log10(view_count + 1))) AS view_range,
is_comments_enabled,
avg(like_count / dislike_count) AS like_ratio
FROM youtube WHERE dislike_count > 0
GROUP BY
view_range,
is_comments_enabled HAVING view_range > 1
ORDER BY
view_range ASC,
is_comments_enabled ASC
);
```
```response theme={null}
┌─view_range────────┬─is_comments_enabled─┬─like_ratio─┐
│ < 10.00 │ false │ 0.66 │
│ < 10.00 │ true │ 0.66 │
│ < 100.00 │ false │ 3 │
│ < 100.00 │ true │ 3.95 │
│ < 1.00 thousand │ false │ 8.45 │
│ < 1.00 thousand │ true │ 13.07 │
│ < 10.00 thousand │ false │ 18.57 │
│ < 10.00 thousand │ true │ 30.92 │
│ < 100.00 thousand │ false │ 23.55 │
│ < 100.00 thousand │ true │ 42.13 │
│ < 1.00 million │ false │ 19.23 │
│ < 1.00 million │ true │ 37.86 │
│ < 10.00 million │ false │ 12.13 │
│ < 10.00 million │ true │ 30.72 │
│ < 100.00 million │ false │ 6.67 │
│ < 100.00 million │ true │ 23.32 │
│ < 1.00 billion │ false │ 3.08 │
│ < 1.00 billion │ true │ 20.69 │
│ < 10.00 billion │ false │ 1.77 │
│ < 10.00 billion │ true │ 19.5 │
└───────────────────┴─────────────────────┴────────────┘
```
### How are views distributed?
```sql theme={null}
SELECT
labels AS percentile,
round(quantiles) AS views
FROM
(
SELECT
quantiles(0.999, 0.99, 0.95, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1)(view_count) AS quantiles,
['99.9th', '99th', '95th', '90th', '80th', '70th','60th', '50th', '40th', '30th', '20th', '10th'] AS labels
FROM youtube
)
ARRAY JOIN
quantiles,
labels;
```
```response theme={null}
┌─percentile─┬───views─┐
│ 99.9th │ 1216624 │
│ 99th │ 143519 │
│ 95th │ 13542 │
│ 90th │ 4054 │
│ 80th │ 950 │
│ 70th │ 363 │
│ 60th │ 177 │
│ 50th │ 97 │
│ 40th │ 57 │
│ 30th │ 32 │
│ 20th │ 16 │
│ 10th │ 6 │
└────────────┴─────────┘
```
# ClickHouse Cloud quick start
Source: https://clickhouse.com/docs/get-started/setup/cloud
Quick start guide for ClickHouse Cloud
This page covers provisioning a ClickHouse Cloud service, connecting to it, and loading data, all from the command line with the [ClickHouse CLI](/docs/concepts/features/interfaces/cli) (`clickhousectl`). Apart from the optional `--interactive` login, commands are non-interactive. `clickhousectl` emits JSON with `--json`, and automatically when a coding agent runs it.
## Prerequisites
Install the ClickHouse CLI:
```bash theme={null}
curl -fsSL https://clickhouse.com/cli | sh
```
The script installs `~/.local/bin/clickhousectl` and a `chctl` alias. Make sure `~/.local/bin` is on your `PATH`.
You also need `jq`.
You need a ClickHouse Cloud account. If you don't have one yet, `clickhousectl cloud auth signup` opens the sign-up page in your browser.
Browser login (`clickhousectl cloud auth login`) is read-only. Creating and deleting a service, and running SQL with the Query API, need an [API key](/docs/products/cloud/features/admin-features/api/openapi) whose role can create services and manage Query API endpoints; each command's `--help` lists the permissions it needs. Log in with the key; `--interactive` prompts for the key and secret, which keeps the secret out of your shell history:
```bash theme={null}
clickhousectl cloud auth login --interactive
```
Alternatively, pass `--api-key` and `--api-secret`, or set the `CLICKHOUSE_CLOUD_API_KEY` and `CLICKHOUSE_CLOUD_API_SECRET` environment variables. A coding agent or script should use the environment variables. Login verifies the key and prints its identity; check the active credentials any time with `clickhousectl cloud auth whoami`. If you belong to more than one organization, add `--org-id ` (or `--org-name `) to each `cloud` command in this guide.
## Create a ClickHouse service
Create the service and save the response; the password for the `default` user is shown only once:
```bash theme={null}
clickhousectl cloud service create \
--name quickstart-ch \
--region us-east-1 \
--json > ch.json
```
The response includes the service ID, endpoints, and the generated password (trimmed here; the full response also contains scaling settings, the IP access list, and tags):
```json theme={null}
{
"password": "dK7mPq2x_-TzrL9vNw0s",
"service": {
"id": "4f7b92f3-4163-403a-b538-b9bc6e2e8f66",
"name": "quickstart-ch",
"provider": "aws",
"region": "us-east-1",
"state": "provisioning",
"endpoints": [
{
"host": "quickstart-abc123.us-east-1.aws.clickhouse.cloud",
"port": 9440,
"protocol": "nativesecure"
},
{
"host": "quickstart-abc123.us-east-1.aws.clickhouse.cloud",
"port": 8443,
"protocol": "https"
}
],
"numReplicas": 3,
"minReplicaMemoryGb": 16.0,
"maxReplicaMemoryGb": 120.0
}
}
```
Extract what the rest of this guide needs:
```bash theme={null}
CH_ID=$(jq -r .service.id ch.json)
CH_PASSWORD=$(jq -r .password ch.json)
CH_HOST=$(jq -r '.service.endpoints[] | select(.protocol=="nativesecure") | .host' ch.json)
```
If the password is lost, generate a new one with `clickhousectl cloud service reset-password "$CH_ID"`.
Services created with `clickhousectl` default to an IP access list that allows all (`0.0.0.0/0`). To restrict access, pass `--ip-allow` when creating the service; see ["Setting IP filters"](/docs/products/cloud/guides/security/connectivity/setting-ip-filters).
## Wait for the service to provision
Provisioning takes about a minute. Poll until the state is `running`:
```bash theme={null}
while [ "$(clickhousectl cloud service get "$CH_ID" --json | jq -r .state)" != "running" ]; do
sleep 15
done
```
## Run SQL with the Query API
`clickhousectl cloud service query` runs SQL over HTTP — no local `clickhouse` binary or service password required. On the first call, clickhousectl adds your API key to the service's Query API endpoint, creating the endpoint if needed; no new key is created. Pass `--no-auto-enable` to fail instead:
```bash theme={null}
clickhousectl cloud service query "$CH_ID" --json --query "SHOW databases"
```
The first line of the output below goes to stderr, and appears only when the CLI changes the endpoint:
```text theme={null}
Binding the authenticated API key to the Query API endpoint for service 'quickstart-ch'...
{"name":"INFORMATION_SCHEMA"}
{"name":"default"}
{"name":"information_schema"}
{"name":"system"}
```
Output defaults to `PrettyCompact` on a terminal, `TabSeparated` when piped, and `JSONEachRow` when a coding agent runs the command. `--json`, used above, selects `JSONEachRow`; for any other format, pass `--format` instead (`--json` and `--format` cannot be combined). A script that parses the output should always pass one of them.
## Create a database and table
```bash theme={null}
clickhousectl cloud service query "$CH_ID" \
--query "CREATE DATABASE IF NOT EXISTS helloworld"
clickhousectl cloud service query "$CH_ID" \
--query "CREATE TABLE helloworld.my_first_table (
user_id UInt32,
message String,
timestamp DateTime,
metric Float32
) ENGINE = MergeTree()
PRIMARY KEY (user_id, timestamp)"
```
Each command prints `OK` on stderr. Insert a few rows:
```bash theme={null}
clickhousectl cloud service query "$CH_ID" <<'SQL'
INSERT INTO helloworld.my_first_table (user_id, message, timestamp, metric) VALUES
(101, 'Hello, ClickHouse!', now(), -1.0),
(102, 'Insert a lot of rows per batch', yesterday(), 1.41421),
(102, 'Sort your data based on your commonly-used queries', today(), 2.718),
(101, 'Granules are the smallest chunks of data read', now() + 5, 3.14159)
SQL
```
Verify it worked:
```bash theme={null}
clickhousectl cloud service query "$CH_ID" --json \
--query "SELECT * FROM helloworld.my_first_table ORDER BY timestamp"
```
```text theme={null}
{"user_id":102,"message":"Insert a lot of rows per batch","timestamp":"2026-08-26 00:00:00","metric":1.41421}
{"user_id":102,"message":"Sort your data based on your commonly-used queries","timestamp":"2026-08-27 00:00:00","metric":2.718}
{"user_id":101,"message":"Hello, ClickHouse!","timestamp":"2026-08-27 10:41:28","metric":-1}
{"user_id":101,"message":"Granules are the smallest chunks of data read","timestamp":"2026-08-27 10:41:33","metric":3.14159}
```
The timestamps depend on when you ran the insert, so yours will differ.
## Load a CSV file
Suppose the following text is in a CSV file named `data.csv`:
```text title="data.csv" theme={null}
102,This is data in a file,2022-02-22 10:43:28,123.45
101,It is comma-separated,2022-02-23 00:00:00,456.78
103,Use FORMAT to specify the format,2022-02-21 10:43:30,678.90
```
`INSERT ... FORMAT` reads the data from stdin, so pipe the query and the file together:
```bash theme={null}
printf 'INSERT INTO helloworld.my_first_table FORMAT CSV\n' | cat - data.csv \
| clickhousectl cloud service query "$CH_ID"
```
The Query API stops waiting after about 30 seconds, and a timed-out `INSERT` may still have run, so check the row count before you retry. For large files, use [`clickhouse client`](#native-client) instead.
Verify the new rows landed:
```bash theme={null}
clickhousectl cloud service query "$CH_ID" --json \
--query "SELECT count() FROM helloworld.my_first_table"
```
```text theme={null}
{"count()":7}
```
## Connect with clickhouse client
You can also connect over the native protocol with [**clickhouse client**](/docs/concepts/features/interfaces/client). The ClickHouse CLI manages the `clickhouse` binary for you, so you don't need a separate client install:
```bash theme={null}
clickhousectl local use latest
```
This installs the latest version of ClickHouse and symlinks it to `~/.local/bin/clickhouse`, so you can run the plain `clickhouse` command once `~/.local/bin` is on your `PATH` (the command prints a note if it isn't).
Then connect using the hostname and password from the create response. With `--query`, the result is printed and the client exits; without it you get the interactive prompt (`:)`), which you leave with `exit`:
```bash theme={null}
clickhouse client --host "$CH_HOST" --secure --port 9440 \
--user default --password "$CH_PASSWORD" \
--query "SELECT * FROM helloworld.my_first_table ORDER BY timestamp FORMAT TabSeparated"
```
```text theme={null}
103 Use FORMAT to specify the format 2022-02-21 10:43:30 678.9
102 This is data in a file 2022-02-22 10:43:28 123.45
101 It is comma-separated 2022-02-23 00:00:00 456.78
102 Insert a lot of rows per batch 2026-08-26 00:00:00 1.41421
102 Sort your data based on your commonly-used queries 2026-08-27 00:00:00 2.718
101 Hello, ClickHouse! 2026-08-27 10:41:28 -1
101 Granules are the smallest chunks of data read 2026-08-27 10:41:33 3.14159
```
The same command form uploads files. It has no Query API timeout, so use it for large files:
```bash theme={null}
clickhouse client --host "$CH_HOST" --secure --port 9440 \
--user default --password "$CH_PASSWORD" \
--query='INSERT INTO helloworld.my_first_table FORMAT CSV' < data.csv
```
ClickHouse Cloud deduplicates identical inserted blocks, so loading the same `data.csv` a second time adds no rows.
## Cleanup
Deleting a service removes all of its data permanently. `--force` stops a running service first and waits for the stop to finish, which can take minutes:
```bash theme={null}
clickhousectl cloud service delete "$CH_ID" --force
```
To keep the data but stop paying for compute, stop the service with `clickhousectl cloud service stop "$CH_ID"` instead.
# Installation instructions
Source: https://clickhouse.com/docs/get-started/setup/install
Install ClickHouse
Recommended
ClickHouse Cloud
Start building instantly without having to size and scale your cluster. ClickHouse Cloud takes care of sizing,
scaling, security, reliability and upgrades.
[Get started free](https://clickhouse.cloud/signUp?loc=docs-install-page-banner)
[Read the quickstart tutorial](/docs/get-started/setup/cloud)
If you prefer to self-manage, select one of the open-source ClickHouse options below.
Use the quick install script or the `clickhousectl` CLI for a local setup, or install
ClickHouse Server from packages or binaries for a production deployment.
### Local development
Run ClickHouse on your own machine to build, test, and experiment.
Quick install downloads a single binary (and installs clickhousectl),
or use clickhousectl to install and manage local versions and servers.
> [card] [Quick install](/docs/get-started/setup/self-managed/quick-install)
> [card] [clickhousectl (CLI)](/docs/get-started/setup/self-managed/clickhousectl)
> [card] [Docker](/docs/get-started/setup/self-managed/docker)
> [card] [Source build](/docs/get-started/setup/self-managed/advanced)
### Production server
Install ClickHouse Server from packages or binaries for a
long-running deployment on your own infrastructure.
> [card] [Debian/Ubuntu](/docs/get-started/setup/self-managed/debian-ubuntu)
> [card] [Redhat](/docs/get-started/setup/self-managed/redhat)
> [card] [NixOS](/docs/get-started/setup/self-managed/nixos)
> [card] [Other Linux](/docs/get-started/setup/self-managed/other-linux)
# Source & CI builds
Source: https://clickhouse.com/docs/get-started/setup/self-managed/advanced
Instructions for compiling ClickHouse from source or installing a CI-generated binary
## Compile from source
To manually compile ClickHouse, follow the instructions for [Linux](/docs/resources/develop-contribute/build/build) or [macOS](/docs/resources/develop-contribute/build/build-osx).
You can compile packages and install them or use programs without installing packages.
```text theme={null}
Client: /programs/clickhouse-client
Server: /programs/clickhouse-server
```
You'll need to create data and metadata folders manually and `chown` them for the desired user. Their paths can be changed in server config (src/programs/server/config.xml), by default they're:
```bash theme={null}
/var/lib/clickhouse/data/default/
/var/lib/clickhouse/metadata/default/
```
On Gentoo, you can just use `emerge clickhouse` to install ClickHouse from sources.
## Install a CI-generated Binary
ClickHouse's continuous integration (CI) infrastructure produces specialized builds for each commit in the [ClickHouse
repository](https://github.com/clickhouse/clickhouse/), e.g. [sanitized](https://github.com/google/sanitizers) builds, unoptimized (Debug)
builds, cross-compiled builds etc. While such builds are normally only useful during development, they can in certain situations also be
interesting for you.
> [!NOTE]
>
> Since ClickHouse's CI is evolving over time, the exact steps to download CI-generated builds may vary.
> Also, CI may delete old build artifacts, making them unavailable for download.
For example, to download an aarch64 binary for ClickHouse v23.4, follow these steps:
* Find the GitHub pull request for release v23.4: [Release pull request for branch 23.4](https://github.com/ClickHouse/ClickHouse/pull/49238)
* Click "Commits", then click on a commit similar to "Update autogenerated version to 23.4.2.1 and contributors" for the particular version you'd like to install.
* Click the green check / yellow dot / red cross to open the list of CI checks.
* Click "Details" next to "Builds" in the list; it will open the CI build report page.
* Find the rows with compiler = "clang-\*-aarch64" — there are multiple rows.
* Download the artifacts for these builds.
# Install with the ClickHouse CLI
Source: https://clickhouse.com/docs/get-started/setup/self-managed/clickhousectl
Install and manage local ClickHouse versions using clickhousectl, the ClickHouse CLI
> [!NOTE]
>
> **Windows users**
>
> ClickHouse runs natively on Linux and macOS. On Windows, run these steps inside
> the [Windows Subsystem for Linux (WSL)](https://learn.microsoft.com/en-us/windows/wsl/about).
The ClickHouse CLI (`clickhousectl`, alias `chctl`) installs and manages local
ClickHouse versions, runs local ClickHouse and Postgres servers, runs queries,
and manages ClickHouse Cloud. This page walks you through installing the CLI,
starting a local ClickHouse server, and connecting to it.
If you used the [quick install](/docs/get-started/setup/self-managed/quick-install)
script, `clickhousectl` is already installed. Skip to
[Install ClickHouse](#cli-install-clickhouse).
**1. Install the ClickHouse CLI**
The install script supports macOS and Linux on x86\_64 and ARM64:
```bash theme={null}
curl -fsSL https://clickhouse.com/cli | sh
```
The script installs `clickhousectl` to `~/.local/bin` and creates a `chctl`
alias next to it. If the script says `~/.local/bin` is not on your `PATH`, add it,
and add the same line to your shell profile (`~/.bashrc`, `~/.zshrc`) for future
sessions:
```bash theme={null}
export PATH="$HOME/.local/bin:$PATH"
```
> [accordion] Other installation methods
>
> | Tool | Command |
> | - | - |
> | npm | `npm install -g clickhousectl` |
> | uv | `uv tool install clickhousectl` |
> | pipx | `pipx install clickhousectl` |
> | pip | `pip install clickhousectl` |
> | cargo-binstall | `cargo binstall clickhousectl` |
> | Cargo, from source (Rust 1.94+) | `cargo install clickhousectl` |
>
> Only the install script and npm create the `chctl` alias.
To check for a newer CLI release, and to install it:
```bash theme={null}
clickhousectl update --check
clickhousectl update
```
**2. Install ClickHouse**
Install the latest version of ClickHouse and make it your default:
```bash theme={null}
clickhousectl local use latest
```
`local use` installs the version if it isn't already present, sets it as your
default, and creates a `clickhouse` symlink in `~/.local/bin`, so you can run the
`clickhouse` binary directly. Any later step in these docs that runs a
`clickhouse` command then works as-is, provided `~/.local/bin` is on your `PATH`.
You can also select the latest long-term support release, or a specific
version:
```bash theme={null}
clickhousectl local use lts # Latest LTS release
clickhousectl local use 26.8 # Latest 26.8.x.x release
clickhousectl local use 26.8.1.1760 # Exact version
```
To see which minor versions are available to download, and which versions you
have installed:
```bash theme={null}
clickhousectl local list --remote
clickhousectl local list
```
> [!NOTE]
>
> **Use vs install**
>
> `clickhousectl local use ` installs a version *and* makes it your
> default, updating the `clickhouse` symlink. `clickhousectl local install `
> only downloads a version; it becomes your default only if you have no default
> yet, and it doesn't update the symlink. To set the default without creating or
> updating the symlink, use `clickhousectl local use --no-global`.
**3. Start clickhouse-server**
`clickhousectl` scopes local servers to the directory you run it from. Create a
project directory and start a server there:
```bash theme={null}
mkdir my-project && cd my-project
clickhousectl local server start
```
```response theme={null}
Server 'default' started in background (PID: 14705)
HTTP port: 8123
TCP port: 9000
Version: 26.9.1.1312
```
The server runs in the background and is named `default`. Its data lives in
`./.clickhouse/`, so run later `clickhousectl local` commands from the same
directory; `clickhousectl` doesn't search parent directories. The server listens
on HTTP port `8123` and TCP port `9000`, or on free ports when those are taken.
To choose the ports, pass `--http-port` and `--tcp-port`. To run more than one
server, give each a name, such as `clickhousectl local server start dev`.
To verify it's running:
```bash theme={null}
clickhousectl local server list
```
```response theme={null}
| Name | Status | PID | Version | HTTP Port | TCP Port |
|---------|---------|-------|-------------|-----------|----------|
| default | running | 14705 | 26.9.1.1312 | 8123 | 9000 |
1 server, 1 running
```
**4. Start clickhouse-client**
Connect to the server with `clickhouse-client`, from the same directory:
```bash theme={null}
clickhousectl local client
```
`local client` connects to the server named `default` and uses the same
ClickHouse version as the server. Pass a name to connect to another server,
such as `clickhousectl local client dev`. You will see something like this:
```response theme={null}
ClickHouse client version 26.9.1.1312 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 26.9.1.
local-host :)
```
You're now ready to start sending SQL commands to ClickHouse! Type `exit` to
leave the client.
To run a single query without opening an interactive session, pass `-q`:
```bash theme={null}
clickhousectl local client -q "SELECT version()"
```
> [!TIP]
>
> [Creating tables in ClickHouse](/docs/get-started/quickstarts/creating-tables) walks you through creating tables and inserting data.
**5. Stop the server**
The server keeps running in the background until you stop it. Stop it from the
same directory:
```bash theme={null}
clickhousectl local server stop
```
Stopping preserves the server's data for the next `clickhousectl local server start`.
> [!WARNING]
>
> `clickhousectl local server remove` permanently deletes a stopped server's data.
## Next steps
* **Run Postgres locally.** With Docker running, start a local Postgres server
and connect to it with `psql`:
```bash theme={null}
clickhousectl local postgres start
clickhousectl local postgres client
```
* **Connect to ClickHouse Cloud.** Create an account, or log in to an existing
one. Browser login is read-only; creating or changing Cloud resources needs an
API key, which `clickhousectl cloud auth login --interactive` prompts for.
```bash theme={null}
clickhousectl cloud auth signup
clickhousectl cloud auth login
```
* **Learn the rest of the CLI.** See the [clickhousectl reference](/docs/concepts/features/interfaces/cli),
or run `clickhousectl --help`.
# Install ClickHouse on Debian/Ubuntu
Source: https://clickhouse.com/docs/get-started/setup/self-managed/debian-ubuntu
Install ClickHouse on Debian/Ubuntu Linux
> It is recommended to use official pre-compiled `deb` packages for **Debian** or **Ubuntu**.
**1. Review recommendations**
Before installing ClickHouse, review the following recommendations:
* **Swap:** Disable the operating system's swap file in production environments.
* **Disk space:** The ClickHouse binary requires at least 2.5 GB of disk space for installation.
* **Network:** For distributed deployments (clustering), use at least 10 Gbit network connectivity. Network bandwidth is critical for processing distributed queries with large amounts of intermediate data, as well as for replication.
**Estimating storage requirements**
To estimate the disk space needed for your data:
1. **Estimate data volume:** Take a sample of your data and calculate the average row size, then multiply by the number of rows you plan to store.
2. **Apply the compression coefficient:** Load a sample into ClickHouse and compare the original data size with the stored table size. Clickstream data, for example, is typically compressed 6-10x.
3. **Account for replicas:** If you plan to store data in multiple replicas, multiply the estimated volume by the number of replicas.
For more detailed hardware requirements, see ["Sizing and hardware recommendations"](/docs/guides/oss/best-practices/sizing-and-hardware-recommendations)
**2. Set up the Debian repository**
To install ClickHouse run the following commands:
```bash theme={null}
# Install prerequisite packages
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
# Download the ClickHouse GPG key and store it in the keyring
curl -fsSL 'https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key' | sudo gpg --dearmor -o /usr/share/keyrings/clickhouse-keyring.gpg
# Get the system architecture
ARCH=$(dpkg --print-architecture)
# Add the ClickHouse repository to apt sources
echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg arch=${ARCH}] https://packages.clickhouse.com/deb stable main" | sudo tee /etc/apt/sources.list.d/clickhouse.list
# Update apt package lists
sudo apt-get update
```
* You can replace `stable` with `lts` to use different [release kinds](/docs/resources/support-center/knowledge-base/setup-installation/production) based on your needs.
* You can download and install packages manually from [packages.clickhouse.com](https://packages.clickhouse.com/deb/pool/main/c/).
> [accordion] Old distributions method for installing the deb-packages
>
> ```bash theme={null}
> # Install prerequisite packages
> sudo apt-get install apt-transport-https ca-certificates dirmngr
>
> # Add the ClickHouse GPG key to authenticate packages
> sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8919F6BD2B48D754
>
> # Add the ClickHouse repository to apt sources
> echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee \
> /etc/apt/sources.list.d/clickhouse.list
>
> # Update apt package lists
> sudo apt-get update
>
> # Install ClickHouse server and client packages
> sudo apt-get install -y clickhouse-server clickhouse-client
>
> # Start the ClickHouse server service
> sudo service clickhouse-server start
>
> # Launch the ClickHouse command line client
> clickhouse-client # or "clickhouse-client --password" if you set up a password.
> ```
**3. Install ClickHouse server and client**
```bash theme={null}
sudo apt-get install -y clickhouse-server clickhouse-client
```
**4. Start ClickHouse**
To start the ClickHouse server, run:
```bash theme={null}
sudo service clickhouse-server start
```
To start ClickHouse client, run:
```bash theme={null}
clickhouse-client
```
If you set up a password for your server, then you will need to run:
```bash theme={null}
clickhouse-client --password
```
**5. Install standalone ClickHouse Keeper**
> [!TIP]
>
> In production environments we strongly recommend running ClickHouse Keeper on dedicated nodes.
> In test environments, if you decide to run ClickHouse Server and ClickHouse Keeper on the same server,
> then you don't need to install ClickHouse Keeper as it is included with ClickHouse server.
To install `clickhouse-keeper` on standalone ClickHouse Keeper servers, run:
```bash theme={null}
sudo apt-get install -y clickhouse-keeper
```
**6. Enable and start ClickHouse Keeper**
```bash theme={null}
sudo systemctl enable clickhouse-keeper
sudo systemctl start clickhouse-keeper
sudo systemctl status clickhouse-keeper
```
## Packages
The various deb packages available are detailed below:
| Package | Description |
| - | - |
| `clickhouse-common-static` | Installs ClickHouse compiled binary files. |
| `clickhouse-server` | Creates a symbolic link for `clickhouse-server` and installs the default server configuration. |
| `clickhouse-client` | Creates a symbolic link for `clickhouse-client` and other client-related tools, and installs client configuration files. |
| `clickhouse-common-static-dbg` | Installs ClickHouse compiled binary files with debug info. |
| `clickhouse-keeper` | Used to install ClickHouse Keeper on dedicated ClickHouse Keeper nodes. If you're running ClickHouse Keeper on the same server as ClickHouse server, then you don't need to install this package. Installs ClickHouse Keeper and the default ClickHouse Keeper configuration files. |
> [!NOTE]
>
> If you need to install a specific version of ClickHouse, you have to install all packages with the same version:
> `sudo apt-get install clickhouse-server=21.8.5.7 clickhouse-client=21.8.5.7 clickhouse-common-static=21.8.5.7`
# Install ClickHouse using Docker
Source: https://clickhouse.com/docs/get-started/setup/self-managed/docker
Install ClickHouse on Debian/Ubuntu Linux
The guide on [Docker Hub](https://hub.docker.com/r/clickhouse/clickhouse-server/)
is reproduced below for convenience. The Docker images available make use of
the official ClickHouse deb packages.
Docker pull command:
```bash theme={null}
docker pull clickhouse/clickhouse-server
```
## Versions
* The `latest` tag points to the latest release of the latest stable branch.
* Branch tags like `22.2` point to the latest release of the corresponding branch.
* Full Version tags like `22.2.3` and `22.2.3.5` point to the corresponding release.
* The tag `head` is built from the latest commit to the default branch.
* Each tag has an optional `-alpine` suffix to reflect that it's built on top of `alpine`.
### Compatibility
* The amd64 image requires support for the [x86-64-v3 microarchitecture level](https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels) (AVX2, BMI1, BMI2, F16C, FMA, LZCNT, MOVBE, XSAVE). Virtually all x86 CPUs after 2015 support x86-64-v3.
* The arm64 image requires support for the [ARMv8.2-A architecture](https://en.wikipedia.org/wiki/AArch64#ARMv8.2-A) and
additionally the Load-Acquire RCpc register. The register is optional in version ARMv8.2-A and mandatory in
[ARMv8.3-A](https://en.wikipedia.org/wiki/AArch64#ARMv8.3-A). Supported in Graviton >=2, Azure and GCP instances.
Examples for unsupported devices are Raspberry Pi 4 (ARMv8.0-A) and Jetson AGX Xavier/Orin (ARMv8.2-A).
* Since ClickHouse 24.11 Ubuntu images began using `ubuntu:22.04` as its base image. It requires docker version >= `20.10.10`
containing [patch](https://github.com/moby/moby/commit/977283509f75303bc6612665a04abf76ff1d2468). As a workaround you could
use `docker run --security-opt seccomp=unconfined` instead, however this has security implications.
## How to use this image
### Start server instance
```bash theme={null}
docker run -d --name some-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server
```
By default, ClickHouse will be accessible only via the Docker network. See the networking section below.
By default, starting above server instance will be run as the `default` user without password.
### Connect to it from a native client
```bash theme={null}
docker run -it --rm --network=container:some-clickhouse-server --entrypoint clickhouse-client clickhouse/clickhouse-server
# OR
docker exec -it some-clickhouse-server clickhouse-client
```
See [ClickHouse client](/docs/concepts/features/interfaces/client) for more information about ClickHouse client.
### Connect to it using curl
```bash theme={null}
echo "SELECT 'Hello, ClickHouse!'" | docker run -i --rm --network=container:some-clickhouse-server buildpack-deps:curl curl 'http://localhost:8123/?query=' -s --data-binary @-
```
See [ClickHouse HTTP Interface](/docs/concepts/features/interfaces/http) for more information about the HTTP interface.
### Stopping / removing the container
```bash theme={null}
docker stop some-clickhouse-server
docker rm some-clickhouse-server
```
### Networking
> [!NOTE]
>
> the predefined user `default` doesn't have the network access unless the password is set,
> see "How to create default database and user on starting" and "Managing `default` user" below
You can expose your ClickHouse running in docker by [mapping a particular port](https://docs.docker.com/config/containers/container-networking/)
from inside the container using host ports:
```bash theme={null}
docker run -d -p 18123:8123 -p19000:9000 -e CLICKHOUSE_PASSWORD=changeme --name some-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server
echo 'SELECT version()' | curl 'http://localhost:18123/?password=changeme' --data-binary @-
```
Or by allowing the container to use [host ports directly](https://docs.docker.com/network/host/) using `--network=host`
(also allows achieving better network performance):
```bash theme={null}
docker run -d --network=host --name some-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server
echo 'SELECT version()' | curl 'http://localhost:8123/' --data-binary @-
```
> [!NOTE]
>
> The user default in the example above is available only for the localhost requests
### Volumes
Typically you may want to mount the following folders inside your container to achieve persistency:
* `/var/lib/clickhouse/` - main folder where ClickHouse stores the data
* `/var/log/clickhouse-server/` - logs
```bash theme={null}
docker run -d \
-v "$PWD/ch_data:/var/lib/clickhouse/" \
-v "$PWD/ch_logs:/var/log/clickhouse-server/" \
--name some-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server
```
You may also want to mount:
* `/etc/clickhouse-server/config.d/*.xml` - files with server configuration adjustments
* `/etc/clickhouse-server/users.d/*.xml` - files with user settings adjustments
* `/docker-entrypoint-initdb.d/` - folder with database initialization scripts (see below).
## Linux capabilities
ClickHouse has some advanced functionality, which requires enabling several [Linux capabilities](https://man7.org/linux/man-pages/man7/capabilities.7.html)
They're optional and can be enabled using the following [docker command-line arguments](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities):
```bash theme={null}
docker run -d \
--cap-add=SYS_NICE --cap-add=NET_ADMIN --cap-add=IPC_LOCK \
--name some-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server
```
For more information see ["Configuring CAP\_IPC\_LOCK and CAP\_SYS\_NICE Capabilities in Docker"](/docs/resources/support-center/knowledge-base/troubleshooting/configure-cap-ipc-lock-and-cap-sys-nice-in-docker)
## Configuration
The container exposes port 8123 for the [HTTP interface](/docs/concepts/features/interfaces/http) and port 9000 for the [native client](/docs/concepts/features/interfaces/tcp).
ClickHouse configuration is represented with a file "config.xml" ([documentation](/docs/concepts/features/configuration/server-config/configuration-files))
### Start server instance with custom configuration
```bash theme={null}
docker run -d --name some-clickhouse-server --ulimit nofile=262144:262144 -v /path/to/your/config.xml:/etc/clickhouse-server/config.xml clickhouse/clickhouse-server
```
### Start server as custom user
```bash theme={null}
# $PWD/data/clickhouse should exist and be owned by current user
docker run --rm --user "${UID}:${GID}" --name some-clickhouse-server --ulimit nofile=262144:262144 -v "$PWD/logs/clickhouse:/var/log/clickhouse-server" -v "$PWD/data/clickhouse:/var/lib/clickhouse" clickhouse/clickhouse-server
```
When you use the image with local directories mounted, you probably want to specify the user to maintain the proper file ownership. Use the `--user` argument and mount `/var/lib/clickhouse` and `/var/log/clickhouse-server` inside the container. Otherwise, the image will complain and not start.
### Start server from root
Starting the server from root is useful in cases where user namespace is enabled.
To do so run:
```bash theme={null}
docker run --rm -e CLICKHOUSE_RUN_AS_ROOT=1 --name clickhouse-server-userns -v "$PWD/logs/clickhouse:/var/log/clickhouse-server" -v "$PWD/data/clickhouse:/var/lib/clickhouse" clickhouse/clickhouse-server
```
### How to create default database and user on start
Sometimes you may want to create a user (user named `default` is used by default) and database on a container start. You can do it using environment variables `CLICKHOUSE_DB`, `CLICKHOUSE_USER`, `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` and `CLICKHOUSE_PASSWORD`:
```bash theme={null}
docker run --rm -e CLICKHOUSE_DB=my_database -e CLICKHOUSE_USER=username -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 -e CLICKHOUSE_PASSWORD=password -p 9000:9000/tcp clickhouse/clickhouse-server
```
#### Managing `default` user
The user `default` has disabled network access by default in the case none of `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, or `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` are set.
There's a way to make `default` user insecurely available by setting environment variable `CLICKHOUSE_SKIP_USER_SETUP` to 1:
```bash theme={null}
docker run --rm -e CLICKHOUSE_SKIP_USER_SETUP=1 -p 9000:9000/tcp clickhouse/clickhouse-server
```
## How to extend this image
To perform additional initialization in an image derived from this one, add one or more `*.sql`, `*.sql.gz`, or `*.sh` scripts under `/docker-entrypoint-initdb.d`. After the entrypoint calls `initdb`, it will run any `*.sql` files, run any executable `*.sh` scripts, and source any non-executable `*.sh` scripts found in that directory to do further initialization before starting the service.
> [!NOTE]
>
> Scripts under `/docker-entrypoint-initdb.d` are executed in **alphabetical order** by filename. If your scripts have dependencies on each other (for example, a script that creates views must run after the script that creates the referenced tables), ensure your filenames sort in the correct order.
Also, you can provide environment variables `CLICKHOUSE_USER` & `CLICKHOUSE_PASSWORD` that will be used for clickhouse-client during initialization.
For example, to add another user and database, add the following to `/docker-entrypoint-initdb.d/init-db.sh`:
```bash theme={null}
#!/bin/bash
set -e
clickhouse client -n <<-EOSQL
CREATE DATABASE docker;
CREATE TABLE docker.docker (x Int32) ENGINE = MergeTree
ORDER BY ();
EOSQL
```
# Install ClickHouse on NixOS
Source: https://clickhouse.com/docs/get-started/setup/self-managed/nixos
Install ClickHouse on NixOS and using Nix
> ClickHouse is available in the Nixpkgs repository and can be installed using Nix on **Linux** and **macOS**.
**1. Review recommendations**
Before installing ClickHouse, review the following recommendations:
* **Swap:** Disable the operating system's swap file in production environments.
* **Disk space:** The ClickHouse binary requires at least 2.5 GB of disk space for installation.
* **Network:** For distributed deployments (clustering), use at least 10 Gbit network connectivity. Network bandwidth is critical for processing distributed queries with large amounts of intermediate data, as well as for replication.
**Estimating storage requirements**
To estimate the disk space needed for your data:
1. **Estimate data volume:** Take a sample of your data and calculate the average row size, then multiply by the number of rows you plan to store.
2. **Apply the compression coefficient:** Load a sample into ClickHouse and compare the original data size with the stored table size. Clickstream data, for example, is typically compressed 6-10x.
3. **Account for replicas:** If you plan to store data in multiple replicas, multiply the estimated volume by the number of replicas.
For more detailed hardware requirements, see ["Sizing and hardware recommendations"](/docs/guides/oss/best-practices/sizing-and-hardware-recommendations)
**2. Install ClickHouse using Nix**
You can use Nix to install ClickHouse without permanently adding it to your system:
```bash theme={null}
# Install the latest stable version
nix shell nixpkgs#clickhouse
# Or install the LTS version
nix shell nixpkgs#clickhouse-lts
```
This will make the `clickhouse` binary available in your current shell session.
* The `nixpkgs#clickhouse` package provides the latest stable version.
* The `nixpkgs#clickhouse-lts` package provides the Long Term Support version.
* Both packages work on Linux and macOS.
**3. Permanent installation**
To install ClickHouse permanently on your system:
**For NixOS users**, add to your `configuration.nix`:
```nix theme={null}
environment.systemPackages = with pkgs; [
clickhouse
];
```
Then rebuild your system:
```bash theme={null}
sudo nixos-rebuild switch
```
**For non-NixOS users**, install using Nix profile:
```bash theme={null}
# Install the latest stable version
nix profile install nixpkgs#clickhouse
# Or install the LTS version
nix profile install nixpkgs#clickhouse-lts
```
**4. Start ClickHouse server**
After installation, you can start the ClickHouse server:
```bash theme={null}
clickhouse-server
```
By default, the server will start with a basic configuration and listen on `localhost:9000`.
For production use on NixOS, you may want to configure ClickHouse as a system service. Refer to the [NixOS manual](https://search.nixos.org/options?query=clickhouse) for available configuration options.
**5. Start ClickHouse client**
To connect to the ClickHouse server, open a new terminal and run:
```bash theme={null}
clickhouse-client
```
## About the Nix Package
The ClickHouse package in Nixpkgs includes:
* `clickhouse-server` - The ClickHouse database server
* `clickhouse-client` - The command-line client for connecting to ClickHouse
* `clickhouse-local` - A tool for running SQL queries on local files
* Other ClickHouse utilities
For more information about the ClickHouse package in Nixpkgs, visit:
* [Nixpkgs ClickHouse package](https://search.nixos.org/packages?query=clickhouse)
* [NixOS ClickHouse service options](https://search.nixos.org/options?query=clickhouse)
# Install ClickHouse using tgz archives
Source: https://clickhouse.com/docs/get-started/setup/self-managed/other-linux
Install ClickHouse on MacOS
> It is recommended to use official pre-compiled `tgz` archives for all Linux distributions, where installation of `deb` or `rpm` packages isn't possible.
**1. Review recommendations**
Before installing ClickHouse, review the following recommendations:
* **Swap:** Disable the operating system's swap file in production environments.
* **Disk space:** The ClickHouse binary requires at least 2.5 GB of disk space for installation.
* **Network:** For distributed deployments (clustering), use at least 10 Gbit network connectivity. Network bandwidth is critical for processing distributed queries with large amounts of intermediate data, as well as for replication.
**Estimating storage requirements**
To estimate the disk space needed for your data:
1. **Estimate data volume:** Take a sample of your data and calculate the average row size, then multiply by the number of rows you plan to store.
2. **Apply the compression coefficient:** Load a sample into ClickHouse and compare the original data size with the stored table size. Clickstream data, for example, is typically compressed 6-10x.
3. **Account for replicas:** If you plan to store data in multiple replicas, multiply the estimated volume by the number of replicas.
For more detailed hardware requirements, see ["Sizing and hardware recommendations"](/docs/guides/oss/best-practices/sizing-and-hardware-recommendations)
**2. Download and install the latest stable version**
The required version can be downloaded with `curl` or `wget` from repository [https://packages.clickhouse.com/tgz/](https://packages.clickhouse.com/tgz/).
After that downloaded archives should be unpacked and installed with installation scripts.
Below is an example of how to install the latest stable version.
> [!NOTE]
>
> For production environments, it's recommended to use the latest `stable`-version.
> You can find the release number on this [GitHub page](https://github.com/ClickHouse/ClickHouse/tags)
> with postfix `-stable`.
**3. Get the latest ClickHouse version**
Get the latest ClickHouse version from GitHub and store it in `LATEST_VERSION` variable.
```bash theme={null}
LATEST_VERSION=$(curl -s https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/utils/list-versions/version_date.tsv | \
grep -Eo '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | sort -V -r | head -n 1)
export LATEST_VERSION
```
**4. Detect your system architecture**
Detect the system architecture and set the ARCH variable accordingly:
```bash theme={null}
case $(uname -m) in
x86_64) ARCH=amd64 ;; # For Intel/AMD 64-bit processors
aarch64) ARCH=arm64 ;; # For ARM 64-bit processors
*) echo "Unknown architecture $(uname -m)"; exit 1 ;; # Exit if architecture isn't supported
esac
```
**5. Download tarballs for each ClickHouse component**
Download tarballs for each ClickHouse component. The loop tries architecture-specific
packages first, then falls back to generic ones.
```bash theme={null}
for PKG in clickhouse-common-static clickhouse-common-static-dbg clickhouse-server clickhouse-client clickhouse-keeper
do
curl -fO "https://packages.clickhouse.com/tgz/stable/$PKG-$LATEST_VERSION-${ARCH}.tgz" \
|| curl -fO "https://packages.clickhouse.com/tgz/stable/$PKG-$LATEST_VERSION.tgz"
done
```
**6. Extract and install packages**
Run the commands below to extract and install the following packages:
* `clickhouse-common-static`
```bash theme={null}
# Extract and install clickhouse-common-static package
tar -xzvf "clickhouse-common-static-$LATEST_VERSION-${ARCH}.tgz" \
|| tar -xzvf "clickhouse-common-static-$LATEST_VERSION.tgz"
sudo "clickhouse-common-static-$LATEST_VERSION/install/doinst.sh"
```
* `clickhouse-common-static-dbg`
```bash theme={null}
# Extract and install debug symbols package
tar -xzvf "clickhouse-common-static-dbg-$LATEST_VERSION-${ARCH}.tgz" \
|| tar -xzvf "clickhouse-common-static-dbg-$LATEST_VERSION.tgz"
sudo "clickhouse-common-static-dbg-$LATEST_VERSION/install/doinst.sh"
```
* `clickhouse-server`
```bash theme={null}
# Extract and install server package with configuration
tar -xzvf "clickhouse-server-$LATEST_VERSION-${ARCH}.tgz" \
|| tar -xzvf "clickhouse-server-$LATEST_VERSION.tgz"
sudo "clickhouse-server-$LATEST_VERSION/install/doinst.sh" configure
sudo /etc/init.d/clickhouse-server start # Start the server
```
* `clickhouse-client`
```bash theme={null}
# Extract and install client package
tar -xzvf "clickhouse-client-$LATEST_VERSION-${ARCH}.tgz" \
|| tar -xzvf "clickhouse-client-$LATEST_VERSION.tgz"
sudo "clickhouse-client-$LATEST_VERSION/install/doinst.sh"
```
# Self-managed install
Source: https://clickhouse.com/docs/get-started/setup/self-managed/overview
Install ClickHouse on your own infrastructure.
Install ClickHouse on the platform you operate.
## Local development
Run ClickHouse on your own machine to build, test, and experiment.
> [card] [Quick install](/docs/get-started/setup/self-managed/quick-install)
>
> One-line `curl` script that downloads a single ClickHouse binary and installs `clickhousectl`.
> [card] [clickhousectl (CLI)](/docs/get-started/setup/self-managed/clickhousectl)
>
> Install ClickHouse versions and run local ClickHouse and Postgres servers with the ClickHouse CLI.
> [card] [Docker](/docs/get-started/setup/self-managed/docker)
>
> Pull the official Docker image and run a container.
> [card] [Source build](/docs/get-started/setup/self-managed/advanced)
>
> Build from source and other advanced installation paths.
## Production server
Install ClickHouse Server from packages or binaries for a long-running deployment.
> [card] [Debian / Ubuntu](/docs/get-started/setup/self-managed/debian-ubuntu)
>
> Install via the official APT repository.
> [card] [Red Hat / CentOS](/docs/get-started/setup/self-managed/redhat)
>
> Install via the official YUM/DNF repository.
> [card] [NixOS](/docs/get-started/setup/self-managed/nixos)
>
> Install on NixOS via the package manager.
> [card] [Other Linux](/docs/get-started/setup/self-managed/other-linux)
>
> Generic Linux distribution install via tarball.
# Quick install
Source: https://clickhouse.com/docs/get-started/setup/self-managed/quick-install
Quickly install ClickHouse using a curl install script
If you don't need to install ClickHouse for production, you can run an install script using curl. The script will determine a suitable binary for your OS.
**1. Install ClickHouse using curl**
Run the following command to download a single binary for your operating system.
```bash theme={null}
curl https://clickhouse.com/ | sh
```
On Linux and macOS (x86\_64 and ARM64), this also installs [`clickhousectl`](/docs/get-started/setup/self-managed/clickhousectl),
the ClickHouse CLI, into `~/.local/bin` (with a `chctl` alias). If the script
says `~/.local/bin` is not on your `PATH`, add it:
```bash theme={null}
export PATH="$HOME/.local/bin:$PATH"
```
`clickhousectl` helps you install and manage multiple local ClickHouse versions,
run ClickHouse and Postgres servers in the background, and manage ClickHouse Cloud.
To have `clickhousectl` install and manage ClickHouse versions for you instead,
run `clickhousectl local use latest`. See the [clickhousectl install page](/docs/get-started/setup/self-managed/clickhousectl)
for the CLI-based workflow.
To install just the `clickhouse` binary without `clickhousectl`, set
`CLICKHOUSE_ONLY=1`:
```bash theme={null}
curl https://clickhouse.com/ | CLICKHOUSE_ONLY=1 sh
```
> [!NOTE]
>
> For Mac users: If you're getting errors that the developer of the binary can't be verified, please see [here](/docs/resources/support-center/knowledge-base/troubleshooting/fix-developer-verification-error-in-macos).
**2. Start clickhouse-local**
`clickhouse-local` allows you to process local and remote files using ClickHouse's
powerful SQL syntax and without the need for configuration. Table data is stored
in a temporary location, meaning that after a restart of `clickhouse-local`
previously created tables are no longer available.
Run the following command to start [clickhouse-local](/docs/concepts/features/tools-and-utilities/clickhouse-local):
```bash theme={null}
./clickhouse
```
**3. Start clickhouse-server**
Should you wish to persist data, you'll want to run `clickhouse-server`. You can
start the ClickHouse server using the following command:
```bash theme={null}
./clickhouse server
```
**4. Start clickhouse-client**
With the server up and running, open a new terminal window and run the following command
to launch `clickhouse-client`:
```bash theme={null}
./clickhouse client
```
You will see something like this:
```response theme={null}
./clickhouse client
ClickHouse client version 24.5.1.117 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 24.5.1.
local-host :)
```
Table data is stored in the current directory and will still be available after a restart
of the ClickHouse server. If necessary, you can pass
`-C config.xml` as an additional command line argument to `./clickhouse server`
and provide further configuration in a configuration
file. All available configuration settings are documented [here](/docs/reference/settings/server-settings/settings) and in the
[example configuration file
template](https://github.com/ClickHouse/ClickHouse/blob/master/programs/server/config.xml).
You're now ready to start sending SQL commands to ClickHouse!
> [!TIP]
>
> [Creating tables in ClickHouse](/docs/get-started/quickstarts/creating-tables) walks you through creating tables and inserting data.
# Install ClickHouse on rpm-based Linux distributions
Source: https://clickhouse.com/docs/get-started/setup/self-managed/redhat
Install ClickHouse on Redhat/CentOS Linux
> It is recommended to use official pre-compiled `rpm` packages for **CentOS**, **RedHat**, and all other rpm-based
> Linux distributions.
**1. Review recommendations**
Before installing ClickHouse, review the following recommendations:
* **Swap:** Disable the operating system's swap file in production environments.
* **Disk space:** The ClickHouse binary requires at least 2.5 GB of disk space for installation.
* **Network:** For distributed deployments (clustering), use at least 10 Gbit network connectivity. Network bandwidth is critical for processing distributed queries with large amounts of intermediate data, as well as for replication.
**Estimating storage requirements**
To estimate the disk space needed for your data:
1. **Estimate data volume:** Take a sample of your data and calculate the average row size, then multiply by the number of rows you plan to store.
2. **Apply the compression coefficient:** Load a sample into ClickHouse and compare the original data size with the stored table size. Clickstream data, for example, is typically compressed 6-10x.
3. **Account for replicas:** If you plan to store data in multiple replicas, multiply the estimated volume by the number of replicas.
For more detailed hardware requirements, see ["Sizing and hardware recommendations"](/docs/guides/oss/best-practices/sizing-and-hardware-recommendations)
**2. Set up the RPM repository**
Add the official repository by running the following command:
```bash theme={null}
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo
```
For systems with `zypper` package manager (openSUSE, SLES), run:
```bash theme={null}
sudo zypper addrepo -r https://packages.clickhouse.com/rpm/clickhouse.repo -g
sudo zypper --gpg-auto-import-keys refresh clickhouse-stable
```
In the steps below, `yum install` can be replaced by `zypper install`, depending
on which package manager you're using.
**3. Install ClickHouse server and client**
To install ClickHouse run the following commands:
```bash theme={null}
sudo yum install -y clickhouse-server clickhouse-client
```
* You can replace `stable` with `lts` to use different [release kinds](/docs/resources/support-center/knowledge-base/setup-installation/production) based on your needs.
* You can download and install packages manually from [packages.clickhouse.com/rpm](https://packages.clickhouse.com/rpm/stable).
* To specify a particular version, add `-$version` to the end of the package name,
for example:
```bash theme={null}
sudo yum install clickhouse-server-22.8.7.34
```
**4. Start ClickHouse server**
To start ClickHouse server, run:
```bash theme={null}
sudo systemctl enable clickhouse-server
sudo systemctl start clickhouse-server
sudo systemctl status clickhouse-server
```
To start ClickHouse client, run:
```bash theme={null}
clickhouse-client
```
If you set up a password for your server, then you will need to run:
```bash theme={null}
clickhouse-client --password
```
**5. Install standalone ClickHouse Keeper**
> [!TIP]
>
> In production environments we strongly recommend running ClickHouse Keeper on dedicated nodes.
> In test environments, if you decide to run ClickHouse Server and ClickHouse Keeper on the same server,
> then you don't need to install ClickHouse Keeper as it is included with ClickHouse server.
To install `clickhouse-keeper` on standalone ClickHouse Keeper servers, run:
```bash theme={null}
sudo yum install -y clickhouse-keeper
```
**6. Enable and start ClickHouse Keeper**
```bash theme={null}
sudo systemctl enable clickhouse-keeper
sudo systemctl start clickhouse-keeper
sudo systemctl status clickhouse-keeper
```
# Agentic analytics
Source: https://clickhouse.com/docs/get-started/use-cases/agentic-analytics
Learn how ClickHouse enables agentic analytics
AI workloads impose a consistent set of requirements regardless of use case:
* high query concurrency
* sub-second response times
* full-fidelity data at scale
This document explains how ClickHouse addresses those requirements across real-time analytics, data warehousing, and observability, and how those use cases are converging into a unified data platform for agentic applications.
## ClickHouse for agentic workloads
> [tab] Real-time analytics
>
> AI-powered application features such as generated insights, anomaly detection, recommendations, and natural language interfaces to product data, all require a tight feedback loop between transactional writes and analytical reads.
> The standard architecture for this is Postgres + ClickHouse:
>
> * Postgres handles transactions and application state, ClickHouse handles analytics.
> * ClickHouse provides fast ingestion, sub-second queries on billions of rows, and the concurrency levels that customer-facing applications require.
>
> As applications become agentic, this pairing becomes more critical.
> Agents must query live product data continuously, which increases both query frequency and concurrency.
> ClickHouse addresses this with a native Postgres + ClickHouse integration that provides automatic data replication and a unified developer experience, removing the need to manage a separate CDC pipeline.
> [tab] Data warehousing
>
> Natural language analytics interfaces (sometimes called AI Analyst) are moving from experimentation into production.
> Users ask questions in plain English and expect answers in seconds.
>
> The infrastructure implication is that a single natural language query does not generate one SQL query — it typically generates dozens in rapid succession as the agent explores available datasets and evaluates multiple reasoning paths.
> As a result, internal analyst workloads start to resemble external customer-facing workloads in their concurrency and latency profile.
>
> Legacy data warehouses were designed for infrequent, batch-oriented queries. They optimize for overall throughput across many queries, not for sub-second response times at high concurrency. Running AI Analyst workloads on that architecture produces either unacceptable latency or costs that scale faster than the value delivered.
>
> ClickHouse was built for high-concurrency interactive queries: petabyte-scale data, thousands of concurrent users, sub-second response times against billions of rows.
> [tab] Observability
>
> Traditional observability stacks are built on three separate pillars — metrics, logs, and traces — with data pre-aggregated and sampled to control storage costs. This tradeoff is acceptable for human-driven workflows but breaks down for AI SRE.
> Automated incident triage, root cause analysis, and anomaly correlation require granular, high-cardinality, long-retention data. An AI agent correlating an error pattern with a deployment event from three days ago cannot work with sampled logs or downsampled metrics.
>
> The architecture that supports AI SRE is a single source of truth based on wide structured events stored in columnar storage. Full-fidelity events are stored once, and metrics, traces, and SLOs are derived from them at query time rather than pre-aggregated on ingest.
> ClickHouse is well-suited for this model:
>
> * High compression on log and event data
> * Sub-second queries on high-cardinality wide events
> * Efficient ingestion at production infrastructure volumes
> * Cost model based on compute and storage, not per-GB ingestion fees
>
> ClickStack is ClickHouse's observability stack built on this model, using OpenTelemetry as the data collection layer.
> It is available open source and as a managed offering.
## Convergence of data warehousing and observability
Data warehousing and observability have historically been separate domains with separate vendors, buyers, and stacks. That separation is increasingly a convention rather than a technical requirement.
Both domains now write to object storage. Both require interactive, low-latency queries at high concurrency. And at the data level, the same events are often stored twice — once in an observability platform and once in a data warehouse — with a fragile synchronization layer in between.
Storing all of it once in open formats, queryable by both AI Analyst and AI SRE tooling, removes that duplication and makes context available across both workflows.
## The platform layer: Agent-ready interfaces and LLM observability
Two additional components are required alongside the database for a complete agentic analytics platform.
**Agent-ready interfaces**
When AI agents are the primary interface to data, the data platform needs to expose its capabilities in ways agents can consume — MCP-compatible APIs, natural language interfaces, and agent frameworks that integrate without bespoke per-use-case work. The Agentic Data Stack combines ClickHouse with LibreChat to provide a turnkey way to deploy analytics agents over your data.
**LLM observability**
As agents proliferate, tracing their execution, monitoring model performance, tracking costs, and debugging failures across multi-step workflows becomes a core engineering requirement. Langfuse runs on ClickHouse Cloud to provide real-time LLM observability at scale.
# Choose a service for your workload
Source: https://clickhouse.com/docs/get-started/use-cases/choosing-a-service
Choose ClickHouse or Postgres services within ClickHouse Cloud for analytics or transactions, Managed ClickStack for observability, or chDB for local analysis.
Start with the queries and correctness requirements your application needs to support. A large row count alone does not determine the database, and an application that generates reports can contain both analytical and transactional workloads.
ClickHouse Cloud is the platform for managed services, including ClickHouse for analytics and ClickHouse Managed Postgres for transactional workloads. These services run different database engines — ClickHouse and PostgreSQL — with different SQL behavior. You can use either service independently or combine them within ClickHouse Cloud.
## Map the workload to a starting point
| What your application needs | Start by evaluating | What to check before choosing |
| - | - | - |
| Aggregations, filtering, and comparisons across large event or result histories | [ClickHouse service in ClickHouse Cloud](/docs/products/cloud/getting-started/intro) | Representative queries, ingestion batches, data freshness, concurrent queries, and the design of updates or retries |
| Transactional application state, such as orders, inventory reservations, or jobs whose state transitions require transactions and constraints | [ClickHouse Managed Postgres in ClickHouse Cloud](/docs/products/managed-postgres/overview) | Transaction boundaries, constraints, indexes, connection management, and the service's supported features and availability |
| Transactional application state plus analytical queries that need an independent serving system | [Postgres and ClickHouse services in ClickHouse Cloud](/docs/products/managed-postgres/clickpipes) | Which tables need replication, acceptable replication lag, schema changes, and the cost and operation of two services |
| A managed experience for searching and investigating application logs, metrics, and traces | [Managed ClickStack](/docs/clickstack/getting-started/managed) | Instrumentation, telemetry ingestion, retention, and the observability workflows your team needs |
| Analysis of files or in-memory data inside a local application or notebook | [chDB](/docs/chdb/index) | Local resources and whether you need a separately hosted, shared database service |
Check the linked product documentation for current availability, regions, and limitations. ClickHouse Managed Postgres is currently in public beta; see its [quickstart](/docs/products/managed-postgres/quickstart).
For a self-managed ClickHouse server or other local options, see [deployment modes](/docs/get-started/about/deployment-modes).
## Example: report results and run history
Suppose an application processes uploaded files, produces reports, and needs to retain queryable results and a history of runs. Before choosing a database, distinguish the result rows from the run records:
* **Results:** Are queries retrieving a few rows for one report, or aggregating across many reports, datasets, and time ranges? Which table is expected to reach hundreds of millions of rows?
* **Run records:** Is one immutable record written when a run completes, or does the application need to coordinate live job ownership and status transitions?
* **Correctness:** Must several records change in one transaction? Must the database enforce uniqueness or prevent two workers from claiming the same job?
* **Freshness:** How soon must a successful write be visible, and can queries tolerate an incomplete or delayed analytical copy?
### Completed runs and analytical results
A ClickHouse service in ClickHouse Cloud is a candidate when the dominant workload is analysis across result rows and run records can be appended on completion. A small metadata table does not by itself require a second database.
Design and test how readers distinguish a complete run from a partially ingested one. Define stable run identifiers, retry behavior, and how duplicate result rows are handled. Separate inserts into a results table and a run-history table must not be treated as a single cross-table transaction.
If you store multiple versions of a run record, define how queries select the current version. [ReplacingMergeTree](/docs/reference/engines/table-engines/mergetree-family/replacingmergetree) supports query-time deduplication with `FINAL`; correctness must not depend on background merges having already happened. The [update reference](/docs/reference/statements/update) describes another update mechanism and its limitations. Neither pattern should be assumed to provide PostgreSQL-style transactional guarantees.
### Transactional job state
Evaluate ClickHouse Managed Postgres when the run table is also a transactional work queue or system of record: for example, a worker must claim a job atomically while other workers compete for it, or several application records must change together with constraints enforced.
Postgres can also serve reporting queries. Add an analytical service when representative query performance, concurrency, or isolation requirements justify it, rather than assuming that every reporting application needs two databases.
### Transactions and analytics together
When both workloads justify separate services, keep transactional state in Postgres and replicate the required tables to ClickHouse using [ClickPipes](/docs/products/managed-postgres/clickpipes) or [WalShadow](/docs/products/managed-postgres/walshadow). Account for replication lag and continue to use the transactional source for decisions that require current application state. Replication does not make a Postgres write and a ClickHouse read part of one transaction.
The [pg\_clickhouse extension](/docs/products/managed-postgres/extensions/pg_clickhouse/introduction) can provide access to ClickHouse through Postgres. A shared query entry point does not turn the two services into one database engine or remove the need to account for data freshness.
## Check fit before provisioning
Write down a small set of representative queries and test them against realistic data volumes. Include narrow lookups as well as cross-history aggregates, your expected concurrent load, and the correctness cases that matter to your application. Report the schema, query shapes, hardware or service size, and ingestion behavior alongside any benchmark result.
For a managed deployment, also check:
* Region, networking, access control, and recovery requirements.
* Expected active time, stored data, backups, and transfer charges in the [ClickHouse Cloud billing guide](/docs/products/cloud/reference/billing/billing-overview) or [Managed Postgres pricing guide](/docs/products/managed-postgres/pricing).
* Whether an intermittent analytical workload can tolerate the connection delay associated with [automatic idling](/docs/products/cloud/features/autoscaling/idling).
* The team's responsibility for schema design, application retries, query tuning, and cost controls, even when the service manages the infrastructure.
For ClickHouse or Postgres services in ClickHouse Cloud, use the [ClickHouse CLI](/docs/products/cloud/features/cli) to create and manage resources from a terminal. For Managed ClickStack and chDB, follow the product guides linked in the workload map.
# Data warehousing
Source: https://clickhouse.com/docs/get-started/use-cases/data-warehousing
Build modern data warehouse architectures by combining the flexibility of data lakes with ClickHouse Cloud's performance
The modern data warehouse no longer tightly couples storage and compute. Instead, distinct but interconnected layers for storage, governance, and query processing give you the flexibility to choose the right tools for your workflows.
By adding open table formats and a high-performance query engine like ClickHouse to cloud object storage, you get database-grade capabilities — ACID transactions, schema enforcement, and fast analytical queries — without sacrificing the openness of your data lake. This combination brings performance together with interoperable, cost-effective storage to support your traditional analytics and modern AI/ML workloads.
## What this architecture provides
By combining open object storage and table formats with ClickHouse as your query engine, you get:
| Benefit | Description |
| - | - |
| **Consistent table updates** | Atomic commits to table state mean concurrent writes don't produce corrupt or partial data. This solves one of the biggest problems with raw data lakes. |
| **Schema management** | Enforced validation and tracked schema evolution prevent the "data swamp" problem where data becomes unusable due to schema inconsistencies. |
| **Query performance** | Indexing, statistics, and data layout optimizations like data skipping and clustering let SQL queries run at speeds comparable to a dedicated data warehouse. Combined with ClickHouse's columnar engine, this holds true even on data stored in object storage. |
| **Governance** | Catalogs and table formats provide fine-grained access control and auditing at row and column levels, addressing the limited security controls in basic data lakes. |
| **Separation of storage and compute** | Storage and compute scale independently on commodity object storage, which is significantly cheaper than proprietary warehouse storage. While separation is standard in modern cloud warehouses, open formats let you choose *which* compute engine scales with your data. |
## How ClickHouse powers your data warehouse
Data flows from streaming platforms and existing warehouses through object storage into ClickHouse, where it's transformed, optimized, and served to your BI/AI tools.
> [accordion] Data ingestion
>
> For bulk data loads, you typically use an object store like S3 or GCS as an intermediary. ClickHouse's [Parquet](/docs/guides/clickhouse/data-formats/parquet) reading performance lets you load data at hundreds of millions of rows per second using the [S3 table engine](/docs/reference/engines/table-engines/integrations/s3). For real-time streaming, [ClickPipes](/docs/integrations/clickpipes/home) connects directly to platforms like Kafka and Confluent.
>
> You can also migrate from existing data warehouses like Snowflake, BigQuery, and Databricks by exporting to object storage and loading into ClickHouse via [table engines](/docs/reference/engines/table-engines).
> [accordion] Querying
>
> You can query data directly from object stores like S3 and GCS, or from data lakes with open table formats like [Iceberg](/docs/reference/engines/table-engines/integrations/iceberg), [Delta Lake](/docs/reference/engines/table-engines/integrations/deltalake), and [Hudi](/docs/reference/engines/table-engines/integrations/hudi) — directly or through data catalogs like [AWS Glue Catalog](/docs/guides/use-cases/data-warehousing/glue-catalog), [Unity Catalog](/docs/guides/use-cases/data-warehousing/unity-catalog), and [Iceberg REST](/docs/guides/use-cases/data-warehousing/rest-catalog).
>
> ClickHouse Cloud offers the [query cache](/docs/concepts/features/performance/caches/query-cache), [sparse indexes](/docs/concepts/features/performance/skip-indexes/skipping-indexes), and [projections](/docs/concepts/features/projections/projections) out of the box, plus 70+ file formats and SQL functions for dates, arrays, JSON, geo, and approximate aggregations at scale.
> [accordion] Data transformations
>
> [Materialized views](/docs/concepts/features/materialized-views) in ClickHouse automate transformations — triggered when new data is inserted into source tables, so you can extract, aggregate, and modify data as it arrives without building bespoke pipelines.
>
> For more complex modeling, ClickHouse's [dbt integration](/docs/integrations/connectors/data-ingestion/etl-tools/dbt) lets you define transformations as version-controlled SQL models.
> [accordion] Integrations
>
> ClickHouse has native connectors for BI tools like [Tableau](/docs/integrations/connectors/data-visualization/tableau/tableau-and-clickhouse) and [Looker](/docs/integrations/connectors/data-visualization/looker-and-clickhouse). Tools without a native connector can connect through the [MySQL wire protocol](/docs/concepts/features/interfaces/mysql). The [MCP server](/docs/guides/use-cases/ai-ml/MCP) connects ClickHouse to LLMs for conversational analytics, and flexible [RBAC](/docs/concepts/features/security/access-rights) controls let you expose read-only tables securely.
## Hybrid architecture: The best of both worlds
Beyond querying your data lake, you can ingest performance-critical data into ClickHouse's native [MergeTree](/docs/reference/engines/table-engines/mergetree-family/mergetree) storage for use cases that demand ultra-low latency — real-time dashboards, operational analytics, or interactive applications.
This gives you a tiered data strategy. Hot, frequently accessed data lives in ClickHouse's optimized storage for sub-second query responses, while the complete data history stays in the lake and remains queryable. You can also use ClickHouse materialized views to continuously transform and aggregate lake data into optimized tables, bridging the two tiers automatically.
You choose where data lives based on performance requirements, not technical limitations.
> [!TIP]
>
> **ClickHouse Academy**
>
> Take the free [Data Warehousing with ClickHouse](https://clickhouse.com/learn/data-warehousing) course to learn more.
# Observability
Source: https://clickhouse.com/docs/get-started/use-cases/observability
Use ClickHouse Cloud for observability, monitoring, logging, and system performance analysis in distributed applications
## ClickHouse for observability
Early adopters of ClickHouse recognized something fundamental: observability is a data problem.
The database you choose defines the cost, scale, and capabilities of your observability platform - which is why selecting the right database for time-series data is often the most important architectural decision when building in-house or starting an observability company.
This is exactly why ClickHouse has been at the core of observability stacks for years.
From industry giants like Netflix and eBay to observability startups like Sentry and Dash0, ClickHouse powers logs, metrics, and traces at a massive scale.
Its column-oriented storage, aggressive compression, and vectorized execution engine dramatically reduce costs and deliver the sub-second queries engineers need to debug live systems without waiting on slow tooling.
## ClickStack
At the core of ClickHouse for the observability use-case is ClickStack, an innovative observability platform that revolutionizes how you handle logs, traces, and metrics by storing them all in a single ClickHouse database.
This unified approach makes it incredibly easy to correlate signals across your entire infrastructure and applications.
ClickStack is built around three core components that work seamlessly together:
> [tab] OpenTelemetry
>
> > The ClickStack distribution of the OpenTelemetry collector serves as your data ingestion gateway. OpenTelemetry has become the industry standard for observability data, making this collector the natural choice for handling your telemetry streams.
> [tab] ClickHouse
>
> > ClickHouse is explicitly chosen for its exceptional performance with high-cardinality data and lightning-fast aggregations that observability workloads demand.
> [tab] HyperDX
>
> > HyperDX provides an intuitive user interface that brings all your data together with powerful querying capabilities and correlation features.
What makes ClickStack particularly compelling is its flexibility. While it's OpenTelemetry native and works best with the OpenTelemetry ecosystem, it's not exclusively tied to it. You can ingest data from various sources, including S3, Kafka, and other agents, giving you the freedom to work with your existing infrastructure.
[Read the ClickStack docs](/docs/clickstack)
# Use cases
Source: https://clickhouse.com/docs/get-started/use-cases/overview
Explore ClickHouse use cases including real-time analytics, observability, data warehousing, and agentic analytics
ClickHouse is suitable for use as both a **primary data store** and as an **analytics layer**.
Its columnar architecture, vectorized processing, and cloud-native design make it uniquely suited for analytical workloads that require both speed and scale.
If you are choosing a database for an application, start with [Choose a service for your workload](/docs/get-started/use-cases/choosing-a-service). It maps analytical results, transactional state, observability, and local analysis to the relevant services and explains when a mixed workload needs more than one database.
| Use case | TLDR |
| - | - |
| [Real-time analytics](/docs/get-started/use-cases/real-time-analytics) | Sub-second queries on billions of rows as data arrives. Ideal for customer-facing dashboards, live leaderboards, and any product feature that reads from a continuously updating event stream. |
| [Observability](/docs/get-started/use-cases/observability) | Ingest and query logs, metrics, and traces in a single ClickHouse database via ClickStack. Eliminates the three-pillar silo problem and gives correlated, full-fidelity analysis without pre-aggregation. |
| [Data warehousing](/docs/get-started/use-cases/data-warehousing) | Pair ClickHouse with open table formats (Iceberg, Delta Lake, Hudi) on object storage to get ACID transactions, schema enforcement, and warehouse-grade query performance without locking your data away. |
| [Agentic analytics](/docs/get-started/use-cases/agentic-analytics) | AI agents demand high concurrency and full-fidelity data simultaneously. ClickHouse converges real-time analytics, data warehousing, and observability into a single platform that agents can query directly via MCP or natural language interfaces. |
# Real-time analytics
Source: https://clickhouse.com/docs/get-started/use-cases/real-time-analytics
Learn how to build real-time analytics applications with ClickHouse Cloud for instant insights and data-driven decision making
In analytics, "real-time" usually means the user experience itself feels live.
A customer refreshes a dashboard, opens a leaderboard, or investigates an issue and expects the data to reflect what just happened.
Technically, this means not just delivering low-latency analytical queries but achieving this while data is being continuously inserted at high volume.
## Properties of a real-time analytics system
When customers evaluate a real-time analytics platform, they often focus only on query latency. "Can it return an answer within 50ms?" is a reasonable question, and one most analytical engines can answer convincingly if you throw enough compute at a static dataset.
The question that decides what users actually experience is harder. It is whether the system can return a 50ms answer **on data that arrived a second ago**, while ingestion is still running, and while other users are also querying.
Data availability requires thinking about end-to-end time-to-insight, which has three components.
* **Time to ingest**: How long does it take for newly generated data to land in the platform and become durably stored?
* **Time to transform and prepare**: How long does it take to clean, enrich, join, pre-aggregate, or update the serving structures (materialized views, rollups, indexes) that queries actually hit?
* **Time to query**: How long does it take to plan and execute the read once the data is available?
## How ClickHouse powers real-time analytics
> [accordion] Data ingestion
>
> With [ClickPipes](/docs/integrations/clickpipes/home), available exclusively in ClickHouse Cloud, you get a turn-key integration engine that makes ingesting large volumes of data effortless. Select an incoming data source and format, tune your schema, and let your pipeline run.
>
> ClickHouse's extensive library of [table engines](/docs/reference/engines/table-engines) supports ingesting data from Kafka topics, S3 buckets, OLTP databases, and more. Unlike other OLAP databases that require you to batch inserts for high throughput, ClickHouse handles smaller payloads equally well — [asynchronous inserts](/docs/concepts/features/operations/insert/asyncinserts) automatically batch them for optimal write performance.
> [accordion] Data transformations and queries
>
> [Materialized views](/docs/concepts/features/materialized-views) make transformations seamless — automatically triggered when new data is inserted, they extract, aggregate, and modify data as it arrives without bespoke pipelines. Chaining them together adds modular flexibility.
>
> Queries on materialized views are exceptionally fast since results are stored in dedicated tables. ClickHouse Cloud includes the [query cache](/docs/concepts/features/performance/caches/query-cache), [sparse indexes](/docs/concepts/features/performance/skip-indexes/skipping-indexes), and [projections](/docs/concepts/features/projections/projections) at no extra tier.
> [accordion] Applications and dashboards
>
> ClickHouse powers user-facing analytics across financial services, gaming, e-commerce, and more. It supports a REST interface so web developers can build lightweight applications without complex binary protocols.
>
> Native connectors exist for BI tools like [Grafana](/docs/integrations/connectors/data-visualization/grafana), [Tableau](/docs/integrations/connectors/data-visualization/tableau/tableau-and-clickhouse), and [Looker](/docs/integrations/connectors/data-visualization/looker-and-clickhouse), plus language clients, SQL clients, and the [MySQL wire protocol](/docs/concepts/features/interfaces/mysql) for tools without a native connector.
#
Source: https://clickhouse.com/docs/index
[Quickstart: Get Started with Cloud](/docs/get-started/setup/cloud)
# Start building with ClickHouse
Docs for the fastest data platform
Search...
Ask AI
[Set up docs MCP server](/docs/resources/support-center/knowledge-base/setup-installation/set-up-clickhouse-documentation-mcp-server)
**Get started**
I am new here, I want to get started and learn more
**Guides**
I want to accomplish a specific task with ClickHouse
**Reference**
I know what I want to do - I just need the details
**Our solutions**
I want to explore ClickHouse solutions
***
## Explore by use case
Discover how ClickHouse powers real-time analytics, observability, data warehousing, and agentic AI at any scale.
### Real-time analytics
Deliver instant insights and dashboards at scale. Analyze billions of rows in real time with millisecond results.
### Observability
Store and query logs, metrics and traces at scale using ClickStack, the open source observability stack powered by ClickHouse.
### Data warehousing
Analyze and explore data instantly for insights and apps. Scale faster by offloading heavy workloads.
### Agentic analytics
Let AI agents query your real-time data directly to deliver instant, conversational insights and visualizations
***
## Explore by product
Browse ClickHouse products and integrations, from the managed cloud service to open source projects.
**ClickHouse Cloud**
### ClickStack
The all in one observability stack powered by ClickHouse, which gives you lightning-fast queries and powerful aggregations on logs, metrics, traces, session replays and errors.
### ClickHouse Managed Postgres
Fast, scalable, enterprise-grade Postgres service, natively integrated with ClickHouse. Easily combine Postgres for transactions and ClickHouse for analytics.
### Integrations
100+ integrations to connect the tools and services that you love with ClickHouse.
### [Langfuse](https://langfuse.com/docs)
Open-source LLM engineering platform for observability, metrics, evaluations, and prompt management.
### Kubernetes Operator
Automates deploying, configuring, and managing ClickHouse and ClickHouse Keeper clusters on Kubernetes.
### chDB
A fast, reliable, and scalable in-process database. Experience the power of ClickHouse, in-process.
# Alternative backup or restore methods
Source: https://clickhouse.com/docs/concepts/features/backup-restore/alternative-methods
Details alternative backup or restore methods
ClickHouse stores data on disk, and there are many ways to back up disks.
These are some alternatives that have been used in the past, and that may fit
your use case.
### Duplicating source data somewhere else
Often data ingested into ClickHouse is delivered through some sort of persistent
queue, such as [Apache Kafka](https://kafka.apache.org). In this case, it is possible to configure an
additional set of subscribers that will read the same data stream while it is
being written to ClickHouse and store it in cold storage somewhere. Most companies
already have some default recommended cold storage, which could be an object store
or a distributed filesystem like [HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html).
### Filesystem Snapshots
Some local filesystems provide snapshot functionality (for example, [ZFS](https://en.wikipedia.org/wiki/ZFS)),
but they might not be the best choice for serving live queries. A possible solution
is to create additional replicas with this kind of filesystem and exclude them
from the [Distributed](/docs/reference/engines/table-engines/special/distributed) tables that are used for `SELECT` queries.
Snapshots on such replicas will be out of reach of any queries that modify data.
As a bonus, these replicas might have special hardware configurations with more
disks attached per server, which would be cost-effective.
For smaller volumes of data, a simple `INSERT INTO ... SELECT ...` to remote tables
might work as well.
### Manipulations with Parts
ClickHouse allows using the `ALTER TABLE ... FREEZE PARTITION ...` query to create
a local copy of table partitions. This is implemented using hardlinks to the `/var/lib/clickhouse/shadow/`
folder, so it usually doesn't consume extra disk space for old data. The created
copies of files aren't handled by ClickHouse server, so you can just leave them there:
you will have a simple backup that doesn't require any additional external system,
but it will still be prone to hardware issues. For this reason, it's better to
remotely copy them to another location and then remove the local copies.
Distributed filesystems and object stores are still a good options for this,
but normal attached file servers with a large enough capacity might work as well
(in this case the transfer will occur via the network filesystem or maybe [rsync](https://en.wikipedia.org/wiki/Rsync)).
Data can be restored from backup using the `ALTER TABLE ... ATTACH PARTITION ...`
For more information about queries related to partition manipulations, see the
[`ALTER` documentation](/docs/reference/statements/alter/partition).
A third-party tool is available to automate this approach: [clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup).
# BACKUP/RESTORE to or from Azure Blob Storage
Source: https://clickhouse.com/docs/concepts/features/backup-restore/azure-blob-storage
Details backup/restore to or from an Azure Blob Storage endpoint
## Syntax
```sql theme={null}
-- core commands
BACKUP | RESTORE
--- what to backup/restore (or exclude)
TABLE [db.]table_name [AS [db.]table_name_in_backup] |
DICTIONARY [db.]dictionary_name [AS [db.]name_in_backup] |
DATABASE database_name [AS database_name_in_backup] |
TEMPORARY TABLE table_name [AS table_name_in_backup] |
VIEW view_name [AS view_name_in_backup] |
[EXCEPT TABLES ...] |
[EXCEPT DATA FROM {TABLE|TABLES} ...] | -- BACKUP only, rejected on RESTORE
ALL [EXCEPT {TABLES|DATABASES}...] } [,...]
---
[ON CLUSTER 'cluster_name']
--- where to backup or restore to or from
TO|FROM
File('/') |
Disk('', '/') |
S3('/', '', '', '') |
AzureBlobStorage('/', '', '', '', '')
--- additional settings
[SETTINGS ...]
[ASYNC]
```
**See ["command summary"](/docs/concepts/features/backup-restore/overview#command-summary) for more details
of each command.**
## Configuring BACKUP / RESTORE to use an AzureBlobStorage endpoint
To write backups to an AzureBlobStorage container you need the following pieces of information:
* AzureBlobStorage endpoint connection string / url,
* Container,
* Path,
* Account name (if url is specified)
* Account Key (if url is specified)
The destination for a backup will be specified as:
```sql theme={null}
AzureBlobStorage('/', '', '', '', '')
```
```sql theme={null}
BACKUP TABLE data TO AzureBlobStorage('DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite1:10000/devstoreaccount1/;',
'testcontainer', 'data_backup');
RESTORE TABLE data AS data_restored FROM AzureBlobStorage('DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite1:10000/devstoreaccount1/;',
'testcontainer', 'data_backup');
```
# BACKUP / RESTORE to disk
Source: https://clickhouse.com/docs/concepts/features/backup-restore/local-disk
Details backup/restore to or from a local disk
## Syntax
```sql theme={null}
-- core commands
BACKUP | RESTORE
--- what to backup/restore (or exclude)
TABLE [db.]table_name [AS [db.]table_name_in_backup] |
DICTIONARY [db.]dictionary_name [AS [db.]name_in_backup] |
DATABASE database_name [AS database_name_in_backup] |
TEMPORARY TABLE table_name [AS table_name_in_backup] |
VIEW view_name [AS view_name_in_backup] |
[EXCEPT TABLES ...] |
[EXCEPT DATA FROM {TABLE|TABLES} ...] | -- BACKUP only, rejected on RESTORE
ALL [EXCEPT {TABLES|DATABASES}...] } [,...]
---
[ON CLUSTER 'cluster_name']
--- where to backup or restore to or from
TO|FROM
File('/') |
Disk('', '/') |
S3('/', '', '', '') |
AzureBlobStorage('/', '', '', '', '')
--- additional settings
[SETTINGS ...]
[ASYNC]
```
**See ["command summary"](/docs/concepts/features/backup-restore/overview#command-summary) for more details
of each command.**
## Configure backup destinations for disk
### Configure a backup destination for local disk
In the examples below you will see the backup destination specified as `Disk('backups', '1.zip')`.\
To use the `Disk` backup engine it is necessary to first add a file specifying
the backup destination at the path below:
```text theme={null}
/etc/clickhouse-server/config.d/backup_disk.xml
```
For example, the configuration below defines a disk named `backups` and then adds that disk to
the **allowed\_disk** list of **backups**:
```xml highlight={4,10-13} theme={null}
local
/backups/
backups
/backups/
```
### Configure a backup destination for S3 disk
It is also possible to `BACKUP`/`RESTORE` to S3 by configuring an S3 disk in the
ClickHouse storage configuration. Configure the disk like this by adding a file to
`/etc/clickhouse-server/config.d` as was done above for the local disk.
```xml theme={null}
s3_plain
s3_plain
s3_plain
```
`BACKUP`/`RESTORE` for S3 disk is done in the same way as for local disk:
```sql theme={null}
BACKUP TABLE data TO Disk('s3_plain', 'cloud_backup');
RESTORE TABLE data AS data_restored FROM Disk('s3_plain', 'cloud_backup');
```
> [!NOTE]
>
> * This disk shouldn't be used for `MergeTree` itself, only for `BACKUP`/`RESTORE`
> * If your tables are backed by S3 storage and the types of the disks are different,
> it doesn't use `CopyObject` calls to copy parts to the destination bucket, instead,
> it downloads and uploads them, which is very inefficient. In this case prefer using
> the `BACKUP ... TO S3()` syntax for this use-case.
## Usage examples of backup/restore to local disk
### Backup and restore a table
Run the following commands below to create the test database and table we will be
making a backup and restoration of in this example:
> [accordion] Setup commands
>
> Create the database and table:
>
> ```sql theme={null}
> CREATE DATABASE test_db;
>
> CREATE TABLE test_db.test_table (
> id UUID,
> name String,
> email String,
> age UInt8,
> salary UInt32,
> created_at DateTime,
> is_active UInt8,
> department String,
> score Float32,
> country String
> ) ENGINE = MergeTree()
> ORDER BY id;
> ```
>
> Preprocess and one thousand rows of random data:
>
> ```sql theme={null}
> INSERT INTO test_table (id, name, email, age, salary, created_at, is_active, department, score, country)
> SELECT
> generateUUIDv4() as id,
> concat('User_', toString(rand() % 10000)) as name,
> concat('user', toString(rand() % 10000), '@example.com') as email,
> 18 + (rand() % 65) as age,
> 30000 + (rand() % 100000) as salary,
> now() - toIntervalSecond(rand() % 31536000) as created_at,
> rand() % 2 as is_active,
> arrayElement(['Engineering', 'Marketing', 'Sales', 'HR', 'Finance', 'Operations'], (rand() % 6) + 1) as department,
> rand() / 4294967295.0 * 100 as score,
> arrayElement(['USA', 'UK', 'Germany', 'France', 'Canada', 'Australia', 'Japan', 'Brazil'], (rand() % 8) + 1) as country
> FROM numbers(1000);
> ```
>
> Next you will need to create a file specifying the backup destination at the
> path below:
>
> ```text theme={null}
> /etc/clickhouse-server/config.d/backup_disk.xml
> ```
>
> ```xml theme={null}
>
>
>
>
> local
> /backups/ -- for MacOS choose: /Users/backups/
>
>
>
>
> backups
> /backups/ -- for MacOS choose: /Users/backups/
>
>
> ```
>
> > [!NOTE]
> >
> > If clickhouse-server is running you will need to restart it for the changes to
> > take effect.
To backup the table you can run:
```sql title="Query" theme={null}
BACKUP TABLE test_db.test_table TO Disk('backups', '1.zip')
```
```response title="Response" theme={null}
┌─id───────────────────────────────────┬─status─────────┐
1. │ 065a8baf-9db7-4393-9c3f-ba04d1e76bcd │ BACKUP_CREATED │
└──────────────────────────────────────┴────────────────┘
```
The table can be restored from the backup using the following command if the table is empty:
```sql title="Query" theme={null}
RESTORE TABLE test_db.test_table FROM Disk('backups', '1.zip')
```
```response title="Response" theme={null}
┌─id───────────────────────────────────┬─status───┐
1. │ f29c753f-a7f2-4118-898e-0e4600cd2797 │ RESTORED │
└──────────────────────────────────────┴──────────┘
```
> [!NOTE]
>
> The above `RESTORE` would fail if the table `test.table` contains data.
> The setting `allow_non_empty_tables=true` allows `RESTORE TABLE` to insert data
> into non-empty tables. This will mix earlier data in the table with the data extracted from the backup.
> This setting can therefore cause data duplication in the table, and should be used with caution.
To restore the table with data already in it, run:
```sql theme={null}
RESTORE TABLE test_db.test_table FROM Disk('backups', '1.zip')
SETTINGS allow_non_empty_tables=true
```
Tables can be restored, or backed up, with new names:
```sql theme={null}
RESTORE TABLE test_db.test_table AS test_db.test_table_renamed FROM Disk('backups', '1.zip')
```
The backup archive for this backup has the following structure:
```text theme={null}
├── .backup
└── metadata
└── test_db
└── test_table.sql
```
Formats other than zip can be used. See ["Backups as tar archives"](#backups-as-tar-archives)
below for further details.
### Incremental backups to disk
A base backup in ClickHouse is the initial, full backup from which the following
incremental backups are created. Incremental backups only store the changes
made since the base backup, so the base backup must be kept available to
restore from any incremental backup. The base backup destination can be set with setting
`base_backup`.
> [!NOTE]
>
> Incremental backups depend on the base backup. The base backup must be kept available
> to be able to restore from an incremental backup.
To make an incremental backup of a table, first make a base backup:
```sql theme={null}
BACKUP TABLE test_db.test_table TO Disk('backups', 'd.zip')
```
```sql theme={null}
BACKUP TABLE test_db.test_table TO Disk('backups', 'incremental-a.zip')
SETTINGS base_backup = Disk('backups', 'd.zip')
```
All data from the incremental backup and the base backup can be restored into a
new table `test_db.test_table2` with command:
```sql theme={null}
RESTORE TABLE test_db.test_table AS test_db.test_table2
FROM Disk('backups', 'incremental-a.zip');
```
### Securing a backup
Backups written to disk can have a password applied to the file.
The password can be specified using the `password` setting.
> [!NOTE]
>
> Password protection is only supported for ZIP archives (`.zip`, `.zipx`).
> The backup path must end with `.zip` or `.zipx` for the password to be accepted.
> Using a password with any other format - including tar archives and non-archive paths - will
> result in a `BAD_ARGUMENTS` error: `Password is not applicable, backup cannot be encrypted`.
```sql theme={null}
BACKUP TABLE test_db.test_table
TO Disk('backups', 'password-protected.zip')
SETTINGS password='qwerty'
```
To restore a password-protected backup, the password must again
be specified using the `password` setting:
```sql theme={null}
RESTORE TABLE test_db.test_table
FROM Disk('backups', 'password-protected.zip')
SETTINGS password='qwerty'
```
### Backups as tar archives
Backups can be stored not only as zip archives, but also as tar archives.
The functionality is the same as for zip, except that password protection isn't
supported for tar archives. Additionally, tar archives support a variety of
compression methods.
To make a backup of a table as a tar:
```sql theme={null}
BACKUP TABLE test_db.test_table TO Disk('backups', '1.tar')
```
to restore from a tar archive:
```sql theme={null}
RESTORE TABLE test_db.test_table FROM Disk('backups', '1.tar')
```
To change the compression method, the correct file suffix should be appended to
the backup name. For example, to compress the tar archive using gzip run:
```sql theme={null}
BACKUP TABLE test_db.test_table TO Disk('backups', '1.tar.gz')
```
The supported compression file suffixes are:
* `tar.gz`
* `.tgz`
* `tar.bz2`
* `tar.lzma`
* `.tar.zst`
* `.tzst`
* `.tar.xz`
### Compression settings
The compression method and level of compression can be specified using
setting `compression_method` and `compression_level` respectively.
```sql theme={null}
BACKUP TABLE test_db.test_table
TO Disk('backups', 'filename.zip')
SETTINGS compression_method='lzma', compression_level=3
```
### Restore specific partitions
If specific partitions associated with a table need to be restored, these can be specified.
Let's create a simple partitioned table into four parts, insert some data into it and then
take a backup of only the first and fourth partitions:
> [accordion] Setup
>
> ```sql theme={null}
> CREATE IF NOT EXISTS test_db;
>
> -- Create a partitioend table
> CREATE TABLE test_db.partitioned (
> id UInt32,
> data String,
> partition_key UInt8
> ) ENGINE = MergeTree()
> PARTITION BY partition_key
> ORDER BY id;
>
> INSERT INTO test_db.partitioned VALUES
> (1, 'data1', 1),
> (2, 'data2', 2),
> (3, 'data3', 3),
> (4, 'data4', 4);
>
> SELECT count() FROM test_db.partitioned;
>
> SELECT partition_key, count()
> FROM test_db.partitioned
> GROUP BY partition_key
> ORDER BY partition_key;
> ```
>
> ```response theme={null}
> ┌─count()─┐
> 1. │ 4 │
> └─────────┘
> ┌─partition_key─┬─count()─┐
> 1. │ 1 │ 1 │
> 2. │ 2 │ 1 │
> 3. │ 3 │ 1 │
> 4. │ 4 │ 1 │
> └───────────────┴─────────┘
> ```
Run the following command to back up partitions 1 and 4:
```sql theme={null}
BACKUP TABLE test_db.partitioned PARTITIONS '1', '4'
TO Disk('backups', 'partitioned.zip')
```
Run the following command to restore partitions 1 and 4:
```sql theme={null}
RESTORE TABLE test_db.partitioned PARTITIONS '1', '4'
FROM Disk('backups', 'partitioned.zip')
SETTINGS allow_non_empty_tables=true
```
# Backing up and restoring named collections
Source: https://clickhouse.com/docs/concepts/features/backup-restore/named-collections
How BACKUP and RESTORE handle SQL-created named collections, and the settings and privileges controlling them.
SQL-created [named collections](/docs/concepts/features/configuration/server-config/named-collections)
can be exported to a backup and restored from it through the `system.named_collections` table.
An exported collection carries its values, including the secret ones, so backing collections up
and restoring them is governed by their own privileges and settings, documented on this page.
Collections defined in the server configuration files are rendered from the configuration and are
never part of a backup; restoring a backed-up collection does not overwrite a configuration-defined
collection with the same name.
## Selecting the collections
A `BACKUP` query exports the named collections when it selects the `system.named_collections`
table:
* Explicitly: `BACKUP TABLE system.named_collections`. The required privileges (see below) are
mandatory.
* Implicitly, through a `DATABASE system` or `ALL` element. By default such a backup exports the
collections only if the user may read their values, and otherwise skips them with a warning, so
that `BACKUP ALL` keeps working for a user who may back up everything but may not read the
values.
A selection excluding the table's data - `BACKUP TABLE system.named_collections EXCEPT DATA FROM
TABLE system.named_collections`, or a wildcard element with `EXCEPT DATA FROM TABLES
system.named_collections` - keeps only the table's definition in the backup. It exports no
collection and requires none of the privileges below.
## Required privileges
Exporting the collections writes their values into the backup, so it requires the same privileges
as reading the values from `system.named_collections`:
[`SHOW NAMED COLLECTIONS`](/docs/reference/statements/grant#named-collection-admin),
`SHOW NAMED COLLECTIONS SECRETS` and
[`displaySecretsInShowAndSelect`](/docs/reference/statements/grant#displaysecretsinshowandselect).
This holds for an encrypted named collections storage too: the entries are copied as ciphertext,
but any server holding the same key restores them into collections whose values can then be read.
Restoring the collections requires the
[`CREATE NAMED COLLECTION`](/docs/reference/statements/grant#named-collection-admin) privilege. It does
not require the privileges listed above: an operator may restore a backup without being able to
read it, and the errors and logs of such a restore never name a collection.
For `BACKUP ON CLUSTER` and `RESTORE ON CLUSTER` the privileges are checked against the initiating
user on the initiating host before the query is dispatched to the other hosts. During a rolling
upgrade, a host running a version with this feature never exports or restores the collections for
a query dispatched by an initiator running a version without it, because such an initiator cannot
check the privileges: the export is skipped (or, for an explicit selection, rejected), and the
restore of collection entries is rejected.
## `backup_named_collections`
Type: `Bool`. Optional; unset by default.
Controls whether a `BACKUP` query exports the named collections:
* Unset (the default): an explicit `TABLE system.named_collections` element exports them and
requires the privileges; a `DATABASE system` or `ALL` element exports them only if the user
holds the privileges, and skips them with a warning otherwise.
* `true`: the backup exports the collections whenever the query selects the table, and the missing
privileges are an error even for an implicit selection.
* `false`: the backup never exports the collections, even for an explicitly named
`system.named_collections` table, and requires none of the privileges.
```sql theme={null}
-- Fails instead of skipping the collections when the privileges are missing:
BACKUP ALL TO Disk('backups', 'everything') SETTINGS backup_named_collections = 1
-- Never exports the collections:
BACKUP ALL TO Disk('backups', 'no_collections') SETTINGS backup_named_collections = 0
```
## `restore_named_collections`
Type: `Bool`. Default: `true`.
Whether a `RESTORE` query restores the named collections held by the backup. Unlike
[`restore_table_data`](/docs/concepts/features/backup-restore/restore-settings) and its siblings, the default stays
`true` under `structure_only`: the collections are definitions, and a table whose engine
references a collection (e.g. `ENGINE = URL(collection)`) cannot be created without it.
```sql theme={null}
-- Restores everything except the named collections:
RESTORE ALL FROM Disk('backups', 'everything') SETTINGS restore_named_collections = 0
```
## `create_named_collection`
Type: `String`. Default: `'if-not-exists'`.
How `RESTORE` handles a named collection which already exists:
* `'if-not-exists'` (the default): the existing collection is kept and the backed-up one is
skipped.
* `'create'`: an existing collection fails the restore.
* `'replace'`: the existing collection is replaced with the backed-up one. A collection defined in
the server configuration cannot be replaced and fails the restore.
```sql theme={null}
RESTORE TABLE system.named_collections FROM Disk('backups', 'everything')
SETTINGS create_named_collection = 'replace'
```
## Encrypted named collections storage
With an [encrypted storage for named collections](/docs/concepts/features/configuration/server-config/named-collections)
(`local_encrypted` or `zookeeper_encrypted`), the entries are copied into the backup as the
ciphertext the storage holds, followed by an authentication code: no plaintext value appears
anywhere in the backup, and a modified entry fails the restore before anything is decrypted.
Restoring such a backup requires a server whose named collections storage is also encrypted and
holds the key the collections were encrypted with. The restored entries are re-encrypted under the
restoring server's key. A backup with encrypted entries cannot be restored into an unencrypted
storage; either configure an encrypted storage holding the key, or skip the collections with
`restore_named_collections = 0`.
# Backup and restore in ClickHouse
Source: https://clickhouse.com/docs/concepts/features/backup-restore/overview
Overview of ClickHouse backup and restore
> This section broadly covers backups and restores in ClickHouse. For a more
> detailed description of each backup method, see the pages for specific methods
> in the sidebar.
## Introduction
While [replication](/docs/reference/engines/table-engines/mergetree-family/replication) provides protection from hardware failures, it doesn't
protect against human errors: accidental deletion of data, deletion of the wrong
table or a table on the wrong cluster, and software bugs that result in incorrect
data processing or data corruption.
In many cases mistakes like these will affect all replicas. ClickHouse has built-in
safeguards to prevent some types of mistakes, for example, by [default](/docs/reference/settings/session-settings/max#max_table_size_to_drop)
you can't just drop tables with a `MergeTree` family engine containing more than
50 Gb of data. However, these safeguards don't cover all possible cases and
problems can still occur.
To effectively mitigate possible human errors, you should carefully prepare a
strategy for backing up and restoring your data **in advance**.
Each company has different resources available and business requirements, so
there's no universal solution for ClickHouse backups and restores that will fit
every situation. What works for one gigabyte of data likely won't work for tens
of petabytes of data. There are a variety of possible approaches with their own pros
and cons, which are presented in this section of the docs. It is a good idea to
use several approaches instead of just one such as to compensate for their various
shortcomings.
> [!NOTE]
>
> Keep in mind that if you backed something up and never tried to restore it,
> chances are that the restore won't work properly when you actually need it (or at
> least it will take longer than the business can tolerate). So whatever backup
> approach you choose, make sure to automate the restore process as well, and practice
> it on a spare ClickHouse cluster regularly.
The following pages detail the various backup and
restore methods available in ClickHouse:
| Page | Description |
| - | - |
| [Backup/restore using local disk or S3 disk](/docs/concepts/features/backup-restore/local-disk) | Details backup/restore to or from a local disk or S3 disk |
| [Backup/restore using S3 endpoint](/docs/concepts/features/backup-restore/s3-endpoint) | Details backup/restore to or from an S3 endpoint |
| [Backup/restore using AzureBlobStorage](/docs/concepts/features/backup-restore/azure-blob-storage) | Details backup/restore to or from Azure blob storage |
| [Alternative methods](/docs/concepts/features/backup-restore/alternative-methods) | Discusses alternative backup methods |
| [Snapshot backup](/docs/concepts/features/backup-restore/snapshot) | Lightweight snapshots for SharedMergeTree tables using cloud object storage |
Backups can:
* be [full or incremental](#backup-types)
* be [synchronous or asynchronous](#synchronous-vs-asynchronous)
* be [concurrent or non-concurrent](#concurrent-vs-non-concurrent)
* be [compressed or uncompressed](#compressed-vs-uncompressed)
* use [named collections](#using-named-collections)
* be password protected
* be taken of [system tables, log tables, or access management tables](#system-backups)
## Backup types
Backups can be either full or incremental. Full backups are a complete copy of the
data, while incremental backups are a delta of the data from the last full backup.
Full backups have the advantage of being a simple, independent (of other backups)
and reliable recovery method. However, they can take a long time to complete and
can consume a lot of space. Incremental backups, on the other hand, are more
efficient in terms of both time and space, but restoring the data requires all
the backups to be available.
Depending on your needs, you may want to use:
* **Full backups** for smaller databases or critical data.
* **Incremental backups** for larger databases or when backups need to be done frequently and cost effectively.
* **Both**, for instance, weekly full backups and daily incremental backups.
## Synchronous vs asynchronous backups
`BACKUP` and `RESTORE` commands can also be marked `ASYNC`. In this case, the
backup command returns immediately, and the backup process runs in the background.
If the commands aren't marked `ASYNC`, the backup process is synchronous and
the command blocks until the backup completes.
## Concurrent vs non-concurrent backups
By default, ClickHouse allows concurrent backups and restores. This means you
can initiate multiple backup or restore operations simultaneously. However,
there are server-level settings that let you disallow this behavior. If you set
these settings to false, only one backup or restore operation is allowed to run
on a cluster at a time. This can help avoid resource contention or potential
conflicts between operations.
To disallow concurrent backup/restore, you can use these settings respectively:
```xml theme={null}
false
false
```
The default value for both is true, so by default concurrent backup/restores are
allowed. When these settings are false on a cluster, only a single backup/restore
is allowed to run on a cluster at a time.
## Compressed vs uncompressed backups
ClickHouse backups support compression through the `compression_method` and `compression_level` settings.
When creating a backup, you can specify:
```sql theme={null}
BACKUP TABLE test.table
TO Disk('backups', 'filename.zip')
SETTINGS compression_method='lzma', compression_level=3
```
## Using named collections
Named collections allow you to store key-value pairs (like S3 credentials, endpoints, and settings) that can be reused across backup/restore operations.
They help to:
* Hide credentials from users without admin access
* Simplify commands by storing complex configuration centrally
* Maintain consistency across operations
* Avoid credential exposure in query logs
See ["named collections"](/docs/concepts/features/configuration/server-config/named-collections) for further details.
The SQL-created named collections themselves can also be exported to a backup and restored from
it through the `system.named_collections` table; see
["Backing up and restoring named collections"](/docs/concepts/features/backup-restore/named-collections).
## Backing up system, log or access management tables
System tables can also be included in your backup and restore workflows, but their
inclusion depends on your specific use case.
System tables that store historic data, such as those with a `_log` suffix (e.g.,
`query_log`, `part_log`), can be backed up and restored like any other table.
If your use case relies on analyzing historic data - for example, using `query_log`
to track query performance or debug issues - it's recommended to include these
tables in your backup strategy. However, if historic data from these tables is
not required, they can be excluded to save backup storage space.
System tables related to access management, such as users, roles, row\_policies,
settings\_profiles, and quotas, receive special treatment during backup and restore operations.
When these tables are included in a backup, their content is exported to a special
`accessXX.txt` file, which encapsulates the equivalent SQL statements for creating
and configuring the access entities. Upon restoration, the restore process
interprets these files and re-applies the SQL commands to recreate the users,
roles, and other configurations. This feature ensures that the access control
configuration of a ClickHouse cluster can be backed up and restored as part of
the cluster's overall setup.
This functionality only works for configurations managed through SQL commands
(referred to as ["SQL-driven Access Control and Account Management"](/docs/concepts/features/security/access-rights#enabling-access-control)).
Access configurations defined in ClickHouse server configuration files (e.g. `users.xml`)
aren't included in backups and can't be restored through this method.
> [!NOTE]
>
> In ClickHouse versions before 26.10, row policies, settings profiles and quotas restored with `RESTORE` are
> listed in `system.row_policies`, `system.settings_profiles` and `system.quotas`, but aren't applied to queries
> right away. Run [`SYSTEM RELOAD USERS`](/docs/reference/statements/system#reload-users) after the restore completes
> to apply them.
## General syntax
```sql theme={null}
-- core commands
BACKUP | RESTORE
--- what to backup/restore (or exclude)
TABLE [db.]table_name [AS [db.]table_name_in_backup] |
DICTIONARY [db.]dictionary_name [AS [db.]name_in_backup] |
DATABASE database_name [AS database_name_in_backup] |
TEMPORARY TABLE table_name [AS table_name_in_backup] |
VIEW view_name [AS view_name_in_backup] |
[EXCEPT TABLES ...] |
[EXCEPT DATA FROM {TABLE|TABLES} ...] | -- BACKUP only, rejected on RESTORE
ALL [EXCEPT {TABLES|DATABASES}...] } [,...]
---
[ON CLUSTER 'cluster_name']
--- where to backup or restore to or from
TO|FROM
File('/') |
Disk('', '/') |
S3('/', '', '', '') |
AzureBlobStorage('/', '', '', '', '')
--- additional settings
[SETTINGS ...]
[ASYNC]
```
**See ["command summary"](/docs/concepts/features/backup-restore/overview#command-summary) for more details
of each command.**
### Command summary
Each of the commands above is detailed below:
| **Command** | **Description** | |
| - | - | - |
| `BACKUP` | Creates a backup of specified objects | |
| `RESTORE` | Restores objects from a backup | |
| `TABLE [db.]table_name [AS [db.]table_name_in_backup]` | Backs up/restores a specific table (can be renamed) | |
| `[PARTITION[S] partition_expr [,...]]` | Only backup/restore specific partitions of the table | |
| `DICTIONARY [db.]dictionary_name [AS [db.]name_in_backup]` | Backs up/restores a dictionary object | |
| `DATABASE database_name [AS database_name_in_backup]` | Backs up/restores an entire database (can be renamed) | |
| `TEMPORARY TABLE table_name [AS table_name_in_backup]` | Backs up/restores a temporary table (can be renamed) | |
| `VIEW view_name [AS view_name_in_backup]` | Backs up/restores a view (can be renamed) | |
| `[EXCEPT TABLES ...]` | Exclude specific tables when backing up a database | |
| `[EXCEPT DATA FROM {TABLE\|TABLES} ...]` | **(BACKUP only)** Backs up a table's definition without its rows. Differs from `EXCEPT TABLES`, which omits the table entirely. Scoped to the element it is written on: see [Backing up a table without its data](#backup-without-data). | |
| `ALL` | Backs up/restores everything (all databases, tables, etc.). Prior to version 23.4 of ClickHouse, `ALL` was only applicable to the `RESTORE` command. | |
| `[EXCEPT {TABLES\|DATABASES}...]` | Exclude specific tables or databases when using `ALL` | |
| `[ON CLUSTER 'cluster_name']` | Execute the backup/restore across a ClickHouse cluster | |
| `TO\|FROM` | Direction: `TO` for backup destination, `FROM` for restore source | |
| `File('/')` | Store to/restore from local file system | |
| `Disk('', '/')` | Store to/restore from a configured disk | |
| `S3('/', '', '')` | Store to/restore from Amazon S3 or S3-compatible storage | |
| `[SETTINGS ...]` | See below for complete list of settings | |
| `[ASYNC]` | Makes the operation run asynchronously (returns immediately with an ID you can monitor) | |
### Backing up a table without its data
`EXCEPT DATA FROM {TABLE|TABLES}` puts a table's definition in the backup without its rows. It is useful
for tables whose contents can be regenerated after a restore, for example the target tables of materialized
views.
The clause decides what `BACKUP` writes; it does not change what `RESTORE` is able to do with the result. For
a table that `RESTORE` can recreate, a definition without rows restores as an empty table. It does not make a
table restorable that would not be restorable anyway - whether a definition in the backup can be turned back
into a table is up to the engine, and some engines refuse it outright: see
[Engines that `RESTORE` cannot recreate](#engines-restore-cannot-recreate).
The clause is scoped to the element it is written on:
* On a `DATABASE` or `ALL` element it may name any table that element backs up:
```sql theme={null}
BACKUP DATABASE mydb EXCEPT DATA FROM TABLES events, metrics TO Disk('backups', 'mydb/')
```
In this list, back-quote a table or database named `table`, `dictionary`, `view`, `database` or `all`: after a
comma, an unquoted one is read as the start of the next element.
* On a single-object element (`TABLE`, `DICTIONARY`, `VIEW`, `TEMPORARY TABLE`) the element's own object is
the only one in scope, so the clause must name it and nothing else. Give every table its own element and
its own clause:
```sql theme={null}
-- Backs up both tables, without the data of `t1`
BACKUP TABLE mydb.t1 EXCEPT DATA FROM TABLE mydb.t1, TABLE mydb.t2 TO Disk('backups', 'tables/')
-- Rejected: the clause of the `mydb.t2` element names a different table
BACKUP TABLE mydb.t1, TABLE mydb.t2 EXCEPT DATA FROM TABLE mydb.t1 TO Disk('backups', 'tables/')
```
If a table is named both by an element of its own and by the `EXCEPT DATA FROM` clause of a wider element,
its data **is** backed up: an element asking for a table asks for its data. This matches `EXCEPT TABLES`,
where a table named explicitly is backed up even though a wider element excludes it.
The data of the inner table of a materialized view (or of the inner tables of a `TimeSeries` table) is
excluded through the outer object's name; inner table names are not accepted by the clause.
`EXCEPT DATA FROM {TABLE|TABLES}` is only valid for `BACKUP`. `RESTORE` rejects it, rather than accepting it
as a no-op.
#### Engines that `RESTORE` cannot recreate
A few engines cannot be recreated by `RESTORE` at all, with or without `EXCEPT DATA FROM {TABLE|TABLES}`. For
those tables the clause still controls the size of the backup, but it does not give you a table that comes
back empty - nothing comes back in place, because `RESTORE` refuses to create the table.
A standalone `MaterializedPostgreSQL` table is the case to know about. It would start replicating from the
live PostgreSQL source as soon as it was created, mixing the backup snapshot with the current remote state, so
`RESTORE` rejects it in the storage factory - before the table is created - with `NOT_IMPLEMENTED`. This is a
property of the engine and applies to an ordinary full backup just the same.
The supported flow is to restore under a different name, into a `ReplacingMergeTree` that you created
beforehand with the structure of the nested table (including the `_sign` and `_version` columns):
```sql theme={null}
-- Back up the definition without the rows: they are replicated from PostgreSQL and can be replicated again
BACKUP DATABASE mydb EXCEPT DATA FROM TABLE mydb.pg_table TO Disk('backups', 'mydb/')
CREATE TABLE mydb.plain_target
(
key Int32,
value Int32,
_sign Int8 MATERIALIZED 1,
_version UInt64 MATERIALIZED 1
)
ENGINE = ReplacingMergeTree(_version) ORDER BY key
-- Succeeds, and leaves `mydb.plain_target` empty because the backup carries no rows for `mydb.pg_table`
RESTORE TABLE mydb.pg_table AS mydb.plain_target FROM Disk('backups', 'mydb/')
SETTINGS allow_different_table_def = 1
```
`allow_different_table_def` is required because the definition stored in the backup uses the
`MaterializedPostgreSQL` engine, which is deliberately being restored into a plain `ReplacingMergeTree`.
Without `EXCEPT DATA FROM TABLE` the same `RESTORE` brings the rows across instead.
### Settings
**Generic backup/restore settings**
| Setting | Description | Default value |
| - | - | - |
| `id` | ID of backup or restore operation, randomly generated UUID is used if not specified. If there's already a running operation with the same ID, an exception is thrown. | |
| `compression_method` | Specifies the compression method for the backup. See section ["column compression codecs"](/docs/reference/statements/create/table/codec) | |
| `compression_level` | Specifies the compression level for the backup | |
| `password` | Password for the backup archive. Only supported for ZIP archives (`.zip`, `.zipx`). | |
| `base_backup` | The destination of the base backup used for incremental backups. For example: `Disk('backups', '1.zip')` | |
| `use_same_password_for_base_backup` | Whether base backup archive should inherit the password from the query. | |
| `structure_only` | If enabled, only backs up or restores the CREATE statements without the actual table data. | |
| `restore_table_data` | Whether to restore the data of tables. Only applicable to the `RESTORE` command. When not set it defaults to `NOT structure_only`; setting it explicitly overrides `structure_only`. See ["Restore settings"](/docs/concepts/features/backup-restore/restore-settings). | |
| `restore_access_entities` | Whether to restore access entities (users, roles, settings profiles, row policies, quotas). Only applicable to the `RESTORE` command. When not set it defaults to `NOT structure_only`; setting it explicitly overrides `structure_only`. See ["Restore settings"](/docs/concepts/features/backup-restore/restore-settings). | |
| `restore_functions` | Whether to restore user-defined functions. Only applicable to the `RESTORE` command. When not set it defaults to `NOT structure_only`; setting it explicitly overrides `structure_only`. See ["Restore settings"](/docs/concepts/features/backup-restore/restore-settings). | |
| `backup_named_collections` | Whether to export the SQL-created named collections when the `BACKUP` command selects the `system.named_collections` table. When not set, an explicit `TABLE system.named_collections` element exports them, while a `DATABASE system` or `ALL` element exports them only if the user may read their values and skips them with a warning otherwise. `1` makes the missing privileges an error for an implicit selection too; `0` never exports them. See ["Named collections"](/docs/concepts/features/backup-restore/named-collections). | |
| `restore_named_collections` | Whether to restore the SQL-created named collections held by the backup. Only applicable to the `RESTORE` command. The default stays `1` under `structure_only`, because a restored table engine may reference a collection in its `CREATE`. See ["Named collections"](/docs/concepts/features/backup-restore/named-collections). | `1` |
| `create_named_collection` | How the `RESTORE` command handles a named collection which already exists: `'if-not-exists'` keeps the existing collection, `'create'` fails the restore, `'replace'` replaces the existing collection with the backed up one. See ["Named collections"](/docs/concepts/features/backup-restore/named-collections). | `'if-not-exists'` |
| `fsync_backup_files` | Whether to `fsync` the backup's files, the `.backup` manifest and the containing directories to local storage before the backup is reported as created, so that an acknowledged backup survives power loss. Applies to `File` destinations and to `Disk` destinations whose disk stores plain local files. On an object-storage disk (`S3`/`Azure`) the uploaded objects are already durable once uploaded, and if the disk keeps its metadata in local files (`metadata_type = local`, the default), those metadata files and their directories are fsynced the same way, since the backup is reached through them. A local object-storage disk (`local_blob_storage`) writes its blobs without a sync, so the setting cannot make a backup on it durable. The guarantee covers `backups.allowed_path` and everything the backup writes below it; directories above `backups.allowed_path` are synced only when they can be opened for `fsync`, so if one of them was itself created without being made durable the backup tree can still be lost as a whole. For `BACKUP ON CLUSTER`, the guarantee is only given for a cluster whose hosts all know this setting: a setting left at its default is not forwarded to the other hosts, so a host running a version that predates `fsync_backup_files` writes its share of the backup on the pre-`fsync` path. Naming the setting explicitly in the query makes such a host reject its internal query instead of silently downgrading its share, but the initiator is not told about that rejection: it keeps waiting for the host, so such a `BACKUP ON CLUSTER` does not fail on its own. Run it with `async = 1` and cancel it with `KILL QUERY` if it does not finish. Disable to trade durability for backup speed. | `1` |
| `storage_policy` | Storage policy for the tables being restored. See ["using multiple block devices for data storage](/docs/reference/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-multiple-volumes). Only applicable to the `RESTORE` command. Applies only to tables with an engine from the `MergeTree` family. | |
| `allow_non_empty_tables` | Allows `RESTORE TABLE` to insert data into non-empty tables. This will mix earlier data in the table with the data extracted from the backup. This setting can therefore cause data duplication in the table, use with caution. | `0` |
| `backup_restore_keeper_max_retries` | Max retries for \[Zoo]Keeper operations in the middle of a BACKUP or RESTORE operation. Should be big enough so the whole operation won't fail because of a temporary \[Zoo]Keeper failure. | `1000` |
| `backup_restore_keeper_retry_initial_backoff_ms` | Initial backoff timeout for \[Zoo]Keeper operations during backup or restore | `100` |
| `backup_restore_keeper_retry_max_backoff_ms` | Max backoff timeout for \[Zoo]Keeper operations during backup or restore | `5000` |
| `backup_restore_failure_after_host_disconnected_for_seconds` | If a host during a `BACKUP ON CLUSTER` or `RESTORE ON CLUSTER` operation doesn't recreate its ephemeral 'alive' node in ZooKeeper for this amount of time then the whole backup or restore is considered as failed. This value should be bigger than any reasonable time for a host to reconnect to ZooKeeper after a failure. Zero means unlimited. | `3600` |
| `backup_restore_keeper_max_retries_while_initializing` | Max retries for \[Zoo]Keeper operations during the initialization of a `BACKUP ON CLUSTER` or `RESTORE ON CLUSTER` operation. | `20` |
| `backup_restore_keeper_max_retries_while_handling_error` | Max retries for \[Zoo]Keeper operations while handling an error of a `BACKUP ON CLUSTER` or `RESTORE ON CLUSTER` operation. | `20` |
| `backup_restore_finish_timeout_after_error_sec` | How long the initiator should wait for other host to react to the 'error' node and stop their work on the current `BACKUP ON CLUSTER` or `RESTORE ON CLUSTER` operation. | `180` |
| `backup_restore_keeper_value_max_size` | Maximum size of data of a \[Zoo]Keeper's node during backup | `1048576` |
| `backup_restore_batch_size_for_keeper_multi` | Maximum size of batch for multi request to \[Zoo]Keeper during backup or restore | `1000` |
| `backup_restore_batch_size_for_keeper_multiread` | Maximum size of batch for multiread request to \[Zoo]Keeper during backup or restore | `10000` |
| `backup_restore_keeper_fault_injection_probability` | Approximate probability of failure for a keeper request during backup or restore. Valid value is in interval `[0.0f, 1.0f]` | `0` |
| `backup_restore_keeper_fault_injection_seed` | `0` for a random seed, otherwise the setting value | `0` |
| `backup_restore_s3_retry_attempts` | Setting for Aws::Client::RetryStrategy, Aws::Client does retries itself, 0 means no retries. It takes place only for backup/restore. | `1000` |
| `max_backup_bandwidth` | The maximum read speed in bytes per second for particular backup on server. Zero means unlimited. | `0` |
| `max_backups_io_thread_pool_size` | ClickHouse uses threads from the Backups IO Thread pool to do S3 backup IO operations. `max_backups_io_thread_pool_size` limits the maximum number of threads in the pool. | `1000` |
| `max_backups_io_thread_pool_free_size` | If the number of **idle** threads in the Backups IO Thread pool exceeds `max_backup_io_thread_pool_free_size`, ClickHouse will release resources occupied by idling threads and decrease the pool size. Threads can be created again if necessary. | `0` |
| `backups_io_thread_pool_queue_size` | The maximum number of jobs that can be scheduled on the Backups IO Thread pool. It is recommended to keep this queue unlimited due to the current S3 backup logic. Note: A value of `0` (default) means unlimited. | `0` |
| `backup_threads` | The maximum number of threads to execute `BACKUP` requests. | |
| `max_backup_bandwidth_for_server` | The maximum read speed in bytes per second for all backups on server. Zero means unlimited. | `0` |
| `shutdown_wait_backups_and_restores` | If set to true ClickHouse will wait for running backups and restores to finish before shutdown. | `1` |
**S3 specific settings**
| Setting | Description | Default value |
| - | - | - |
| `use_same_s3_credentials_for_base_backup` | Whether base backup to S3 should inherit credentials from the query. Only works with `S3`. | |
| `s3_storage_class` | The storage class used for S3 backup. For example: `STANDARD` | |
**Azure specific settings**
| Setting | Description | Default value | | |
| - | - | - | - | - |
| `azure_attempt_to_create_container` | When using Azure Blob Storage, whether to attempt creating the specified container if it doesn't exist. | `true` | | |
## Administration and troubleshooting
The backup command returns an `id` and `status`, and that `id` can be used to
get the status of the backup. This is very useful to check the progress of long
`ASYNC` backups. The example below shows a failure that happened when trying to
overwrite an existing backup file:
```sql theme={null}
BACKUP TABLE helloworld.my_first_table TO Disk('backups', '1.zip') ASYNC
```
```response theme={null}
┌─id───────────────────────────────────┬─status──────────┐
│ 7678b0b3-f519-4e6e-811f-5a0781a4eb52 │ CREATING_BACKUP │
└──────────────────────────────────────┴─────────────────┘
1 row in set. Elapsed: 0.001 sec.
```
```sql theme={null}
SELECT
*
FROM system.backups
WHERE id='7678b0b3-f519-4e6e-811f-5a0781a4eb52'
FORMAT Vertical
```
```response highlight={5,9} theme={null}
Row 1:
──────
id: 7678b0b3-f519-4e6e-811f-5a0781a4eb52
name: Disk('backups', '1.zip')
status: BACKUP_FAILED
num_files: 0
uncompressed_size: 0
compressed_size: 0
error: Code: 598. DB::Exception: Backup Disk('backups', '1.zip') already exists. (BACKUP_ALREADY_EXISTS) (version 22.8.2.11 (official build))
start_time: 2022-08-30 09:21:46
end_time: 2022-08-30 09:21:46
1 row in set. Elapsed: 0.002 sec.
```
Along with the [`system.backups`](/docs/reference/system-tables/backups) table, all backup and restore operations are also tracked in the system log table
[`system.backup_log`](/docs/reference/system-tables/backup_log):
```sql theme={null}
SELECT *
FROM system.backup_log
WHERE id = '7678b0b3-f519-4e6e-811f-5a0781a4eb52'
ORDER BY event_time_microseconds ASC
FORMAT Vertical
```
```response highlight={26} theme={null}
Row 1:
──────
event_date: 2023-08-18
event_time_microseconds: 2023-08-18 11:13:43.097414
id: 7678b0b3-f519-4e6e-811f-5a0781a4eb52
name: Disk('backups', '1.zip')
status: CREATING_BACKUP
error:
start_time: 2023-08-18 11:13:43
end_time: 1970-01-01 03:00:00
num_files: 0
total_size: 0
num_entries: 0
uncompressed_size: 0
compressed_size: 0
files_read: 0
bytes_read: 0
Row 2:
──────
event_date: 2023-08-18
event_time_microseconds: 2023-08-18 11:13:43.174782
id: 7678b0b3-f519-4e6e-811f-5a0781a4eb52
name: Disk('backups', '1.zip')
status: BACKUP_FAILED
error: Code: 598. DB::Exception: Backup Disk('backups', '1.zip') already exists. (BACKUP_ALREADY_EXISTS) (version 23.8.1.1)
start_time: 2023-08-18 11:13:43
end_time: 2023-08-18 11:13:43
num_files: 0
total_size: 0
num_entries: 0
uncompressed_size: 0
compressed_size: 0
files_read: 0
bytes_read: 0
2 rows in set. Elapsed: 0.075 sec.
```
# Restore settings
Source: https://clickhouse.com/docs/concepts/features/backup-restore/restore-settings
Settings of the RESTORE query that control which categories of objects are restored from a backup.
The [`RESTORE`](/docs/concepts/features/backup-restore/overview#syntax) query accepts
settings in its `SETTINGS` clause that control which categories of objects are read
from a backup. This page documents the settings that select what is restored: table
data, access entities, and user-defined functions.
## `structure_only`
Type: `Bool`. Default: `false`.
When set to `true`, only `CREATE` queries are read from the backup, without the
contents of the restored objects. It is a convenient shortcut that sets the
effective defaults of all three category settings below to `false`:
* `restore_table_data`
* `restore_access_entities`
* `restore_functions`
## `restore_table_data`
Type: `Bool`. Optional; when not set it defaults to `!structure_only` (i.e. `true`
unless `structure_only` is set).
Whether to restore the data of tables. When set explicitly it overrides
`structure_only`: set it to `true` to restore table data even when
`structure_only=true`, or to `false` to skip table data even when `structure_only`
is not set.
## `restore_access_entities`
Type: `Bool`. Optional; when not set it defaults to `!structure_only`.
Whether to restore access entities — users, roles, settings profiles, row policies,
and quotas. When set explicitly it overrides `structure_only`.
## `restore_functions`
Type: `Bool`. Optional; when not set it defaults to `!structure_only`.
Whether to restore user-defined functions. When set explicitly it overrides
`structure_only`.
## Effective behavior
Each category is restored when its corresponding setting is `true`. When a category
setting is not specified, its value is derived from `structure_only`: all three are
`true` by default, and all three become `false` when `structure_only=true`.
| `structure_only` | `restore_*` not set | `restore_* = true` | `restore_* = false` |
| :-: | :-: | :-: | :-: |
| `false` | restored | restored | skipped |
| `true` | skipped | restored | skipped |
This is fully backward compatible: a `RESTORE` query that does not set any of the
new settings behaves exactly as before.
## Examples
Restore table definitions and access entities, but not table data — for example
to set up an environment for testing or replaying queries without copying data:
```sql theme={null}
RESTORE ALL FROM backup_name
SETTINGS structure_only = true, restore_access_entities = true
```
Restore only table data, without re-creating access entities or user-defined
functions:
```sql theme={null}
RESTORE ALL FROM backup_name
SETTINGS structure_only = true, restore_table_data = true
```
Restore everything except user-defined functions:
```sql theme={null}
RESTORE ALL FROM backup_name
SETTINGS restore_functions = false
```
# BACKUP / RESTORE to or from an S3 endpoint
Source: https://clickhouse.com/docs/concepts/features/backup-restore/s3-endpoint
Overview of ClickHouse backup and restore
This article covers backing up or restoring backups to/from an S3 bucket
via an S3 endpoint.
## Syntax
```sql theme={null}
-- core commands
BACKUP | RESTORE
--- what to backup/restore (or exclude)
TABLE [db.]table_name [AS [db.]table_name_in_backup] |
DICTIONARY [db.]dictionary_name [AS [db.]name_in_backup] |
DATABASE database_name [AS database_name_in_backup] |
TEMPORARY TABLE table_name [AS table_name_in_backup] |
VIEW view_name [AS view_name_in_backup] |
[EXCEPT TABLES ...] |
[EXCEPT DATA FROM {TABLE|TABLES} ...] | -- BACKUP only, rejected on RESTORE
ALL [EXCEPT {TABLES|DATABASES}...] } [,...]
---
[ON CLUSTER 'cluster_name']
--- where to backup or restore to or from
TO|FROM
File('/') |
Disk('', '/') |
S3('/', '', '', '') |
AzureBlobStorage('/', '', '', '', '')
--- additional settings
[SETTINGS ...]
[ASYNC]
```
**See ["command summary"](/docs/concepts/features/backup-restore/overview#command-summary) for more details
of each command.**
## Usage example
### Incremental backup to an S3 endpoint
In this example, we will create a backup to an S3 endpoint and then restore from it
again.
> [!NOTE]
>
> For an explanation of the differences between a full backup and an incremental
> backup, see ["Backup types"](/docs/concepts/features/backup-restore/overview#backup-types)
You will need the following information to use this method:
| Parameter | Example |
| - | - |
| An S3 endpoint | `https://backup-ch-docs.s3.us-east-1.amazonaws.com/backups/` |
| Access key ID | `BKIOZLE2VYN3VXXTP9RC` |
| Secret access key | `40bwYnbqN7xU8bVePaUCh3+YEyGXu8UOMV9ANpwL` |
> [!TIP]
>
> Creating an S3 bucket is covered in section ["use S3 Object Storage as a ClickHouse disk"](/docs/integrations/connectors/data-ingestion/AWS/integrating-s3-with-clickhouse#configuring-s3-for-clickhouse-use)
The destination for a backup is specified as:
```sql theme={null}
S3('/', '', '', '')
```
**1. Setup**
Create the following database and table and insert some random data into it:
```sql theme={null}
CREATE DATABASE IF NOT EXISTS test_db;
CREATE TABLE test_db.test_table
(
`key` Int,
`value` String,
`array` Array(String)
)
ENGINE = MergeTree
ORDER BY tuple()
```
```sql theme={null}
INSERT INTO test_db.test_table SELECT *
FROM generateRandom('key Int, value String, array Array(String)')
LIMIT 1000
```
**2. Create a base backup**
Incremental backups require a *base* backup to start from. The first parameter of
the S3 destination is the S3 endpoint followed by the directory within the bucket
to use for this backup. In this example the directory is named `my_backup`.
Run the following command to create the base backup:
```sql theme={null}
BACKUP TABLE test_db.test_table TO S3(
'https://backup-ch-docs.s3.us-east-1.amazonaws.com/backups/base_backup',
'',
''
)
```
```response theme={null}
┌─id───────────────────────────────────┬─status─────────┐
│ de442b75-a66c-4a3c-a193-f76f278c70f3 │ BACKUP_CREATED │
└──────────────────────────────────────┴────────────────┘
```
**3. Add more data**
Incremental backups are populated with the difference between the base backup and
the current content of the table being backed up. Add more data before taking the
incremental backup:
```sql theme={null}
INSERT INTO test_db.test_table SELECT *
FROM generateRandom('key Int, value String, array Array(String)')
LIMIT 100
```
**4. Take an incremental backup**
This backup command is similar to the base backup, but adds `SETTINGS base_backup` and the location of the base backup. Note that the destination for the incremental backup isn't the same directory as the base, it is the same endpoint with a different target directory within the bucket. The base backup is in `my_backup`, and the incremental will be written to `my_incremental`:
```sql theme={null}
BACKUP TABLE test_db.test_table TO S3(
'https://backup-ch-docs.s3.us-east-1.amazonaws.com/backups/incremental_backup',
'',
''
)
SETTINGS base_backup = S3(
'https://backup-ch-docs.s3.us-east-1.amazonaws.com/backups/base_backup',
'',
''
)
```
```response theme={null}
┌─id───────────────────────────────────┬─status─────────┐
│ f6cd3900-850f-41c9-94f1-0c4df33ea528 │ BACKUP_CREATED │
└──────────────────────────────────────┴────────────────┘
```
**5. Restore from the incremental backup**
This command restores the incremental backup into a new table, `test_table_restored`.\
Note that when an incremental backup is restored, the base backup is also included.
Specify only the **incremental backup** when restoring:
```sql theme={null}
RESTORE TABLE data AS test_db.test_table_restored FROM S3(
'https://backup-ch-docs.s3.us-east-1.amazonaws.com/backups/incremental_backup',
'',
''
)
```
```response theme={null}
┌─id───────────────────────────────────┬─status───┐
│ ff0c8c39-7dff-4324-a241-000796de11ca │ RESTORED │
└──────────────────────────────────────┴──────────┘
```
**6. Verify the count**
There were two inserts into the original table `data`, one with 1,000 rows and one with 100 rows, for a total of 1,100.
Verify that the restored table has 1,100 rows:
```sql theme={null}
SELECT count()
FROM test_db.test_table_restored
```
```response theme={null}
┌─count()─┐
│ 1100 │
└─────────┘
```
**7. Verify the content**
This compares the content of the original table, `test_table` with the restored table `test_table_restored`:
```sql theme={null}
SELECT throwIf((
SELECT groupArray(tuple(*))
FROM test_db.test_table
) != (
SELECT groupArray(tuple(*))
FROM test_db.test_table_restored
), 'Data does not match after BACKUP/RESTORE')
```
# Snapshot backup and restore
Source: https://clickhouse.com/docs/concepts/features/backup-restore/snapshot
How to create and restore lightweight snapshots of cloud-native tables using cloud object storage.
Snapshot backup is a lightweight backup mode for cloud-native table engines. Instead of copying data, it writes per-part lock nodes into ClickHouse Keeper. These locks prevent the server from deleting the referenced object storage parts for as long as the snapshot is retained. The backup then records the object storage references rather than physically copying any data, making snapshot creation fast regardless of table size.
The lightweight path applies to [SharedMergeTree](/docs/products/cloud/features/infrastructure/shared-merge-tree), SharedSet, and SharedJoin tables. For all other engine types — such as Log or Memory — the backup falls back to a standard copy-based backup automatically.
## Create a snapshot
Snapshot backup uses the standard [`BACKUP`](/docs/concepts/features/backup-restore/overview#syntax) command with `experimental_lightweight_snapshot = true`. The `id` setting is required — it names the snapshot and is used to reference it in unlock and observability commands:
```sql theme={null}
BACKUP { TABLE [db.]table_name | DATABASE db_name | ALL [EXCEPT {TABLES | DATABASES} ...] }
TO { S3(...) | AzureBlobStorage(...) }
SETTINGS experimental_lightweight_snapshot = true, id = ''
```
The command returns the `id` and `status`, and the `id` can be used to track the operation in [`system.backups`](/docs/reference/system-tables/backups).
Backup a single table to S3:
```sql theme={null}
BACKUP TABLE mydb.events
TO S3('https://my-bucket.s3.us-east-1.amazonaws.com/snapshots/events/', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY')
SETTINGS experimental_lightweight_snapshot = true, id = 'events_snapshot_1'
```
Backup a full database:
```sql theme={null}
BACKUP DATABASE mydb
TO S3('https://my-bucket.s3.us-east-1.amazonaws.com/snapshots/mydb/', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY')
SETTINGS experimental_lightweight_snapshot = true, id = 'mydb_snapshot_1'
```
Backup all tables, skipping one:
```sql theme={null}
BACKUP ALL
EXCEPT TABLES mydb.staging_table
TO S3('https://my-bucket.s3.us-east-1.amazonaws.com/snapshots/full/', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY')
SETTINGS experimental_lightweight_snapshot = true, id = 'full_snapshot_1'
```
Backup table DDL without data (restores an empty table with the correct schema):
```sql theme={null}
BACKUP TABLE mydb.events
EXCEPT DATA FROM TABLE mydb.events
TO S3('https://my-bucket.s3.us-east-1.amazonaws.com/snapshots/events_schema/', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY')
SETTINGS experimental_lightweight_snapshot = true, id = 'events_schema_snapshot'
```
The same commands work with Azure Blob Storage:
```sql theme={null}
BACKUP TABLE mydb.events
TO AzureBlobStorage('DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=...', 'my-container', 'snapshots/events/')
SETTINGS experimental_lightweight_snapshot = true, id = 'events_snapshot_1'
```
## Restore to the same service
Because a snapshot stores references to object storage files rather than copies of the data, restoring to a new or different ClickHouse service requires access to the original object storage. For that reason, cross-service restore is not supported via SQL — it is only available through the UI. By SQL, you can restore a snapshot to the same service from an external backup bucket using `snapshot_from_current_service = 1`. This reads objects directly via the destination disk instead of going through a remote snapshot reader:
```sql theme={null}
RESTORE TABLE mydb.events AS mydb.events_restored
FROM S3('https://my-bucket.s3.us-east-1.amazonaws.com/snapshots/events/', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY')
SETTINGS snapshot_from_current_service = 1
```
The `AS` clause restores into a new table name, leaving the original table intact. To overwrite the original table, drop it first:
```sql theme={null}
DROP TABLE mydb.events;
RESTORE TABLE mydb.events
FROM S3('https://my-bucket.s3.us-east-1.amazonaws.com/snapshots/events/', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY')
SETTINGS snapshot_from_current_service = 1
```
## Unlock a snapshot
Each snapshot holds locks in ClickHouse Keeper that prevent the referenced object storage files from being garbage collected. After a restore completes — or when a snapshot is no longer needed — unlock it to release those locks.
There are two forms: a system-level unlock that removes all locks for the snapshot at once, and a per-table unlock that removes the lock for a single table while leaving the rest of the snapshot intact.
**System-level unlock** — removes all locks for the snapshot:
```sql theme={null}
SYSTEM UNLOCK SNAPSHOT ''
FROM S3('https://my-bucket.s3.us-east-1.amazonaws.com/snapshots/events/', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY')
```
**Per-table unlock** — removes the lock for one table only:
```sql theme={null}
ALTER TABLE mydb.events UNLOCK SNAPSHOT ''
FROM S3('https://my-bucket.s3.us-east-1.amazonaws.com/snapshots/events/', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY')
```
The `FROM` clause is optional when the snapshot destination was stored in Keeper at creation time (visible in the `info` column of `system.snapshot_locks`):
```sql theme={null}
SYSTEM UNLOCK SNAPSHOT ''
-- or per-table:
ALTER TABLE mydb.events UNLOCK SNAPSHOT ''
```
After unlocking, the corresponding row disappears from `system.snapshot_locks`, and parts no longer referenced by other snapshots drop out of `system.snapshot_parts`.
## Observability
### system.backups
All snapshot operations appear in [`system.backups`](/docs/reference/system-tables/backups) alongside regular backup and restore operations. Query it with the `id` you set (or the UUID returned by the command):
```sql theme={null}
SELECT id, name, status, error, start_time, end_time, num_files, uncompressed_size, compressed_size
FROM system.backups
WHERE id = 'events_snapshot_1'
FORMAT Vertical
```
```response theme={null}
Row 1:
──────
id: events_snapshot_1
name: S3('https://my-bucket.s3.us-east-1.amazonaws.com/snapshots/events/', '[HIDDEN]')
status: BACKUP_CREATED
error:
start_time: 2024-06-01 10:00:00
end_time: 2024-06-01 10:00:03
num_files: 42
uncompressed_size: 1073741824
compressed_size: 0
```
### system.snapshot\_locks
`system.snapshot_locks` shows the committed snapshots currently registered in Keeper. When a snapshot is committed, a Keeper node is created at `/clickhouse/snapshot/committed/{snapshot_id}`. Before deleting any data part, the server checks whether a committed snapshot holds a lock on that part. If one does, deletion is skipped. The lock persists until you explicitly unlock the snapshot.
```sql theme={null}
SELECT *
FROM system.snapshot_locks
```
| Column | Type | Description |
| - | - | - |
| `id` | `String` | Snapshot ID |
| `info` | `String` | Snapshot destination, e.g. `S3('...')` |
| `ctime` | `DateTime` | When this lock was created in Keeper |
| `lock_path` | `String` | Keeper path for this lock |
Each row represents one committed snapshot. If you see locks for snapshots that no longer have a valid backup destination, run `SYSTEM UNLOCK SNAPSHOT` to clean them up.
To check whether a specific snapshot lock is present:
```sql theme={null}
SELECT id, info, lock_path
FROM system.snapshot_locks
WHERE id = 'events_snapshot_1'
```
### system.snapshot\_parts
`system.snapshot_parts` shows the data parts currently pinned by at least one snapshot lock. For each locked part, a Keeper node exists at `/clickhouse/snapshot/{table_uuid}/{part_name}` containing the part's compressed and uncompressed size. This table reads those nodes to show which parts are currently protected from deletion.
```sql theme={null}
SELECT *
FROM system.snapshot_parts
ORDER BY data_compressed_bytes DESC
LIMIT 20
```
| Column | Type | Description |
| - | - | - |
| `name` | `String` | Data part name |
| `table_id` | `String` | UUID of the table this part belongs to |
| `data_compressed_bytes` | `UInt64` | Compressed size of this part |
| `data_uncompressed_bytes` | `UInt64` | Uncompressed size of this part |
| `snapshots_size` | `UInt64` | Number of snapshots currently holding a lock on this part |
Parts with `snapshots_size > 1` are referenced by multiple snapshots and won't be removed from object storage until all holding snapshots are unlocked.
To check total pinned storage:
```sql theme={null}
SELECT
formatReadableSize(sum(data_compressed_bytes)) AS total_pinned_compressed,
formatReadableSize(sum(data_uncompressed_bytes)) AS total_pinned_uncompressed,
count() AS parts_count
FROM system.snapshot_parts
```
To find parts that are locked by a snapshot but have already been dropped or are no longer active on the server — that is, data being retained in object storage solely because of snapshot locks:
```sql theme={null}
SELECT
count(*),
sum(data_uncompressed_bytes)
FROM system.snapshot_parts
WHERE (name, table_id) NOT IN (
SELECT
name,
toString(tables.uuid)
FROM system.parts
INNER JOIN system.tables ON (parts.`table` = tables.name) AND parts.active
)
```
```response theme={null}
┌─count()─┬─sum(data_uncompressed_bytes)─┐
│ 1000 │ 96037 │
└─────────┴──────────────────────────────┘
```
This is useful for understanding the storage overhead of holding snapshots after the original data has changed or been removed.
## Server settings
The following server configuration parameters control snapshot behavior. They are set in the server configuration file, not in SQL.
| Setting | Type | Default | Changeable without restart | Description |
| - | - | - | - | - |
| [`max_held_snapshots`](/docs/reference/settings/server-settings/settings/max#max_held_snapshots) | UInt64 | `0` | No | Maximum number of lightweight snapshots that can be held at the same time. `0` means unlimited. If the limit is reached, creating a new snapshot throws an exception. |
| [`max_snapshot_commit_thread_pool_size`](/docs/reference/settings/server-settings/settings/max-snapshot#max_snapshot_commit_thread_pool_size) | UInt64 | `64` | Yes | Number of threads used to commit snapshot lock nodes into Keeper. Increase this if snapshot creation is slow on large tables with many parts. |
| [`max_snapshot_commit_thread_pool_free_size`](/docs/reference/settings/server-settings/settings/max-snapshot#max_snapshot_commit_thread_pool_free_size) | UInt64 | `0` | Yes | If the number of idle threads in the snapshot commit pool exceeds this value, ClickHouse releases those threads and shrinks the pool. Threads are created again on demand. `0` means idle threads are never released. |
| [`snapshot_cleaner_period`](/docs/reference/settings/server-settings/settings/snapshot-cleaner#snapshot_cleaner_period) | UInt64 | `120` | No | How often (in seconds) the snapshot cleaner runs to remove parts that are no longer referenced by any snapshot lock. ClickHouse Cloud only. |
| [`snapshot_cleaner_pool_size`](/docs/reference/settings/server-settings/settings/snapshot-cleaner#snapshot_cleaner_pool_size) | UInt64 | `128` | No | Number of threads in the snapshot cleaner thread pool. ClickHouse Cloud only. |
# Composable protocols
Source: https://clickhouse.com/docs/concepts/features/configuration/server-config/composable-protocols
Composable protocols allows more flexible configuration of TCP access to the ClickHouse server.
## Overview
Composable protocols allow more flexible configuration of TCP access to the
ClickHouse server. This configuration can co-exist alongside, or replace,
conventional configuration.
## Configuring composable protocols
Composable protocols can be configured in an XML configuration file. The protocols
section is denoted with `protocols` tags in the XML config file:
```xml theme={null}
```
### Configuring protocol layers
You can define protocol layers using basic modules. For example, to define an
HTTP layer, you can add a new basic module to the `protocols` section:
```xml theme={null}
http
```
Modules can be configured according to:
* `plain_http` - name which can be referred to by another layer
* `type` - denotes the protocol handler which will be instantiated to process data.
It has the following set of predefined protocol handlers:
* `tcp` - native clickhouse protocol handler
* `http` - HTTP clickhouse protocol handler
* `tls` - TLS encryption layer
* `proxy1` - PROXYv1 layer
* `mysql` - MySQL compatibility protocol handler
* `postgres` - PostgreSQL compatibility protocol handler
* `prometheus` - Prometheus protocol handler
* `interserver` - clickhouse interserver handler
> [!NOTE]
>
> `gRPC` protocol handler is not implemented for `Composable protocols`
### Configuring endpoints
Endpoints (listening ports) are denoted by `` and optional `` tags.
For example, to configure an endpoint on the previously added HTTP layer we
could modify our configuration as follows:
```xml theme={null}
http
127.0.0.1
8123
```
If the `` tag is omitted, then the `` from the root config is
used.
### Configuring layer sequences
Layers sequences are defined using the `` tag, and referencing another
module. For example, to configure a TLS layer on top of our plain\_http module
we could further modify our configuration as follows:
```xml theme={null}
http
tls
plain_http
127.0.0.1
8443
```
### Attaching endpoints to layers
Endpoints can be attached to any layer. For example, we can define endpoints for
HTTP (port 8123) and HTTPS (port 8443):
```xml theme={null}
http
127.0.0.1
8123
tls
plain_http
127.0.0.1
8443
```
### Defining additional endpoints
Additional endpoints can be defined by referencing any module and omitting the
`` tag. For example, we can define `another_http` endpoint for the
`plain_http` module as follows:
```xml theme={null}
http
127.0.0.1
8123
tls
plain_http
127.0.0.1
8443
plain_http
127.0.0.1
8223
```
### Custom HTTP handlers per endpoint
By default, all `type=http` protocol entries share the same ``
configuration. You can override this by adding a `` tag that points
to a different configuration section. This allows each HTTP port to serve a
different set of HTTP routing rules.
For example, to run an alternative HTTP API on port 8124 with its own handlers:
```xml theme={null}
http
127.0.0.1
8123
http
127.0.0.1
8124
http_handlers_alt
/custom
predefined_query_handler
SELECT 'custom_endpoint'
```
In this example, requests to port 8123 use the standard `` rules,
while requests to port 8124 use the `` rules. If ``
is omitted, the endpoint falls back to the default ``.
The custom handlers section follows the same format as
[``](/docs/reference/settings/server-settings/settings/http#http_handlers).
Changes to the custom handlers section are detected during config reload, and the
corresponding endpoint is automatically restarted.
### Default session user per endpoint
When a client connects without specifying a user name (for example, an HTTP request
without the `user` parameter, or a native protocol `Hello` packet with an empty user
name), the server authenticates it as the default session user - the
`default_session_user` [server setting](/docs/reference/settings/server-settings/settings),
whose default value is `default`.
The `` tag overrides this setting for a single endpoint. This
allows different listening ports to serve different anonymous users:
```xml theme={null}
http
127.0.0.1
8123
plain_http
127.0.0.1
8124
readonly_user
```
In this example, requests without credentials on port 8123 are authenticated as the
globally configured default session user, while those on port 8124 are authenticated
as `readonly_user`. A client that passes a user name explicitly is not affected.
The tag is looked up from the endpoint's module towards the referenced (`impl`)
modules, and the value closest to the endpoint wins. It applies to the `tcp`, `http`,
`mysql` and `postgres` protocol handlers, and to the `prometheus` handlers that
authenticate requests (`remote_write`, `remote_read`, `query` and `api_v1`); the
metrics exposition endpoints (including Keeper metrics-only endpoints) are served
without authentication and ignore the setting. Handlers with a fixed user (the `user`
key inside `handler` of an `http_handlers` rule, or the `user` key inside `handler`
of a `prometheus.handlers` rule) authenticate as their configured user and also ignore the setting - in
particular, an empty `default_session_user` does not reject them. It cannot be used with the
`interserver` protocol: interserver connections are authenticated by the cluster
secret and the initial user and never use the default session user.
### Specifying additional layer parameters
Some modules can contain additional layer parameters. For example, the TLS layer
allows a private key (`privateKeyFile`) and certificate files (`certificateFile`)
to be specified as follows:
```xml theme={null}
http
127.0.0.1
8123
tls
plain_http
127.0.0.1
8443
another_server.key
another_server.crt
```
A TLS layer can also set `cipherList` and `cipherSuites`, but they only take effect when the layer
has its own `privateKeyFile` (and `certificateFile`, if the certificate is in a separate file). A
non-empty `cipherSuites` in a layer without them is a startup error.
# Configuration Files
Source: https://clickhouse.com/docs/concepts/features/configuration/server-config/configuration-files
This page explains how ClickHouse server can be configured with configuration files in XML or YAML syntax.
> [!NOTE]
>
> XML-based settings profiles and configuration files are not supported for ClickHouse Cloud. Therefore, in ClickHouse Cloud, you won't find a config.xml file. Instead, you should use SQL commands to manage settings through settings profiles.
>
> For further details, see ["Configuring Settings"](/docs/products/cloud/reference/settings)
The ClickHouse server can be configured with configuration files in XML or YAML syntax.
In most installation types, the ClickHouse server runs with `/etc/clickhouse-server/config.xml` as the default configuration file, but it is also possible to specify the location of the configuration file manually at server startup using command line option `--config-file` or `-C`.
When neither of these options is given, the server looks for `config.xml`, `config.yaml` and `config.yml`, in this order, in the current working directory, and uses the first one that exists.
If none of them exists, the server starts with the configuration embedded into the binary.
A file requested with `--config-file` is used as is: if it does not exist, the server reports an error instead of looking for a file with a different extension.
`clickhouse-keeper` follows the same rules with `keeper_config.xml`, `keeper_config.yaml` and `keeper_config.yml`.
Additional configuration files may be placed into directory `config.d/` relative to the main configuration file, for example into directory `/etc/clickhouse-server/config.d/`.
Files in this directory and the main configuration are merged in a preprocessing step before the configuration is applied in ClickHouse server.
Configuration fragments are merged in lexicographical order by their full paths. For files in the standard `config.d/` directory, this is equivalent to ordering by file name. The legacy `conf.d/` directory is also merged; because `conf.d` sorts before `config.d`, all its fragments are processed first when both directories exist.
To simplify updates and improve modularization, it is a best practice to keep the default `config.xml` file unmodified and place additional customization into `config.d/`.
The ClickHouse keeper configuration lives in `/etc/clickhouse-keeper/keeper_config.xml`.
Similarly, additional configuration files for Keeper need to be placed in `/etc/clickhouse-keeper/keeper_config.d/`.
It is possible to mix XML and YAML configuration files, for example you could have a main configuration file `config.xml` and additional configuration files `config.d/network.xml`, `config.d/timezone.yaml` and `config.d/keeper.yaml`.
Mixing XML and YAML within a single configuration file is not supported.
XML configuration files should use `...` as the top-level tag.
In YAML configuration files, `clickhouse:` is optional, if absent the parser inserts it automatically.
## Merging configuration
Two configuration files (usually the main configuration file and another configuration file from `config.d/`) are merged as follows:
* If a node (i.e. a path leading to an element) appears in both files and does not have attributes `replace` or `remove`, it is included in the merged configuration file and children from both nodes are included and merged recursively.
* If one of the two nodes contains the `replace` attribute, it is included in the merged configuration file but only children from the node with attribute `replace` are included.
* If one of the two nodes contains the `remove` attribute, the node is not included in the merged configuration file (if it exists already, it is deleted).
For example, given two configuration files:
```xml title="config.xml" theme={null}
1
2
3
```
and
```xml title="config.d/other_config.xml" theme={null}
4
5
6
```
The resulting merged configuration file will be:
```xml theme={null}
1
4
5
```
### Substitution by environment variables and ZooKeeper nodes
To specify that a value of an element should be replaced by the value of an environment variable, you can use the attribute `from_env`.
For example, with environment variable `$MAX_QUERY_SIZE = 150000`:
```xml theme={null}
```
The resulting configuration will be:
```xml theme={null}
150000
```
The same is possible using `from_zk` (ZooKeeper node):
```xml theme={null}
```
```shell theme={null}
# clickhouse-keeper-client
/ :) touch /zk_configs
/ :) create /zk_configs/postgresql_port "9005"
/ :) get /zk_configs/postgresql_port
9005
```
Resulting in the following configuration:
```xml theme={null}
9005
```
#### How values are interpreted
`from_env` and `from_zk` differ in how the substituted value is interpreted:
* A direct value substituted via `from_env` is taken as literal text. XML special characters such as `&`, `<` and `>` are escaped automatically, so an environment variable can contain any XML-representable value, for example `a&bd`, without breaking the configuration (see the note on XML-illegal control characters below). A direct value is never interpreted as XML, even if it happens to look like an XML fragment: `1` substitutes the literal text `1`. This also applies when the direct attribute is placed on an ordinary container such as ``: an existing XML-subtree substitution must be migrated to ``. For backward compatibility, `` is different: it parses the environment variable as an XML fragment and splices its children into the parent element.
* A value substituted via `from_zk` depends on whether it begins with `<`. A value that begins with `<` is always parsed as an XML fragment: this is how a subtree is spliced into any element, both a structural `` and an ordinary container such as ``; it is kept for backward compatibility, because earlier versions always interpreted a `from_zk` value as XML. A non-`<` value is parsed as a YAML subtree only with the explicit form ``; malformed YAML then rejects the configuration. The `yaml="true"` attribute enables YAML parsing wherever the generic `` form is used, including inside a leaf element, so do not use it for a literal setting or secret. Without `yaml="true"`, a non-`<` value is literal text, including when it is used through the generic leaf-replacement form ``. This preserves values that happen to contain YAML syntax or are malformed YAML, such as `abc # rotated` or `[1`. For that form, exact bytes are preserved only when the `` is inline with no surrounding whitespace text nodes; for secrets and leaf settings where exact bytes matter, prefer the direct `` form. On any other element — whether a leaf substitution such as `` or ``, or an ordinary container such as `` — a non-`<` value is also kept as literal text using its exact original bytes (XML special characters are escaped automatically, the same as for `from_env`), so a value such as `abc: def` stays the literal text `abc: def` instead of becoming an `def` sub-element. To splice a subtree into an ordinary element (rather than a structural ``), provide it as an XML fragment (a value beginning with `<`).
> [!NOTE]
>
> **Literal values must be valid XML 1.0 text**
>
> A literal substitution (`from_env`, or a non-`<` leaf `from_zk` value) preserves the exact bytes of the value for every character that XML 1.0 can represent as text. Control characters that are illegal in XML 1.0 — `0x00`–`0x08`, `0x0B`, `0x0C` and `0x0E`–`0x1F` — cannot be carried through an XML configuration at all (the XML specification forbids them even as numeric character references such as ``), and a value containing one is rejected when the substitution is processed. Tab (`\t`), line feed (`\n`) and carriage return (`\r`) are legal and round-trip byte-for-byte.
> [!NOTE]
>
> **Upgrade behaviour for entity-encoded leaf values**
>
> Earlier versions reparsed every direct `from_env` value as XML, so a raw `&`, `<` or `>` was rejected as not well-formed and the only way to embed one was to XML-entity-encode it: an environment variable set to `a&b` decoded to `a&b`, `<a>1</a>` decoded to `1`, and `
` decoded to a carriage return. A direct `from_env` value is now literal text, so those same environment values resolve to the literal `a&b`, `<a>1</a>` and `
`. This means only *plain-text* values keep their previous meaning unchanged: an environment value that was entity-encoded for the old XML path must be set raw instead (for example set `a&b`, which the old path rejected but the new one accepts as the literal text `a&b`).
>
> Earlier versions reparsed every leaf `from_zk` value as XML, so a raw `&`, `<` or `>` was rejected as not well-formed and the only way to embed one was to XML-entity-encode it: a value stored as `a&b` decoded to `a&b`, `<a>1</a>` decoded to `1`, and `
` decoded to a carriage return. A non-`<` leaf value is now kept as its exact original bytes, so those same nodes now resolve to the literal `a&b`, `<a>1</a>` and `
`. This means only *plain-text* scalars keep their previous meaning unchanged: a leaf value that was entity-encoded for the old XML path must be stored raw instead (for example store `a&b`, which the old path rejected but the new one accepts as the literal text `a&b`).
>
> The same change applies to a plain scalar substituted through an inline `` under a leaf setting or a secret, for example ``: a scalar is now kept as its exact original bytes there too, so `/secret = a&b` (which used to decode to `a&b`) now resolves to the literal `a&b` and must be stored raw as `a&b` instead. Do not use surrounding whitespace for this form, because that whitespace is part of the resulting setting value; prefer the direct `` form when exact bytes matter. (A `from_zk` value that begins with `<`, or a YAML subtree referenced with ``, is unaffected — only unmarked non-`<` values are kept literal.)
#### Default values
An element with the `from_env` or `from_zk` attributes may additionally have the attribute `replace="1"` (the latter must appear before `from_env`/`from_zk`).
In this case, the element may define a default value.
The element takes on the value of the environment variable or ZooKeeper node if set, otherwise it takes on the default value.
The previous example is repeated, but assuming `MAX_QUERY_SIZE` is not set:
```xml theme={null}
150000
```
Resulting in configuration:
```xml theme={null}
150000
```
## Substitution with file content
It is also possible to replace parts of the configuration by file contents. This can be done in two ways:
* *Substituting Values*: If an element has the attribute `incl`, its value will be replaced by the content of the referenced file. The path to the file with substitutions is set by the [`include_from`](/docs/reference/settings/server-settings/settings/other#include_from) element in the server config; there is no default path, so substitutions are only read when it is specified. The substitution values are specified in `/clickhouse/substitution_name` elements in this file. If a substitution specified in `incl` does not exist, it is recorded in the log. To prevent ClickHouse from logging missing substitutions, specify attribute `optional="true"` (for example, settings for [macros](/docs/reference/settings/server-settings/settings/other#macros)).
* *Substituting elements*: If you want to replace the entire element with a substitution, use `include` as the element name. The element name `include` can be combined with the attribute `from_zk = "/path/to/node"`. In this case, the element value is replaced by the contents of the ZooKeeper node at `/path/to/node`. This also works with you store an entire XML subtree as a Zookeeper node, it will be fully inserted into the source element.
> [!NOTE]
>
> Changed in version 26.8: before that version, the file `/etc/metrika.xml` was used implicitly whenever it existed, even when `include_from` was not specified. If you rely on that file, specify its path in `include_from` explicitly. Note that `include_from` is read from each configuration file individually: files that are loaded separately from the main server configuration — the users' configuration (e.g. `users.xml` when it is not included in the main file) and XML dictionary configurations — each need their own `include_from` element; specifying it only in the main server configuration does not apply to them.
An example of this is shown below:
```xml theme={null}
```
If you want to merge the substituting content with the existing configuration instead of appending, you can use the attribute `merge="true"`. For example: ``. In this case, the existing configuration will be merged with the content from the substitution and the existing configuration settings will be replaced with values from the substitution.
## Encrypting and hiding configuration
You can use symmetric encryption to encrypt a configuration element, for example, a plaintext password or private key.
To do so, first configure the [encryption codec](/docs/reference/statements/create/table/codec#encryption-codecs), then add the attribute `encrypted_by` with the name of the encryption codec as the value to the element to encrypt.
Unlike attributes `from_zk`, `from_env` and `incl`, or element `include`, no substitution (i.e. decryption of the encrypted value) is performed in the preprocessed file.
Decryption happens only at runtime in the server process.
For example:
```xml theme={null}
00112233445566778899aabbccddeeff
admin
961F000000040000000000EEDDEF4F453CFE6457C4234BD7C09258BD651D85
```
The attributes [`from_env`](#from_env_zk) and [`from_zk`](#from_env_zk) can also be applied to `encryption_codecs`:
```xml theme={null}
admin
961F000000040000000000EEDDEF4F453CFE6457C4234BD7C09258BD651D85
```
```xml theme={null}
admin
961F000000040000000000EEDDEF4F453CFE6457C4234BD7C09258BD651D85
```
Encryption keys and encrypted values can be defined in either config file.
An example `config.xml` is given as:
```xml theme={null}
```
An example `users.xml` is given as:
```xml theme={null}
96280000000D000000000030D4632962295D46C6FA4ABF007CCEC9C1D0E19DA5AF719C1D9A46C446
default
```
To encrypt a value, you can use the (example) program `encrypt_decrypt`:
```bash theme={null}
./encrypt_decrypt /etc/clickhouse-server/config.xml -e AES_128_GCM_SIV abcd
```
```text theme={null}
961F000000040000000000EEDDEF4F453CFE6457C4234BD7C09258BD651D85
```
Even with encrypted configuration elements, encrypted elements still appear in the preprocessed configuration file.
If this is a problem for your ClickHouse deployment, there are two alternatives: Either set file permissions of the preprocessed file to 600 or use the attribute `hide_in_preprocessed`.
For example:
```xml theme={null}
admin
secret
```
## User settings
The `config.xml` file can specify a separate config with user settings, profiles, and quotas. The relative path to this config is set in the `users_config` element. By default, it is `users.xml`. If `users_config` is omitted, the user settings, profiles, and quotas are specified directly in `config.xml`.
User configuration can be split into separate files similar to `config.xml` and `config.d/`.
The directory name is defined as `users_config` setting without `.xml` postfix concatenated with `.d`.
The directory `users.d` is used by default, as `users_config` defaults to `users.xml`.
User configuration fragments are merged in lexicographical order by their full paths. For files in the standard `users.d/` directory, this is equivalent to ordering by file name. The legacy `conf.d/` directory is also merged; because `conf.d` sorts before `users.d`, all its fragments are processed first when both directories exist.
Note that configuration files are first [merged](#merging) taking into account settings, and includes are processed after that.
## XML example
For example, you can have a separate config file for each user like this:
```bash theme={null}
$ cat /etc/clickhouse-server/users.d/alice.xml
```
```xml theme={null}
analytics
::/0
...
analytics
```
## YAML examples
Here you can see the default config written in YAML: [`config.yaml.example`](https://github.com/ClickHouse/ClickHouse/blob/master/programs/server/config.yaml.example).
There are some differences between YAML and XML formats in terms of ClickHouse configurations.
Tips for writing configuration in YAML format are presented below.
An XML tag with a text value is represented by a YAML key-value pair
```yaml theme={null}
key: value
```
Corresponding XML:
```xml theme={null}
value
```
A nested XML node is represented by a YAML map:
```yaml theme={null}
map_key:
key1: val1
key2: val2
key3: val3
```
Corresponding XML:
```xml theme={null}
val1
val2
val3
```
To create the same XML tag multiple times, use a YAML sequence:
```yaml theme={null}
seq_key:
- val1
- val2
- key1: val3
- map:
key2: val4
key3: val5
```
Corresponding XML:
```xml theme={null}
val1
val2
val3
```
To provide an XML attribute, you can use an attribute key with a `@` prefix. Note that `@` is reserved by YAML standard, so must be wrapped in double quotes:
```yaml theme={null}
map:
"@attr1": value1
"@attr2": value2
key: 123
```
Corresponding XML:
```xml theme={null}
```
It is also possible to use attributes in YAML sequence:
```yaml theme={null}
seq:
- "@attr1": value1
- "@attr2": value2
- 123
- abc
```
Corresponding XML:
```xml theme={null}
123
abc
```
The aforementioned syntax does not allow to express XML text nodes with XML attributes as YAML. This special case can be achieved using an
`#text` attribute key:
```yaml theme={null}
map_key:
"@attr1": value1
"#text": value2
```
Corresponding XML:
```xml theme={null}
value2
```
## Implementation details
For each config file, the server also generates `file-preprocessed.xml` files when starting. These files contain all the completed substitutions and overrides, and they are intended for informational use. If ZooKeeper substitutions were used in the config files but ZooKeeper is not available on the server start, the server loads the configuration from the preprocessed file.
The server tracks changes in config files, as well as files and ZooKeeper nodes that were used when performing substitutions and overrides, and reloads the settings for users and clusters on the fly. This means that you can modify the cluster, users, and their settings without restarting the server.
# Introspection port
Source: https://clickhouse.com/docs/concepts/features/configuration/server-config/introspection-port
A native protocol port that serves queries while the server is starting up or shutting down, when the regular ports do not accept connections.
When `clickhouse-server` is stuck in startup or in shutdown, the regular ports do not accept connections and the server cannot be asked what it is doing.
An introspection port is a native protocol TCP listener that starts before the server begins attaching tables and stops only after the tables' detach completes. During these windows an operator can connect to it with a stock `clickhouse-client` and run queries such as `SHOW PROCESSLIST`, `SELECT * FROM system.stack_trace`, or `SYSTEM INSTRUMENT ADD 'QueryMetricLog::startQuery' SLEEP ENTRY 0.5`.
## Configuration
An introspection port is a [composable protocol](/docs/concepts/features/configuration/server-config/composable-protocols) endpoint marked with `true`. No introspection port is configured by default:
```xml theme={null}
tcp
true
introspection native protocol (tcp)
127.0.0.1
9010
tls
introspection_native
true
secure introspection native protocol (tcp_secure)
127.0.0.1
9011
```
Because it is a regular composable protocol endpoint, it supports everything an endpoint supports: `host`, `port`, `description`, a `networks` allow list, `default_database`, and wrapping into `tls` and `proxy1` layers.
The `introspection` flag is a property of the endpoint - the entry that carries the `port` - and is not inherited through `impl` references, so a layer can be shared between an introspection endpoint and a regular one.
The stack must end in a `tcp` layer, because the semantics below are implemented by the native protocol handler. Marking an endpoint whose stack contains an `http`, `mysql`, `postgres`, `prometheus` or `interserver` layer is rejected with `INVALID_CONFIG_PARAMETER` at startup, rather than producing a port that stops answering exactly when it is needed.
| Key | Description |
| - | - |
| `introspection` | Marks a composable protocol endpoint as an introspection port. Default: `false`. |
## Behavior
* Accepted queries are `SELECT`, `SHOW`, `DESCRIBE`, `EXPLAIN`, `EXISTS`, `KILL QUERY`, `SYSTEM`, `SET` and `USE`. Everything else, including DDL and data-modifying queries, is rejected with `QUERY_IS_PROHIBITED`, because the port is open while the server state is either not fully constructed or being torn down.
* `SYSTEM RELOAD CONFIG` and `SYSTEM RELOAD USERS` are rejected with `QUERY_IS_PROHIBITED` until the server is completely started, because reloading the configuration during startup may break the initialization order.
* Queries on these ports bypass `max_concurrent_queries` and the workload scheduler, like `SHOW PROCESSLIST` does.
* Connections are served by a dedicated thread pool, so exhaustion of the regular connection pool does not affect these ports. The pool is sized by `max_connections`.
* Connections are accepted even when the server refuses regular connections because of CPU overload.
* Full normal authentication and authorization apply: connecting to this port grants no extra access rights. It does lift the concurrency limits above, so bind it to an address that only operators can reach.
* These ports are not subject to `SYSTEM START LISTEN` / `SYSTEM STOP LISTEN` (including `ALL` and `CUSTOM`) and are not reconfigured on config reload.
# Named collections
Source: https://clickhouse.com/docs/concepts/features/configuration/server-config/named-collections
Documentation for Named collections
Named collections provide a way to store collections of key-value pairs to be
used to configure integrations with external sources. You can use named collections with
dictionaries, tables, table functions, and object storage.
> [!NOTE]
>
> DDL-created named collections can be enabled on select ClickHouse Cloud services. Contact Support to confirm availability. Named collections defined in configuration files aren't available because users can't modify server configuration files in ClickHouse Cloud.
Named collections can be configured with DDL or in configuration files and are applied
when ClickHouse starts. They simplify the creation of objects and the hiding of credentials
from users without administrative access.
The keys in a named collection must match the parameter names of the corresponding
function, table engine, database, etc. In the examples below the parameter list is
linked to for each type.
Overriding a parameter stored in a named collection requires the `SHOW NAMED COLLECTIONS SECRETS` privilege on that collection, including when the parameter is marked `OVERRIDABLE`. Keys marked `NOT OVERRIDABLE` in SQL or `overridable="false"` in XML cannot be overridden. Adding a parameter that the collection does not store does not require this additional privilege. Aliases, such as `addresses_expr` for `host`, count as overrides of the stored parameter.
```sql theme={null}
GRANT SHOW NAMED COLLECTIONS SECRETS ON my_collection TO my_user;
```
This check requires the privilege alone; it does not depend on the settings that control displaying secrets. The `allow_named_collection_override_by_default` setting is obsolete and has no effect.
Dictionary sources follow the same rule. The privilege is checked when the dictionary is created, attached, or restored. When the dictionary is loaded, only keys marked `NOT OVERRIDABLE` are enforced.
## Storing named collections in the system database
### DDL example
```sql theme={null}
CREATE NAMED COLLECTION name AS
key_1 = 'value' OVERRIDABLE,
key_2 = 'value2' NOT OVERRIDABLE,
url = 'https://connection.url/'
```
In the above example:
* `key_1` can be overridden by users with `SHOW NAMED COLLECTIONS SECRETS` on the collection.
* `key_2` can never be overridden.
* `url` can be overridden by users with `SHOW NAMED COLLECTIONS SECRETS` on the collection.
### Permissions to create named collections with DDL
To manage named collections with DDL a user must have the `named_collection_control` privilege. This can be assigned by adding a file to `/etc/clickhouse-server/users.d/`. The example gives the user `default` both the `access_management` and `named_collection_control` privileges:
```xml title='/etc/clickhouse-server/users.d/user_default.xml' highlight={6} theme={null}
65e84be33532fb784c48129675f9eff3a682b27168c0ea744b2cf58ee02337c5
1
1
```
> [!TIP]
>
> In the above example the `password_sha256_hex` value is the hexadecimal representation of the SHA256 hash of the password. This configuration for the user `default` has the attribute `replace=true` as in the default configuration has a plain text `password` set, and it is not possible to have both plain text and sha256 hex passwords set for a user.
`named_collection_control` allows the user to create, alter and drop named collections, but not to read back the values that are already stored in them. That is a separate privilege, `SHOW NAMED COLLECTIONS SECRETS`, enabled by the [`show_named_collections_secrets`](/docs/concepts/features/configuration/settings/settings-users#show_named_collections_secrets-user-setting) setting. A user can only grant the privileges that they have themselves, so while `show_named_collections_secrets` is disabled, the user does not have the complete set of privileges and `GRANT ALL ON *.* TO another_user WITH GRANT OPTION` is rejected with `(Missing permissions: SHOW NAMED COLLECTIONS SECRETS ON *)`.
### Storage for named collections
Named collections can either be stored on local disk or in ZooKeeper/Keeper. By default local storage is used.
They can also be stored using encryption with the same algorithms used for [disk encryption](/docs/concepts/features/configuration/server-config/storing-data#encrypted-virtual-file-system),
where `aes_128_ctr` is used by default.
To configure named collections storage you need to specify a `type`. This can be either `local` or `keeper`/`zookeeper`. For encrypted storage,
you can use `local_encrypted` or `keeper_encrypted`/`zookeeper_encrypted`.
To use ZooKeeper/Keeper we also need to set up a `path` (path in ZooKeeper/Keeper, where named collections will be stored) to
`named_collections_storage` section in configuration file. The following example uses encryption and ZooKeeper/Keeper:
```xml theme={null}
zookeeper_encrypted
bebec0cabebec0cabebec0cabebec0ca
aes_128_ctr
/named_collections_path/
1000
```
An optional configuration parameter `update_timeout_ms` by default is equal to `5000`.
You can inspect the active storage type through `system.server_settings` and `getServerSetting`:
```sql theme={null}
SELECT value, getServerSetting('named_collections_storage_type')
FROM system.server_settings
WHERE name = 'named_collections_storage.type';
```
Changing the storage type requires a server restart; `SYSTEM RELOAD CONFIG` does not change the active backend.
## Storing named collections in configuration files
### XML example
```xml title='/etc/clickhouse-server/config.d/named_collections.xml' theme={null}
value
value_2
https://connection.url/
```
In the above example:
* `key_1` can be overridden by users with `SHOW NAMED COLLECTIONS SECRETS` on the collection.
* `key_2` can never be overridden.
* `url` can be overridden by users with `SHOW NAMED COLLECTIONS SECRETS` on the collection.
## Modifying named collections
Named collections that are created with DDL queries can be altered or dropped with DDL. Named collections created with XML files can be managed by editing or deleting the corresponding XML.
### Alter a DDL named collection
Change or add the keys `key1` and `key3` of the collection `collection2`
(this will not change the value of the `overridable` flag for those keys):
```sql theme={null}
ALTER NAMED COLLECTION collection2 SET key1=4, key3='value3'
```
Change or add the key `key1` and allow it to be always overridden:
```sql theme={null}
ALTER NAMED COLLECTION collection2 SET key1=4 OVERRIDABLE
```
Remove the key `key2` from `collection2`:
```sql theme={null}
ALTER NAMED COLLECTION collection2 DELETE key2
```
Change or add the key `key1` and delete the key `key3` of the collection `collection2`:
```sql theme={null}
ALTER NAMED COLLECTION collection2 SET key1=4, DELETE key3
```
To force a key to use the default settings for the `overridable` flag, you have to
remove and re-add the key.
```sql theme={null}
ALTER NAMED COLLECTION collection2 DELETE key1;
ALTER NAMED COLLECTION collection2 SET key1=4;
```
### Drop the DDL named collection `collection2`:
```sql theme={null}
DROP NAMED COLLECTION collection2
```
## Named collections for accessing S3
The description of parameters see [s3 Table Function](/docs/reference/functions/table-functions/s3).
### DDL example
```sql theme={null}
CREATE NAMED COLLECTION s3_mydata AS
access_key_id = 'AKIAIOSFODNN7EXAMPLE',
secret_access_key = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
format = 'CSV',
url = 'https://s3.us-east-1.amazonaws.com/yourbucket/mydata/'
```
### XML example
```xml theme={null}
AKIAIOSFODNN7EXAMPLE
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
CSV
https://s3.us-east-1.amazonaws.com/yourbucket/mydata/
```
### s3() function and S3 Table named collection examples
Both of the following examples use the same named collection `s3_mydata`:
#### s3() function
```sql theme={null}
INSERT INTO FUNCTION s3(s3_mydata, filename = 'test_file.tsv.gz',
format = 'TSV', structure = 'number UInt64', compression_method = 'gzip')
SELECT * FROM numbers(10000);
```
> [!TIP]
>
> The first argument to the `s3()` function above is the name of the collection, `s3_mydata`. Without named collections, the access key ID, secret, format, and URL would all be passed in every call to the `s3()` function.
#### S3 table
```sql theme={null}
CREATE TABLE s3_engine_table (number Int64)
ENGINE=S3(s3_mydata, url='https://s3.us-east-1.amazonaws.com/yourbucket/mydata/test_file.tsv.gz', format = 'TSV')
SETTINGS input_format_with_names_use_header = 0;
SELECT * FROM s3_engine_table LIMIT 3;
┌─number─┐
│ 0 │
│ 1 │
│ 2 │
└────────┘
```
## Named collections for accessing MySQL database
The description of parameters see [mysql](/docs/reference/functions/table-functions/mysql).
### DDL example
```sql theme={null}
CREATE NAMED COLLECTION mymysql AS
user = 'myuser',
password = 'mypass',
host = '127.0.0.1',
port = 3306,
database = 'test',
connection_pool_size = 8,
replace_query = 1
```
### XML example
```xml theme={null}
myuser
mypass
127.0.0.1
3306
test
8
1
```
### mysql() function, MySQL table, MySQL database, and Dictionary named collection examples
The four following examples use the same named collection `mymysql`:
#### mysql() function
```sql theme={null}
SELECT count() FROM mysql(mymysql, table = 'test');
┌─count()─┐
│ 3 │
└─────────┘
```
> [!NOTE]
>
> The named collection does not specify the `table` parameter, so it is specified in the function call as `table = 'test'`.
#### MySQL table
```sql theme={null}
CREATE TABLE mytable(A Int64) ENGINE = MySQL(mymysql, table = 'test', connection_pool_size=3, replace_query=0);
SELECT count() FROM mytable;
┌─count()─┐
│ 3 │
└─────────┘
```
> [!NOTE]
>
> The DDL overrides the named collection setting for connection\_pool\_size.
#### MySQL database
```sql theme={null}
CREATE DATABASE mydatabase ENGINE = MySQL(mymysql);
SHOW TABLES FROM mydatabase;
┌─name───┐
│ source │
│ test │
└────────┘
```
#### MySQL Dictionary
```sql theme={null}
CREATE DICTIONARY dict (A Int64, B String)
PRIMARY KEY A
SOURCE(MYSQL(NAME mymysql TABLE 'source'))
LIFETIME(MIN 1 MAX 2)
LAYOUT(HASHED());
SELECT dictGet('dict', 'B', 2);
┌─dictGet('dict', 'B', 2)─┐
│ two │
└─────────────────────────┘
```
## Named collections for accessing PostgreSQL database
The description of parameters see [postgresql](/docs/reference/functions/table-functions/postgresql). Additionally, there are aliases:
* `username` for `user`
* `db` for `database`.
The connection-pool settings of the [PostgreSQL table engine](/docs/reference/engines/table-engines/integrations/postgresql) (`postgresql_connection_pool_size` and the other `postgresql_*` settings) can also be stored in the collection or passed as `key = value` overrides. They apply to the `PostgreSQL` table engine, the `postgresql` table function, and the `PostgreSQL` database engine; an explicit `SETTINGS` clause on a table takes precedence over the values from the collection.
Parameter `addresses_expr` is used in a collection instead of `host:port`. The parameter is optional, because there are other optional ones: `host`, `hostname`, `port`. The following pseudo code explains the priority:
```sql theme={null}
CASE
WHEN collection['addresses_expr'] != '' THEN collection['addresses_expr']
WHEN collection['host'] != '' THEN collection['host'] || ':' || if(collection['port'] != '', collection['port'], '5432')
WHEN collection['hostname'] != '' THEN collection['hostname'] || ':' || if(collection['port'] != '', collection['port'], '5432')
END
```
Example of creation:
```sql theme={null}
CREATE NAMED COLLECTION mypg AS
user = 'pguser',
password = 'jw8s0F4',
host = '127.0.0.1',
port = 5432,
database = 'test',
schema = 'test_schema'
```
Example of configuration:
```xml theme={null}
pguser
jw8s0F4
127.0.0.1
5432
test
test_schema
```
### Example of using named collections with the postgresql function
```sql theme={null}
SELECT * FROM postgresql(mypg, table = 'test');
┌─a─┬─b───┐
│ 2 │ two │
│ 1 │ one │
└───┴─────┘
SELECT * FROM postgresql(mypg, table = 'test', schema = 'public');
┌─a─┐
│ 1 │
│ 2 │
│ 3 │
└───┘
```
### Example of using named collections with database with engine PostgreSQL
```sql theme={null}
CREATE TABLE mypgtable (a Int64) ENGINE = PostgreSQL(mypg, table = 'test', schema = 'public');
SELECT * FROM mypgtable;
┌─a─┐
│ 1 │
│ 2 │
│ 3 │
└───┘
```
> [!NOTE]
>
> PostgreSQL copies data from the named collection when the table is being created. A change in the collection does not affect the existing tables.
### Example of using named collections with database with engine PostgreSQL
```sql theme={null}
CREATE DATABASE mydatabase ENGINE = PostgreSQL(mypg);
SHOW TABLES FROM mydatabase
┌─name─┐
│ test │
└──────┘
```
### Example of using named collections with a dictionary with source POSTGRESQL
```sql theme={null}
CREATE DICTIONARY dict (a Int64, b String)
PRIMARY KEY a
SOURCE(POSTGRESQL(NAME mypg TABLE test))
LIFETIME(MIN 1 MAX 2)
LAYOUT(HASHED());
SELECT dictGet('dict', 'b', 2);
┌─dictGet('dict', 'b', 2)─┐
│ two │
└─────────────────────────┘
```
## Named collections for accessing a remote ClickHouse database
The description of parameters see [remote](/docs/reference/functions/table-functions/remote#parameters).
Example of configuration:
```sql theme={null}
CREATE NAMED COLLECTION remote1 AS
host = 'remote_host',
port = 9000,
database = 'system',
user = 'foo',
password = 'secret',
secure = 1
```
```xml theme={null}
remote_host
9000
system
foo
secret
1
```
`secure` is not needed for connection because of `remoteSecure`, but it can be used for dictionaries.
### Example of using named collections with the `remote`/`remoteSecure` functions
```sql theme={null}
SELECT * FROM remote(remote1, table = one);
┌─dummy─┐
│ 0 │
└───────┘
SELECT * FROM remote(remote1, database = merge(system, '^one'));
┌─dummy─┐
│ 0 │
└───────┘
INSERT INTO FUNCTION remote(remote1, database = default, table = test) VALUES (1,'a');
SELECT * FROM remote(remote1, database = default, table = test);
┌─a─┬─b─┐
│ 1 │ a │
└───┴───┘
```
### Example of using named collections with a dictionary with source ClickHouse
> [!NOTE]
>
> When creating a dictionary that reads from a ClickHouse table, include a dedicated user's username and password in the `SOURCE` clause. That user needs `SELECT` permission on the source table. See the [ClickHouse dictionary source example](/docs/reference/statements/create/dictionary/sources/clickhouse).
```sql theme={null}
CREATE NAMED COLLECTION remote_dict AS
host = 'localhost',
port = 9000,
database = 'default',
user = 'dict_reader',
password = 'dict_reader_password';
CREATE DICTIONARY dict(a Int64, b String)
PRIMARY KEY a
SOURCE(CLICKHOUSE(NAME remote_dict TABLE test))
LIFETIME(MIN 1 MAX 2)
LAYOUT(HASHED());
SELECT dictGet('dict', 'b', 1);
┌─dictGet('dict', 'b', 1)─┐
│ a │
└─────────────────────────┘
```
## Named collections for accessing Kafka
The description of parameters see [Kafka](/docs/reference/engines/table-engines/integrations/kafka).
### DDL example
```sql theme={null}
CREATE NAMED COLLECTION my_kafka_cluster AS
kafka_broker_list = 'localhost:9092',
kafka_topic_list = 'kafka_topic',
kafka_group_name = 'consumer_group',
kafka_format = 'JSONEachRow',
kafka_max_block_size = '1048576';
```
### XML example
```xml theme={null}
localhost:9092
kafka_topic
consumer_group
JSONEachRow
1048576
```
### OAUTHBEARER/OIDC authentication
`OAUTHBEARER` options are extended `librdkafka` settings, so they must be nested under `` inside each named collection. Top-level `kafka_*` keys configure the Kafka table engine. ClickHouse converts underscores in the nested XML element names to periods before passing them to `librdkafka`.
The following example configures OAuth client credentials for an Azure Event Hubs namespace:
```xml title='/etc/clickhouse-server/config.d/named_collections.xml' theme={null}
domain.one.servicebus.windows.net:9093
my-topic
my-group
JSONEachRow
sasl_ssl
OAUTHBEARER
oidc
https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token
https://domain.one.servicebus.windows.net/.default
```
When `ENGINE = Kafka(eventhub_one)` uses a named collection, ClickHouse does not merge the global server `` configuration. Create one collection per Event Hubs namespace when the broker address or OAuth scope differs, and include every required OIDC property in that collection.
The `from_env` attributes in this example apply to XML-defined named collections and read their values from the ClickHouse server process environment. See [substitution by environment variables](/docs/concepts/features/configuration/server-config/configuration-files#from_env_zk) for deployment and default-value details, and [encrypting and hiding configuration](/docs/concepts/features/configuration/server-config/configuration-files#encryption) to protect secrets in preprocessed configuration files.
### Example of using named collections with a Kafka table
Both of the following examples use the same named collection `my_kafka_cluster`:
```sql theme={null}
CREATE TABLE queue
(
timestamp UInt64,
level String,
message String
)
ENGINE = Kafka(my_kafka_cluster)
CREATE TABLE queue
(
timestamp UInt64,
level String,
message String
)
ENGINE = Kafka(my_kafka_cluster)
SETTINGS kafka_num_consumers = 4,
kafka_thread_per_consumer = 1;
```
## Named collections for backups
For the description of parameters see [Backup and Restore](/docs/concepts/features/backup-restore/overview).
### DDL example
```sql theme={null}
BACKUP TABLE default.test to S3(named_collection_s3_backups, 'directory')
```
### XML example
```xml theme={null}
https://my-s3-bucket.s3.amazonaws.com/backup-S3/
ABC123
Abc+123
```
## Named collections for accessing MongoDB Table and Dictionary
For the description of parameters see [mongodb](/docs/reference/functions/table-functions/mongodb).
### DDL example
```sql theme={null}
CREATE NAMED COLLECTION mymongo AS
user = '',
password = '',
host = '127.0.0.1',
port = 27017,
database = 'test',
collection = 'my_collection',
options = 'connectTimeoutMS=10000'
```
### XML example
```xml theme={null}
127.0.0.1
27017
test
my_collection
connectTimeoutMS=10000
```
#### MongoDB table
```sql theme={null}
CREATE TABLE mytable(log_type VARCHAR, host VARCHAR, command VARCHAR) ENGINE = MongoDB(mymongo, options='connectTimeoutMS=10000&compressors=zstd')
SELECT count() FROM mytable;
┌─count()─┐
│ 2 │
└─────────┘
```
> [!NOTE]
>
> The DDL overrides the named collection setting for options.
#### MongoDB Dictionary
```sql theme={null}
CREATE DICTIONARY dict
(
`a` Int64,
`b` String
)
PRIMARY KEY a
SOURCE(MONGODB(NAME mymongo))
LIFETIME(MIN 1 MAX 2)
LAYOUT(HASHED())
SELECT dictGet('dict', 'b', 2);
┌─dictGet('dict', 'b', 2)─┐
│ two │
└─────────────────────────┘
```
> [!NOTE]
>
> The dictionary uses `my_collection`, as specified in the named collection. To use another MongoDB collection, define a separate named collection with that value.
# Quotas
Source: https://clickhouse.com/docs/concepts/features/configuration/server-config/quotas
Guide to configuring and managing resource usage quotas in ClickHouse
> [!NOTE]
>
> **Quotas in ClickHouse Cloud**
>
> Quotas are supported in ClickHouse Cloud but must be created using the [DDL syntax](/docs/reference/statements/create/quota). The XML configuration approach documented below is **not supported**.
Quotas allow you to limit resource usage over a period of time or track the use of resources.
Quotas are set up in the user config, which is usually 'users.xml'.
The system also has a feature for limiting the complexity of a single query. See the section [Restrictions on query complexity](/docs/concepts/features/configuration/settings/query-complexity).
In contrast to query complexity restrictions, quotas:
* Place restrictions on a set of queries that can be run over a period of time, instead of limiting a single query.
* Account for resources spent on all remote servers for distributed query processing.
Let's look at the section of the 'users.xml' file that defines quotas.
```xml theme={null}
3600
0
0
0
0
0
0
0
```
By default, the quota tracks resource consumption for each hour, without limiting usage.
The resource consumption calculated for each interval is output to the server log after each request.
```xml theme={null}
3600
1000
100
100
5000000
100000
100
1000000000
100000000000
900
5
86400
10000
10000
10000
1000
5000000000
160000000000
500000000000
16000000000000
7200
```
For the 'statbox' quota, restrictions are set for every hour and for every 24 hours (86,400 seconds). The time interval is counted, starting from an implementation-defined fixed moment in time. In other words, the 24-hour interval does not necessarily begin at midnight.
When the interval ends, all collected values are cleared. For the next hour, the quota calculation starts over.
Here are the amounts that can be restricted:
`queries` – The total number of requests.
`query_selects` – The total number of select requests.
`query_inserts` – The total number of insert requests.
`errors` – The number of queries that threw an exception.
`result_rows` – The total number of rows given as a result.
`result_bytes` - The total size of rows given as a result.
`read_rows` – The total number of source rows read from tables for running the query on all remote servers.
`read_bytes` - The total size read from tables for running the query on all remote servers.
`written_bytes` - The total size of a writing operation.
`written_rows` - The total number of rows written by `INSERT` queries.
`execution_time` – The total query execution time, in seconds (wall time).
`failed_sequential_authentications` - The total number of sequential authentication errors.
`queries_per_normalized_hash` – The maximum number of executions of any single normalized query. Normalized queries are queries with literals replaced by placeholders, so `SELECT 1` and `SELECT 2` are considered the same normalized query. This limit is tracked per distinct normalized query pattern independently.
If the limit is exceeded for at least one time interval, an exception is thrown with a text about which restriction was exceeded, for which interval, and when the new interval begins (when queries can be sent again).
> [!NOTE]
>
> **Queries excluded from quota accounting**
>
> To allow quota configuration and usage to be inspected even when a quota is exhausted, `SELECT` queries that exclusively read from one of the following system tables are not charged to quotas and are not subject to query-complexity limits:
>
> * `system.quotas`
> * `system.quota_limits`
> * `system.quota_usage`
> * `system.quotas_usage`
> * `system.one`
Quotas can use the "quota key" feature to report on resources for multiple keys independently. Here is an example of this:
```xml theme={null}
```
You can also key quotas by normalized query hash, so that each distinct query pattern gets its own independent quota bucket. In the XML configuration this is written as ``:
```xml theme={null}
3600
100
```
The same can be expressed with the DDL syntax:
```sql theme={null}
CREATE QUOTA my_quota KEYED BY normalized_query_hash FOR INTERVAL 1 hour MAX queries = 100 TO my_user;
```
In this example, the user can execute up to 100 instances of each distinct normalized query per hour. `SELECT number FROM numbers(1)` and `SELECT number FROM numbers(2)` share the same bucket (because they have the same normalized form), but `SELECT number, number FROM numbers(1)` uses a separate bucket.
The quota is assigned to users in the 'users' section of the config. See the section "Access rights".
For distributed query processing, the accumulated amounts are stored on the requestor server. So if the user goes to another server, the quota there will "start over".
When the server is restarted, quotas are reset.
## Related Content
* Blog: [Building single page applications with ClickHouse](https://clickhouse.com/blog/building-single-page-applications-with-clickhouse-and-http)
# Startup scripts
Source: https://clickhouse.com/docs/concepts/features/configuration/server-config/startup-scripts
Guide to configuring and using SQL startup scripts in ClickHouse for automatic schema creation and migrations
ClickHouse can run arbitrary SQL queries from the server configuration during startup. This can be useful for migrations or automatic schema creation.
```xml theme={null}
false
CREATE ROLE OR REPLACE test_role
CREATE TABLE TestTable (id UInt64) ENGINE=TinyLog
SELECT 1;
CREATE DICTIONARY test_dict (...) SOURCE(CLICKHOUSE(...))
default
```
ClickHouse executes all queries from the `startup_scripts` sequentially in the specified order. If any of the queries fail, the execution of the following queries won't be interrupted. However, if `throw_on_error` is set to true,
the server will not start if an error occurs during script execution.
You can specify a conditional query in the config. In that case, the corresponding query executes only when the condition query returns the value `1` or `true`.
> [!NOTE]
>
> If the condition query returns any other value than `1` or `true`, the result will be interpreted as `false`, and the corresponding won't be executed.
# External disks for storing data
Source: https://clickhouse.com/docs/concepts/features/configuration/server-config/storing-data
Documentation for highlight-next-line
Data processed in ClickHouse is usually stored in the local file system of the
machine on which ClickHouse server is running. That requires large-capacity disks,
which can be expensive. To avoid storing data locally, various storage options are supported:
1. [Amazon S3](https://aws.amazon.com/s3/) object storage.
2. [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs).
3. Unsupported: The Hadoop Distributed File System ([HDFS](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html))
> [!NOTE]
>
> ClickHouse also has support for external table engines, which are different from
> the external storage option described on this page, as they allow reading data
> stored in some general file format (like Parquet). On this page we are describing
> storage configuration for the ClickHouse `MergeTree` family or `Log` family tables.
>
> 1. to work with data stored on `Amazon S3` disks, use the [S3](/docs/reference/engines/table-engines/integrations/s3) table engine.
> 2. to work with data stored in Azure Blob Storage, use the [AzureBlobStorage](/docs/reference/engines/table-engines/integrations/azureBlobStorage) table engine.
> 3. to work with data in the Hadoop Distributed File System (unsupported), use the [HDFS](/docs/reference/engines/table-engines/integrations/hdfs) table engine.
## Configure external storage
[`MergeTree`](/docs/reference/engines/table-engines/mergetree-family/mergetree) and [`Log`](/docs/reference/engines/table-engines/log-family/log)
family table engines can store data to `S3`, `AzureBlobStorage`, `HDFS` (unsupported) using a disk with types `s3`,
`azure_blob_storage`, `hdfs` (unsupported) respectively.
> [!NOTE]
>
> The `s3` disk type can also be used with S3-compatible object storage providers, such as Alibaba Cloud Object Storage Service. Configure `endpoint` with the provider's S3-compatible endpoint and account for any provider-specific compatibility requirements.
Disk configuration requires:
1. A `type` section, equal to one of `s3`, `azure_blob_storage`, `hdfs` (unsupported), `local_blob_storage`, `web`.
2. Configuration of a specific external storage type.
Starting from 24.1 clickhouse version, it is possible to use a new configuration option.
It requires specifying:
1. A `type` equal to `object_storage`
2. `object_storage_type`, equal to one of `s3`, `azure_blob_storage` (or just `azure` from `24.3`), `hdfs` (unsupported), `local_blob_storage` (or just `local` from `24.3`), `web`.
Optionally, `metadata_type` can be specified (it is equal to `local` by default), but it can also be set to `plain`, `web` and, starting from `24.4`, `plain_rewritable`.
Usage of `plain` metadata type is described in [plain storage section](/docs/concepts/features/configuration/server-config/storing-data#plain-storage), `web` metadata type can be used only with `web` object storage type, `local` metadata type stores metadata files locally (each metadata files contains mapping to files in object storage and some additional meta information about them).
For example:
```xml theme={null}
s3
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
```
is equal to the following configuration (from version `24.1`):
```xml theme={null}
object_storage
s3
local
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
```
The following configuration:
```xml theme={null}
s3_plain
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
```
is equal to:
```xml theme={null}
object_storage
s3
plain
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
```
An example of full storage configuration will look like:
```xml theme={null}
s3
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
s3
```
Starting with version 24.1, it can also look like:
```xml theme={null}
object_storage
s3
local
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
s3
```
To make a specific kind of storage a default option for all `MergeTree` tables,
add the following section to the configuration file:
```xml theme={null}
s3
```
If you want to configure a specific storage policy for a specific table,
you can define it in settings while creating the table:
```sql theme={null}
CREATE TABLE test (a Int32, b String)
ENGINE = MergeTree() ORDER BY a
SETTINGS storage_policy = 's3';
```
You can also use `disk` instead of `storage_policy`. In this case it is not necessary
to have the `storage_policy` section in the configuration file, and a `disk`
section is enough.
```sql theme={null}
CREATE TABLE test (a Int32, b String)
ENGINE = MergeTree() ORDER BY a
SETTINGS disk = 's3';
```
## refresh\_parts\_interval and table\_disk
This setting is intended for non-Replicated MergeTree tables where parts may be written externally and metadata discovery must be refreshed from storage.
The MergeTree setting `refresh_parts_interval` enables periodic refresh of the list of data parts from the underlying storage (e.g. to pick up parts written externally). The important distinction is **shared metadata across replicas** vs **replica-local metadata** (e.g. S3 with local metadata per replica): only when metadata is shared will new parts be visible to all replicas. Using object storage alone does not imply shared metadata.
* **Object storage (e.g. `disk = 's3'`) does not imply shared metadata.** When metadata is stored locally per replica (the default), each replica independently manages its pointers to blobs in object storage. Changes made on one replica are not visible to others. In that case, `refresh_parts_interval` does not make new parts visible across replicas, because the metadata each replica reads is replica-local.
* **Automatic part refreshing requires that the filesystem metadata be shared** (or that the table use table-owned, readonly metadata so that refresh is applicable). Setting `table_disk = true` together with a table-local disk (e.g. `SETTINGS disk = disk(type=object_storage, ...), table_disk = true`) is one way to get the correct semantics: the table owns the metadata life cycle and the storage is treated as readonly, so `refresh_parts_interval` runs and externally added parts can be discovered.
* **With a globally defined disk** (e.g. `disk = 's3'` in `storage_configuration`) and default local metadata, each replica has its own metadata state. Even though blobs may be in S3, the storage is not considered shared for the purpose of `refresh_parts_interval`, and new parts created outside ClickHouse or on another replica will not be detected.
For automatic part refreshing, ensure the metadata is shared or use a table-level disk with `table_disk = true` as above. Relying only on `refresh_parts_interval` with replica-local metadata will not refresh parts as expected.
> [!NOTE]
>
> `refresh_parts_interval` is not used for ReplicatedMergeTree tables.
> Replicated tables already synchronize parts through the replication mechanism.
> This setting is only applicable to non-replicated MergeTree tables where parts are written externally and metadata refresh is required.
## Dynamic Configuration
There is also a possibility to specify storage configuration without a predefined
disk in configuration in a configuration file, but can be configured in the
`CREATE`/`ATTACH` query settings.
The following example query builds on the above dynamic disk configuration and
shows how to use a local disk to cache data from a table stored at a URL.
```sql highlight={20-23} theme={null}
ATTACH TABLE uk_price_paid UUID 'cf712b4f-2ca8-435c-ac23-c4393efe52f7'
(
price UInt32,
date Date,
postcode1 LowCardinality(String),
postcode2 LowCardinality(String),
type Enum8('other' = 0, 'terraced' = 1, 'semi-detached' = 2, 'detached' = 3, 'flat' = 4),
is_new UInt8,
duration Enum8('unknown' = 0, 'freehold' = 1, 'leasehold' = 2),
addr1 String,
addr2 String,
street LowCardinality(String),
locality LowCardinality(String),
town LowCardinality(String),
district LowCardinality(String),
county LowCardinality(String)
)
ENGINE = MergeTree
ORDER BY (postcode1, postcode2, addr1, addr2)
SETTINGS disk = disk(
type=web,
endpoint='https://raw.githubusercontent.com/ClickHouse/web-tables-demo/main/web/'
);
```
The example below adds a cache to external storage.
```sql highlight={20-28} theme={null}
ATTACH TABLE uk_price_paid UUID 'cf712b4f-2ca8-435c-ac23-c4393efe52f7'
(
price UInt32,
date Date,
postcode1 LowCardinality(String),
postcode2 LowCardinality(String),
type Enum8('other' = 0, 'terraced' = 1, 'semi-detached' = 2, 'detached' = 3, 'flat' = 4),
is_new UInt8,
duration Enum8('unknown' = 0, 'freehold' = 1, 'leasehold' = 2),
addr1 String,
addr2 String,
street LowCardinality(String),
locality LowCardinality(String),
town LowCardinality(String),
district LowCardinality(String),
county LowCardinality(String)
)
ENGINE = MergeTree
ORDER BY (postcode1, postcode2, addr1, addr2)
SETTINGS disk = disk(
type=cache,
max_size='1Gi',
path='/var/lib/clickhouse/custom_disk_cache/',
disk=disk(
type=web,
endpoint='https://raw.githubusercontent.com/ClickHouse/web-tables-demo/main/web/'
)
);
```
In the settings highlighted below notice that the disk of `type=web` is nested within
the disk of `type=cache`.
> [!NOTE]
>
> The example uses `type=web`, but any disk type can be configured as dynamic,
> including local disk. Local disks require a path argument to be inside the
> server config parameter `custom_local_disks_base_directory`, which has no
> default, so set that also when using local disk.
A combination of config-based configuration and sql-defined configuration is
also possible:
```sql highlight={20-28} theme={null}
ATTACH TABLE uk_price_paid UUID 'cf712b4f-2ca8-435c-ac23-c4393efe52f7'
(
price UInt32,
date Date,
postcode1 LowCardinality(String),
postcode2 LowCardinality(String),
type Enum8('other' = 0, 'terraced' = 1, 'semi-detached' = 2, 'detached' = 3, 'flat' = 4),
is_new UInt8,
duration Enum8('unknown' = 0, 'freehold' = 1, 'leasehold' = 2),
addr1 String,
addr2 String,
street LowCardinality(String),
locality LowCardinality(String),
town LowCardinality(String),
district LowCardinality(String),
county LowCardinality(String)
)
ENGINE = MergeTree
ORDER BY (postcode1, postcode2, addr1, addr2)
SETTINGS disk = disk(
type=cache,
max_size='1Gi',
path='/var/lib/clickhouse/custom_disk_cache/',
disk=disk(
type=web,
endpoint='https://raw.githubusercontent.com/ClickHouse/web-tables-demo/main/web/'
)
);
```
where `web` is from the server configuration file:
```xml theme={null}
web
'https://raw.githubusercontent.com/ClickHouse/web-tables-demo/main/web/'
```
### Using S3 Storage
#### Required parameters
| Parameter | Description |
| - | - |
| `endpoint` | S3 endpoint URL in `path` or `virtual hosted` [styles](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html). Should include the bucket and root path for data storage. |
| `access_key_id` | S3 access key ID used for authentication. |
| `secret_access_key` | S3 secret access key used for authentication. |
#### Optional parameters
| Parameter | Description | Default Value |
| - | - | - |
| `region` | S3 region name. | - |
| `support_batch_delete` | Controls whether to check for batch delete support. Set to `false` when using Google Cloud Storage (GCS) as GCS doesn't support batch deletes. | `true` |
| `use_environment_credentials` | Reads AWS credentials from environment variables: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` if they exist. Note: environment credentials are shared across all S3 disks. To use different credentials for different disks, specify explicit `access_key_id` and `secret_access_key` per disk instead. | `false` |
| `use_insecure_imds_request` | If `true`, uses insecure IMDS request when obtaining credentials from Amazon EC2 metadata. | `false` |
| `expiration_window_seconds` | Grace period (in seconds) for checking if expiration-based credentials have expired. | `120` |
| `proxy` | Proxy configuration for S3 endpoint. Each `uri` element inside `proxy` block should contain a proxy URL. | - |
| `connect_timeout_ms` | Socket connect timeout in milliseconds. | `10000` (10 seconds) |
| `request_timeout_ms` | Request timeout in milliseconds. | `5000` (5 seconds) |
| `s3_retry_attempts` | Number of retry attempts for failed requests. | `500` |
| `s3_max_single_read_retries` | Number of retry attempts for connection drops during read. | `4` |
| `min_bytes_for_seek` | Minimum number of bytes to use seek operation instead of sequential read. | `1 MB` |
| `metadata_path` | Local filesystem path to store S3 metadata files. | `/var/lib/clickhouse/disks//` |
| `skip_access_check` | If `true`, skips disk access checks during startup. | `false` |
| `header` | Adds specified HTTP header to requests. Can be specified multiple times. | - |
| `server_side_encryption_customer_key_base64` | Required headers for accessing S3 objects with SSE-C encryption. | - |
| `server_side_encryption_kms_key_id` | Required headers for accessing S3 objects with [SSE-KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). Empty string uses AWS managed S3 key. | - |
| `server_side_encryption_kms_encryption_context` | Encryption context header for SSE-KMS (used with `server_side_encryption_kms_key_id`). | - |
| `server_side_encryption_kms_bucket_key_enabled` | Enables S3 bucket keys for SSE-KMS (used with `server_side_encryption_kms_key_id`). | Matches bucket-level setting |
| `s3_max_put_rps` | Maximum PUT requests per second before throttling. | `0` (unlimited) |
| `s3_max_put_burst` | Maximum concurrent PUT requests before hitting RPS limit. | Same as `s3_max_put_rps` |
| `s3_max_get_rps` | Maximum GET requests per second before throttling. | `0` (unlimited) |
| `s3_max_get_burst` | Maximum concurrent GET requests before hitting RPS limit. | Same as `s3_max_get_rps` |
| `key_template` | Defines object key generation format using [re2](https://github.com/google/re2/wiki/Syntax) syntax. Requires `storage_metadata_write_full_object_key` flag. Incompatible with `root path` in `endpoint`. Requires `key_compatibility_prefix`. | - |
| `key_compatibility_prefix` | Required with `key_template`. Specifies the previous `root path` from `endpoint` for reading older metadata versions. | - |
| `read_only` | Only allowing reading from the disk. | - |
> [!NOTE]
>
> Google Cloud Storage (GCS) is also supported using the type `s3`. See [GCS backed MergeTree](/docs/integrations/connectors/data-sources/gcs).
### Using Plain Storage
In `22.10` a new disk type `s3_plain` was introduced, which provides a write-once storage.
Configuration parameters for it are the same as for the `s3` disk type.
Unlike the `s3` disk type, it stores data as is. In other words,
instead of having randomly generated blob names, it uses normal file names
(the same way as ClickHouse stores files on local disk) and does not store any
metadata locally. For example, it is derived from data on `s3`.
This disk type allows keeping a static version of the table, as it does not
allow executing merges on the existing data and does not allow inserting of new
data. A use case for this disk type is to create backups on it, which can be done
via `BACKUP TABLE data TO Disk('plain_disk_name', 'backup_name')`. Afterward,
you can do `RESTORE TABLE data AS data_restored FROM Disk('plain_disk_name', 'backup_name')`
or use `ATTACH TABLE data (...) ENGINE = MergeTree() SETTINGS disk = 'plain_disk_name'`.
Configuration:
```xml theme={null}
s3_plain
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
```
Starting from `24.1` it is possible configure any object storage disk (`s3`, `azure`, `hdfs` (unsupported), `local`) using
the `plain` metadata type.
Configuration:
```xml theme={null}
object_storage
azure
plain
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
```
### Using S3 Plain Rewritable Storage
A new disk type `s3_plain_rewritable` was introduced in `24.4`.
Similar to the `s3_plain` disk type, it does not require additional storage for
metadata files. Instead, metadata is stored in S3.
Unlike the `s3_plain` disk type, `s3_plain_rewritable` allows executing merges
and supports `INSERT` operations.
Replication of tables is not supported.
[Mutations](/docs/reference/statements/alter/index#mutations) and `ALTER` operations
are supported unless hard links are disabled with `enable_hard_links`, see below.
A use case for this disk type is for non-replicated `MergeTree` tables. Although
the `s3` disk type is suitable for non-replicated `MergeTree` tables, you may opt
for the `s3_plain_rewritable` disk type if you do not require local metadata
for the table. This could be useful, for example, for system tables.
Configuration:
```xml theme={null}
s3_plain_rewritable
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
```
is equal to
```xml theme={null}
object_storage
s3
plain_rewritable
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
```
Starting from `24.5` it is possible to configure any object storage disk
(`s3`, `azure`, `local`) using the `plain_rewritable` metadata type.
#### Metadata snapshot
The metadata of a `plain_rewritable` disk is stored in the object storage as a
`prefix.path` object per directory. Building the in-memory state from these
objects requires listing the metadata directory, reading every `prefix.path`
and listing every directory, which is tens of thousands of requests even for a
disk with a single table.
To speed this up, the disk that writes the data also writes a compact
snapshot of the whole state to `__meta/snapshot.bin` (a versioned binary
format compressed with `ZSTD`) for read-only disks pointing to the same
location (`readonly` set to `true`, or a disk attached with `read_only` set
to `true`).
A read-only disk loads the whole state from the snapshot in a single request,
and its periodic refreshes (see the `MergeTree` setting `refresh_parts_interval`)
check the snapshot and read it again only if it was rewritten. It does not list
the metadata directory or read `prefix.path` at all: the only listing is of the
markers of unfinished operations, which is a single request, so that a file
that the writer is replacing right now is read from its backup. A read-only disk works on a
delayed state anyway, relying on the lifetime of the old parts of `MergeTree`
tables, and the snapshot is a delayed state of the same kind: the changes of
the writer become visible to the readers when the writer rewrites the snapshot.
This includes the files that the writer replaced or rewrote in place: until
the snapshot is rewritten, a reader sees them with their previous size and
blob, which may already be overwritten or removed, exactly as a reader that
lists the object storage sees them until its next refresh. The snapshot only
extends this window by at most `metadata_snapshot_write_delay_ms`.
Only if there is no snapshot (for example, the writer has it disabled or runs
an older version) or it cannot be read, the read-only disk lists the object
storage.
The writer has to have the latest and precise state, so it never reads the
snapshot: it lists the object storage on startup as usual, and writes the
snapshot after it. A writer with `metadata_snapshot_enabled` set to `false`
removes a snapshot left from before on startup, so that the read-only disks do
not keep using a stale state. If a snapshot cannot be read, or there is a
malformed marker of an unfinished operation, the read-only disk lists the
object storage, and does not use that snapshot until it is rewritten.
The snapshot is used only on object storages that replace an object
atomically, so that a read-only disk never sees a partially written snapshot:
`S3`, Azure Blob Storage and local. On `HDFS` and on Azure Data Lake Storage
Gen2 endpoints (such as OneLake) the snapshot is neither written nor read, and
the disks list the object storage as without the snapshot; a snapshot left
there from before is removed by the writer on startup.
| Setting | Description | Default |
| - | - | - |
| `metadata_snapshot_enabled` | Write the snapshot after modifications (if the object storage is writable), or load the state from it instead of listing the object storage (if the object storage is read-only). | `true` |
| `metadata_snapshot_write_delay_ms` | The snapshot is rewritten in the background, not more often than the given number of milliseconds, as long as there were modifications. `0` rewrites it right after every modification, in the committing thread, which makes every commit as expensive as serializing the metadata of the whole disk. The snapshot is also written on server shutdown, and removed right away when the disk becomes empty. | `1000` |
Both settings are read only when the disk is created: a change takes effect
after a restart of the server, `SYSTEM RELOAD CONFIG` does not apply it. The
same holds for the role of the disk: whether it writes or reads the snapshot is
decided by `readonly` when the disk is created, so changing `readonly` requires
a restart of the server as well.
Older versions ignore `__meta/snapshot.bin`: they neither read it nor rewrite or
remove it. So when the writer is downgraded to a version without snapshots
while the read-only disks keep running a newer version, the snapshot left by
the writer must be retired, otherwise the read-only disks keep using it and do
not see the changes made after the downgrade. Either remove
`__meta/snapshot.bin` after stopping the newer writer and before starting the
older one, or set `metadata_snapshot_enabled` to `false` on the read-only disks
and restart them, so that they list the object storage.
```xml theme={null}
s3_plain_rewritable
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
10000
```
The events `DiskPlainRewritableSnapshotRead`, `DiskPlainRewritableSnapshotWritten`
and `DiskPlainRewritableSnapshotWriteFailed` in
[system.events](/docs/reference/system-tables/events) show how the snapshot is used.
#### Hard links on plain rewritable disks
`ALTER` operations and mutations of a `MergeTree` table share the unchanged files
between a part and its mutated successor with hard links, instead of copying
them. The `enable_hard_links` setting (`1` by default) allows a disk with the
`plain_rewritable` metadata to create real hard links, which unlocks these
operations. Set it to `0` to keep the disk readable by servers older than `26.10`;
then the hard links are emulated by copies, and `ALTER` operations that modify
data and mutations are not supported:
```xml theme={null}
s3_plain_rewritable
https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/
1
0
```
A hard link makes the metadata of a directory list its files explicitly, together
with the keys of the objects storing them, so that a file can refer to an object
stored under the prefix of another directory. Consider the following:
* A server that does not have the `enable_hard_links` setting cannot read a
directory in this form - it reads the whole metadata of the directory as its
logical path. A disk that has such directories is therefore not readable after
a downgrade to such a version, and cannot be shared with it. Directories that
never had hard links keep the previous layout, so a disk stays readable by
older servers until the first operation that creates a hard link - a mutation,
an `ALTER`, a `FREEZE` or a backup. Turning the setting off again does not
convert the existing directories back.
* The setting is read only when the disk is created: a change takes effect after
a restart of the server, `SYSTEM RELOAD CONFIG` does not apply it.
* An object whose last link is removed is deleted in two steps: the metadata of
the directory is rewritten first, and the object is deleted after that. If the
server is killed in between, the object is left in the bucket and is not
referenced by anything, so it has to be deleted manually.
### Using Azure Blob Storage
`MergeTree` family table engines can store data to [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/)
using a disk with type `azure_blob_storage`.
Configuration markup:
```xml theme={null}
...
azure_blob_storage
http://account.blob.core.windows.net
container
account
pass123
/var/lib/clickhouse/disks/blob_storage_disk/
false
...
```
#### Connection parameters
| Parameter | Description | Default Value |
| - | - | - |
| `storage_account_url` (Required) | Azure Blob Storage account URL. Examples: `http://account.blob.core.windows.net` or `http://azurite1:10000/devstoreaccount1`. | - |
| `container_name` | Target container name. | `default-container` |
| `container_already_exists` | Controls container creation behavior: - `false`: Creates a new container - `true`: Connects directly to existing container - Unset: Checks if container exists, creates if needed | - |
Authentication parameters (the disk will try all available methods **and** Managed Identity Credential):
| Parameter | Description |
| - | - |
| `connection_string` | For authentication using a connection string. |
| `account_name` | For authentication using Shared Key (used with `account_key`). |
| `account_key` | For authentication using Shared Key (used with `account_name`). |
#### Limit parameters
| Parameter | Description |
| - | - |
| `max_single_part_upload_size` | Maximum size of a single block upload to Blob Storage. |
| `min_bytes_for_seek` | Minimum size of a seekable region. |
| `max_single_read_retries` | Maximum number of attempts to read a chunk of data from Blob Storage. |
| `max_single_download_retries` | Maximum number of attempts to download a readable buffer from Blob Storage. |
| `thread_pool_size` | Maximum number of threads for `IDiskRemote` instantiation. |
| `max_inflight_parts_for_one_file` | Maximum number of concurrent put requests for a single object. |
#### Other parameters
| Parameter | Description | Default Value |
| - | - | - |
| `metadata_path` | Local filesystem path to store metadata files for Blob Storage. | `/var/lib/clickhouse/disks//` |
| `skip_access_check` | If `true`, skips disk access checks during startup. | `false` |
| `metadata_keep_free_space_bytes` | Amount of free metadata disk space to reserve. | - |
Examples of working configurations can be found in integration tests directory (see e.g. [test\_merge\_tree\_azure\_blob\_storage](https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/test_merge_tree_azure_blob_storage/configs/config.d/storage_conf.xml) or [test\_azure\_blob\_storage\_zero\_copy\_replication](https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/test_azure_blob_storage_zero_copy_replication/configs/config.d/storage_conf.xml)).
> [!NOTE]
>
> **Zero-copy replication is not ready for production**
>
> Zero-copy replication is disabled by default in ClickHouse version 22.8 and higher. This feature is not recommended for production use.
## Using HDFS storage (Unsupported)
In this sample configuration:
* the disk is of type `hdfs` (unsupported)
* the data is hosted at `hdfs://hdfs1:9000/clickhouse/`
By the way, HDFS is unsupported and therefore there might be issues when using it. Feel free to make a pull request with the fix if any issue arises.
```xml theme={null}
hdfs
hdfs://hdfs1:9000/clickhouse/
true
local
/
hdfs
hdd
```
Keep in mind that HDFS may not work in corner cases.
### Using Data Encryption
You can encrypt the data stored on [S3](/docs/reference/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-s3), or [HDFS](#using-hdfs-storage-unsupported) (unsupported) external disks, or on a local disk. To turn on the encryption mode, in the configuration file you must define a disk with the type `encrypted` and choose a disk on which the data will be saved. An `encrypted` disk ciphers all written files on the fly, and when you read files from an `encrypted` disk it deciphers them automatically. So you can work with an `encrypted` disk like with a normal one.
Example of disk configuration:
```xml theme={null}
local
/path1/
encrypted
disk1
path2/
_16_ascii_chars_
```
For example, when ClickHouse writes data from some table to a file `store/all_1_1_0/data.bin` to `disk1`, then in fact this file will be written to the physical disk along the path `/path1/store/all_1_1_0/data.bin`.
When writing the same file to `disk2`, it will actually be written to the physical disk at the path `/path1/path2/store/all_1_1_0/data.bin` in encrypted mode.
### Required Parameters
| Parameter | Type | Description |
| - | - | - |
| `type` | String | Must be set to `encrypted` to create an encrypted disk. |
| `disk` | String | Type of disk to use for underlying storage. |
| `key` | Uint64 | Key for encryption and decryption. Can be specified in hexadecimal using `key_hex`. Multiple keys can be specified using the `id` attribute. |
### Optional Parameters
| Parameter | Type | Default | Description |
| - | - | - | - |
| `path` | String | Root directory | Location on the disk where data will be saved. |
| `current_key_id` | String | - | The key ID used for encryption. All specified keys can be used for decryption. |
| `algorithm` | Enum | `AES_128_CTR` | Encryption algorithm. Options: - `AES_128_CTR` (16-byte key) - `AES_192_CTR` (24-byte key) - `AES_256_CTR` (32-byte key) |
Example of disk configuration:
```xml theme={null}
s3
...
encrypted
disk_s3
AES_128_CTR
00112233445566778899aabbccddeeff
ffeeddccbbaa99887766554433221100
1
```
### Using local cache
It is possible to configure local cache over disks in storage configuration starting from version 22.3.
For versions 22.3 - 22.7 cache is supported only for `s3` disk type. For versions >= 22.8 cache is supported for any disk type: S3, Azure, Local, Encrypted, etc.
For versions >= 23.5 cache is supported only for remote disk types: S3, Azure, HDFS (unsupported).
Cache uses the `cache_policy` setting (default `SLRU`) for eviction, listed on [`system.filesystem_cache_settings`](/docs/operations/system-tables/filesystem_cache_settings). The full list of cache-disk settings is in [`system.filesystem_cache_settings`](/docs/operations/system-tables/filesystem_cache_settings).
Example of configuration for versions later or equal to 22.8:
```xml theme={null}
s3
...
... s3 configuration ...
cache
s3
/s3_cache/
10Gi
cache
```
Example of configuration for versions earlier than 22.8:
```xml theme={null}
s3
...
... s3 configuration ...
1
10737418240
s3
```
File Cache **disk configuration settings** (common settings):
These settings should be defined in the disk configuration section. This table lists commonly used cache-disk knobs, not every key accepted from config. Runtime values and the full setting list are in [`system.filesystem_cache_settings`](/docs/operations/system-tables/filesystem_cache_settings).
| Parameter | Type | Default | Description |
| - | - | - | - |
| `path` | String | - | **Required**. Path to the directory where cache will be stored. |
| `max_size` | Size | - | Maximum cache size in bytes or readable format (e.g., `10Gi`). **Required unless** `max_size_ratio_to_total_space` is set; the two cannot be set together. Files are evicted using the policy from `cache_policy` (SLRU by default) when the limit is reached. Supports `ki`, `Mi`, `Gi` formats (since v22.10). |
| `max_size_ratio_to_total_space` | Double | - | Alternative to `max_size`: fraction of total disk space used as the cache size, in `(0, 1]`. **Required unless** `max_size` is set. Cannot be specified together with `max_size`. |
| `cache_policy` | Enum | `SLRU` | Cache eviction policy. Possible values: `LRU`, `SLRU` (default), `SLRU_OVERCOMMIT`, `LRU_OVERCOMMIT`. Overcommit policies require a build with distributed-cache support and cannot be combined with `use_split_cache` (startup rejects both cases). |
| `slru_size_ratio` | Double | `0.6` | For `SLRU` / `SLRU_OVERCOMMIT`, ratio of protected cache to probationary cache. |
| `reserve_granularity` | Size | `4Mi` | When reserving space for a file-segment download, reserve at least this many bytes ahead (capped at the file-segment size). `0` reserves exactly the requested size. |
| `cache_on_write_operations` | Boolean | `false` | Enables write-through cache for `INSERT` queries and background merges. Can be overridden per query with `enable_filesystem_cache_on_write_operations`. |
| `cache_hits_threshold` | Integer | `0` | Deprecated; has no effect. |
| `enable_bypass_cache_with_threshold` | Boolean | `false` | Skips cache for large read ranges. |
| `bypass_cache_threshold` | Size | `256Mi` | Read range size that triggers cache bypass (requires `enable_bypass_cache_with_threshold`). |
| `max_file_segment_size` | Size | `32Mi` | Maximum size of a single cache file in bytes or readable format. For `SLRU` / `SLRU_OVERCOMMIT`, must not exceed the size of each SLRU queue, `max_size * min(slru_size_ratio, 1 - slru_size_ratio)` (of each part with `use_split_cache`); a larger value is rejected. |
| `boundary_alignment` | Size | `4Mi` | Alignment of file segment boundaries in the cache. Must not exceed `max_file_segment_size` (startup rejects a larger value). |
| `max_elements` | Integer | `10000000` | Maximum number of cache files. |
| `load_metadata_threads` | Integer | `16` | Number of threads for loading cache metadata at startup. |
| `background_download_threads` | UInt64 | `5` | Number of threads for background download of file segments. `0` disables background download. |
| `background_download_queue_size_limit` | UInt64 | `5000` | Maximum number of segments waiting in the background download queue. `0` means no limit (the queue is not disabled). Background download itself is disabled only when `background_download_threads` is `0`. |
| `background_download_max_file_segment_size` | Size | `4Mi` | Alignment-rounded threshold for how far a segment is prefetched in the background. The download size is rounded up to `boundary_alignment`, so this is not a hard cap (for example `1Mi` with `boundary_alignment` `4Mi` still downloads `4Mi`). |
| `load_metadata_asynchronously` | Boolean | `false` | Load cache metadata asynchronously on server startup. |
| `use_real_disk_size` | Boolean | `false` | Accounts cache reservations in filesystem block-aligned units instead of reserved bytes. This approximates the reservation footprint on disk but is not the exact allocated size; it affects `current_size` of this cache in `system.filesystem_cache_settings`. The process-wide `FilesystemCacheSize` metric and the eviction-size profile events are summed over all caches, so they uniformly reflect the reservation footprint only when all caches use the same mode. The per-segment `size` in `system.filesystem_cache` remains the logical range size; use `downloaded_size` for bytes written to a partial segment. Restart is required to change this setting. |
| `use_split_cache` | Boolean | `false` | Use separation of files to system/data. Not supported with `SLRU_OVERCOMMIT` or `LRU_OVERCOMMIT`. |
| `split_cache_ratio` | Double | `0.1` | Ratio of system segment to total size of cache for split\_cache. |
> **Note**: Size values support units like `ki`, `Mi`, `Gi`, etc. (e.g., `10Gi`).
## File Cache Query/Profile Settings
| Setting | Type | Default | Description |
| - | - | - | - |
| `enable_filesystem_cache` | Boolean | `true` | Enables/disables cache usage per query, even when using a `cache` disk type. |
| `read_from_filesystem_cache_if_exists_otherwise_bypass_cache` | Boolean | `false` | When enabled, uses cache only if data exists; new data won't be cached. |
| `enable_filesystem_cache_on_write_operations` | Boolean | `false` (Cloud: `true`) | Enables write-through cache. Requires `cache_on_write_operations` in cache config. |
| `enable_filesystem_cache_log` | Boolean | `false` | Enables detailed cache usage logging to `system.filesystem_cache_log`. |
| `filesystem_cache_allow_background_download` | Boolean | `true` | Allows partially downloaded segments to be finished in the background. Disable to keep downloads in the foreground for the current query/session. |
> [!WARNING]
>
> Cache configuration settings and cache query settings correspond to the latest ClickHouse version,
> for earlier versions something might not be supported.
#### Cache system tables
| Table Name | Description | Requirements |
| - | - | - |
| `system.filesystem_cache` | Displays the current state of the filesystem cache. | None |
| `system.filesystem_cache_log` | Provides detailed cache usage statistics per query. | Requires `enable_filesystem_cache_log = true` |
#### Cache commands
##### `SYSTEM CLEAR|DROP FILESYSTEM CACHE () (ON CLUSTER)` -- `ON CLUSTER`
This command is only supported when no `` is provided
##### `SHOW FILESYSTEM CACHES`
Show a list of filesystem caches which were configured on the server.
(For versions less than or equal to `22.8` the command is named `SHOW CACHES`)
```sql title="Query" theme={null}
SHOW FILESYSTEM CACHES
```
```text title="Response" theme={null}
┌─Caches────┐
│ s3_cache │
└───────────┘
```
##### `DESCRIBE FILESYSTEM CACHE ''`
Show cache configuration and some general statistics for a specific cache.
Cache name can be taken from `SHOW FILESYSTEM CACHES` command. (For versions less
than or equal to `22.8` the command is named `DESCRIBE CACHE`)
```sql title="Query" theme={null}
DESCRIBE FILESYSTEM CACHE 's3_cache'
```
```text title="Response" theme={null}
┌────max_size─┬─max_elements─┬─max_file_segment_size─┬─boundary_alignment─┬─cache_on_write_operations─┬─cache_hits_threshold─┬─current_size─┬─current_elements─┬─path───────┬─background_download_threads─┬─enable_bypass_cache_with_threshold─┐
│ 10000000000 │ 1048576 │ 104857600 │ 4194304 │ 1 │ 0 │ 3276 │ 54 │ /s3_cache/ │ 2 │ 0 │
└─────────────┴──────────────┴───────────────────────┴────────────────────┴───────────────────────────┴──────────────────────┴──────────────┴──────────────────┴────────────┴─────────────────────────────┴────────────────────────────────────┘
```
| Cache current metrics | Cache asynchronous metrics | Cache profile events |
| - | - | - |
| `FilesystemCacheSize` | `FilesystemCacheBytes` | `CachedReadBufferReadFromSourceBytes`, `CachedReadBufferReadFromCacheBytes` |
| `FilesystemCacheElements` | `FilesystemCacheFiles` | `CachedReadBufferReadFromSourceMicroseconds`, `CachedReadBufferReadFromCacheMicroseconds` |
| | | `CachedReadBufferCacheWriteBytes`, `CachedReadBufferCacheWriteMicroseconds` |
| | | `CachedWriteBufferCacheWriteBytes`, `CachedWriteBufferCacheWriteMicroseconds` |
### Using static Web storage (read-only)
This is a read-only disk. Its data is only read and never modified. A new table
is loaded to this disk via `ATTACH TABLE` query (see example below). Local disk
is not actually used, each `SELECT` query will result in a `http` request to
fetch required data. All modification of the table data will result in an
exception, i.e. the following types of queries are not allowed: [`CREATE TABLE`](/docs/reference/statements/create/table),
[`ALTER TABLE`](/docs/reference/statements/alter/index), [`RENAME TABLE`](/docs/reference/statements/rename#rename-table),
[`DETACH TABLE`](/docs/reference/statements/detach) and [`TRUNCATE TABLE`](/docs/reference/statements/truncate).
Web storage can be used for read-only purposes. An example use is for hosting
sample data, or for migrating data. There is a tool `clickhouse-static-files-uploader`,
which prepares a data directory for a given table (`SELECT data_paths FROM system.tables WHERE name = 'table_name'`).
For each table you need, you get a directory of files. These files can be uploaded
to, for example, a web server with static files. After this preparation,
you can load this table into any ClickHouse server via `DiskWeb`.
In this sample configuration:
* the disk is of type `web`
* the data is hosted at `http://nginx:80/test1/`
* a cache on local storage is used
```xml theme={null}
web
http://nginx:80/test1/
cache
web
cached_web_cache/
100000000