> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Lakekeeper 目录

> 本指南将逐步介绍如何使用 ClickHouse 和 Lakekeeper 目录 查询数据。

export const ExperimentalBadge = () => {
  return <div className="experimentalBadge">
            <div className="experimentalIcon">
            <svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
                <path strokeWidth="1.25" d="M5.5 2H10.5" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
                <path strokeWidth="1.25" d="M9.50015 2V6.19625L13.4283 12.7425C13.4738 12.8183 13.4985 12.9049 13.4996 12.9934C13.5008 13.0818 13.4785 13.169 13.435 13.246C13.3914 13.323 13.3283 13.3871 13.2519 13.4317C13.1755 13.4764 13.0886 13.4999 13.0002 13.5H3.00015C2.91164 13.5 2.8247 13.4766 2.74822 13.432C2.67174 13.3874 2.60847 13.3233 2.56487 13.2463C2.52126 13.1693 2.49889 13.082 2.50004 12.9935C2.50119 12.905 2.52582 12.8184 2.5714 12.7425L6.50015 6.19625V2" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
                <path strokeWidth="1.25" d="M4.47656 9.56754C5.30344 9.41254 6.47656 9.47942 7.99969 10.25C10.0153 11.2707 11.4216 11.0569 12.2184 10.7282" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
            </svg>
        </div>
            Experimental 功能。 <u><a href="/docs/docs/beta-and-experimental-features#experimental-features">了解详情。</a></u>
        </div>;
};

<ExperimentalBadge />

<Note>
  与 Lakekeeper 目录 的集成仅适用于 Iceberg 表。
  此集成同时支持 AWS S3 和其他云存储提供商。
</Note>

ClickHouse 支持与多个目录 (Unity、Glue、REST、Polaris 等) 集成。本指南将逐步介绍如何使用 ClickHouse 和 [Lakekeeper](https://docs.lakekeeper.io/) 目录查询你的数据。

Lakekeeper 是 Apache Iceberg 的开源 REST 目录 实现，提供：

* 高性能且具备**原生 Rust**实现，兼顾性能与可靠性
* 符合 Iceberg REST 目录 规范的 **REST API**
* 与兼容 S3 存储的**云存储**集成

<Note>
  由于此功能仍处于 Experimental 阶段，你需要使用以下命令启用：
  `SET allow_experimental_database_iceberg = 1;`
</Note>

<div id="local-development-setup">
  ## 本地开发环境设置
</div>

对于本地开发和测试，你可以使用容器化的 Lakekeeper 部署。这种方式非常适合学习、原型设计和开发环境。

<div id="local-prerequisites">
  ### 前置条件
</div>

1. **Docker and Docker Compose**：确保已安装并运行 Docker
2. **示例设置**：你可以使用 Lakekeeper 的 docker-compose 配置

<div id="setting-up-local-lakekeeper-catalog">
  ### 设置本地 Lakekeeper 目录
</div>

你可以使用官方的 [Lakekeeper docker-compose 配置](https://github.com/lakekeeper/lakekeeper/tree/main/examples/minimal)，它提供了一个完整的环境，其中包含 Lakekeeper、PostgreSQL 元数据后端，以及用作对象存储的 MinIO。

**步骤 1：** 创建一个用于运行此示例的新文件夹，然后创建一个名为 `docker-compose.yml` 的文件，内容如下：

```yaml theme={null}
version: '3.8'

services:
  lakekeeper:
    image: quay.io/lakekeeper/catalog:latest
    environment:
      - LAKEKEEPER__PG_ENCRYPTION_KEY=This-is-NOT-Secure!
      - LAKEKEEPER__PG_DATABASE_URL_READ=postgresql://postgres:postgres@db:5432/postgres
      - LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://postgres:postgres@db:5432/postgres
      - RUST_LOG=info
    command: ["serve"]
    healthcheck:
      test: ["CMD", "/home/nonroot/lakekeeper", "healthcheck"]
      interval: 1s
      timeout: 10s
      retries: 10
      start_period: 30s
    depends_on:
      migrate:
        condition: service_completed_successfully
      db:
        condition: service_healthy
      minio:
        condition: service_healthy
    ports:
      - 8181:8181
    networks:
      - iceberg_net

  migrate:
    image: quay.io/lakekeeper/catalog:latest-main
    environment:
      - LAKEKEEPER__PG_ENCRYPTION_KEY=This-is-NOT-Secure!
      - LAKEKEEPER__PG_DATABASE_URL_READ=postgresql://postgres:postgres@db:5432/postgres
      - LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://postgres:postgres@db:5432/postgres
      - RUST_LOG=info
    restart: "no"
    command: ["migrate"]
    depends_on:
      db:
        condition: service_healthy
    networks:
      - iceberg_net

  bootstrap:
    image: curlimages/curl
    depends_on:
      lakekeeper:
        condition: service_healthy
    restart: "no"
    command:
      - -w
      - "%{http_code}"
      - "-X"
      - "POST"
      - "-v"
      - "http://lakekeeper:8181/management/v1/bootstrap"
      - "-H"
      - "Content-Type: application/json"
      - "--data"
      - '{"accept-terms-of-use": true}'
      - "-o"
      - "/dev/null"
    networks:
      - iceberg_net

  initialwarehouse:
    image: curlimages/curl
    depends_on:
      lakekeeper:
        condition: service_healthy
      bootstrap:
        condition: service_completed_successfully
    restart: "no"
    command:
      - -w
      - "%{http_code}"
      - "-X"
      - "POST"
      - "-v"
      - "http://lakekeeper:8181/management/v1/warehouse"
      - "-H"
      - "Content-Type: application/json"
      - "--data"
      - '{"warehouse-name": "demo", "project-id": "00000000-0000-0000-0000-000000000000", "storage-profile": {"type": "s3", "bucket": "warehouse-rest", "key-prefix": "", "assume-role-arn": null, "endpoint": "http://minio:9000", "region": "local-01", "path-style-access": true, "flavor": "minio", "sts-enabled": true}, "storage-credential": {"type": "s3", "credential-type": "access-key", "aws-access-key-id": "minio", "aws-secret-access-key": "ClickHouse_Minio_P@ssw0rd"}}'
      - "-o"
      - "/dev/null"
    networks:
      - iceberg_net

  db:
    image: bitnami/postgresql:16.3.0
    environment:
      - POSTGRESQL_USERNAME=postgres
      - POSTGRESQL_PASSWORD=postgres
      - POSTGRESQL_DATABASE=postgres
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres -p 5432 -d postgres"]
      interval: 2s
      timeout: 10s
      retries: 5
      start_period: 10s
    volumes:
      - postgres_data:/bitnami/postgresql
    networks:
      - iceberg_net

  minio:
    image: bitnami/minio:2025.4.22
    environment:
      - MINIO_ROOT_USER=minio
      - MINIO_ROOT_PASSWORD=ClickHouse_Minio_P@ssw0rd
      - MINIO_API_PORT_NUMBER=9000
      - MINIO_CONSOLE_PORT_NUMBER=9001
      - MINIO_SCHEME=http
      - MINIO_DEFAULT_BUCKETS=warehouse-rest
    networks: 
      iceberg_net:
        aliases:
          - warehouse-rest.minio
    ports:
      - "9002:9000"
      - "9003:9001"
    healthcheck:
      test: ["CMD", "mc", "ls", "local", "|", "grep", "warehouse-rest"]
      interval: 2s
      timeout: 10s
      retries: 3
      start_period: 15s
    volumes:
      - minio_data:/bitnami/minio/data

  clickhouse:
    image: clickhouse/clickhouse-server:head
    container_name: lakekeeper-clickhouse
    user: '0:0'  # Ensures root permissions
    ports:
      - "8123:8123"
      - "9000:9000"
    volumes:
      - clickhouse_data:/var/lib/clickhouse
      - ./clickhouse/data_import:/var/lib/clickhouse/data_import  # 挂载数据集文件夹
    networks:
      - iceberg_net
    environment:
      - CLICKHOUSE_DB=default
      - CLICKHOUSE_USER=default
      - CLICKHOUSE_DO_NOT_CHOWN=1
      - CLICKHOUSE_PASSWORD=
    depends_on:
      lakekeeper:
        condition: service_healthy
      minio:
        condition: service_healthy

volumes:
  postgres_data:
  minio_data:
  clickhouse_data:

networks:
  iceberg_net:
    driver: bridge
```

\*\*第 2 步：\*\*运行以下命令以启动服务：

```bash theme={null}
docker compose up -d
```

\*\*步骤 3：\*\*等待所有服务就绪。你可以查看日志：

```bash theme={null}
docker-compose logs -f
```

<Note>
  Lakekeeper 设置要求先将样本数据加载到 Iceberg 表中。请确保环境中已创建这些表并填充数据，然后再尝试通过 ClickHouse 查询它们。表是否可用取决于具体的 docker-compose 设置以及样本数据加载脚本。
</Note>

<div id="connecting-to-local-lakekeeper-catalog">
  ### 连接到本地 Lakekeeper 目录
</div>

连接到您的 ClickHouse 容器：

```bash theme={null}
docker exec -it lakekeeper-clickhouse clickhouse-client
```

然后创建与 Lakekeeper 目录 的数据库连接：

```sql theme={null}
SET allow_experimental_database_iceberg = 1;

CREATE DATABASE demo
ENGINE = DataLakeCatalog('http://lakekeeper:8181/catalog', 'minio', 'ClickHouse_Minio_P@ssw0rd')
SETTINGS catalog_type = 'rest', storage_endpoint = 'http://minio:9002/warehouse-rest', warehouse = 'demo'
```

<div id="querying-lakekeeper-catalog-tables-using-clickhouse">
  ## 使用 ClickHouse 查询 Lakekeeper 目录 中的表
</div>

建立连接后，你就可以开始通过 Lakekeeper 目录 进行查询。例如：

```sql theme={null}
USE demo;

SHOW TABLES;
```

如果你的环境中包含示例数据 (例如出租车数据集) ，你应该会看到如下这些表：

```response theme={null}
┌─name──────────┐
│ default.taxis │
└───────────────┘
```

<Note>
  如果你没有看到任何表，通常说明：

  1. 环境尚未创建样本表
  2. Lakekeeper 目录 service 尚未完全初始化
  3. 样本数据加载过程尚未完成

  你可以查看 Spark 日志，了解表创建进度：

  ```bash theme={null}
  docker-compose logs spark
  ```
</Note>

如需查询表 (如果可用) ：

```sql theme={null}
SELECT count(*) FROM `default.taxis`;
```

```response theme={null}
┌─count()─┐
│ 2171187 │
└─────────┘
```

<Info>
  **必须使用反引号**

  由于 ClickHouse 不支持多个命名空间，因此必须使用反引号。
</Info>

要查看该表的 DDL：

```sql theme={null}
SHOW CREATE TABLE `default.taxis`;
```

```response theme={null}
┌─statement─────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE demo.`default.taxis`                                                             │
│ (                                                                                             │
│     `VendorID` Nullable(Int64),                                                               │
│     `tpep_pickup_datetime` Nullable(DateTime64(6)),                                           │
│     `tpep_dropoff_datetime` Nullable(DateTime64(6)),                                          │
│     `passenger_count` Nullable(Float64),                                                      │
│     `trip_distance` Nullable(Float64),                                                        │
│     `RatecodeID` Nullable(Float64),                                                           │
│     `store_and_fwd_flag` Nullable(String),                                                    │
│     `PULocationID` Nullable(Int64),                                                           │
│     `DOLocationID` Nullable(Int64),                                                           │
│     `payment_type` Nullable(Int64),                                                           │
│     `fare_amount` Nullable(Float64),                                                          │
│     `extra` Nullable(Float64),                                                                │
│     `mta_tax` Nullable(Float64),                                                              │
│     `tip_amount` Nullable(Float64),                                                           │
│     `tolls_amount` Nullable(Float64),                                                         │
│     `improvement_surcharge` Nullable(Float64),                                                │
│     `total_amount` Nullable(Float64),                                                         │
│     `congestion_surcharge` Nullable(Float64),                                                 │
│     `airport_fee` Nullable(Float64)                                                           │
│ )                                                                                             │
│ ENGINE = Iceberg('http://minio:9002/warehouse-rest/warehouse/default/taxis/', 'minio', '[HIDDEN]') │
└───────────────────────────────────────────────────────────────────────────────────────────────┘
```

<div id="loading-data-from-your-data-lake-into-clickhouse">
  ## 将数据湖中的数据加载到 ClickHouse
</div>

如果您需要将 Lakekeeper 目录 中的数据加载到 ClickHouse，请先创建一个本地 ClickHouse 表：

```sql theme={null}
CREATE TABLE taxis
(
    `VendorID` Int64,
    `tpep_pickup_datetime` DateTime64(6),
    `tpep_dropoff_datetime` DateTime64(6),
    `passenger_count` Float64,
    `trip_distance` Float64,
    `RatecodeID` Float64,
    `store_and_fwd_flag` String,
    `PULocationID` Int64,
    `DOLocationID` Int64,
    `payment_type` Int64,
    `fare_amount` Float64,
    `extra` Float64,
    `mta_tax` Float64,
    `tip_amount` Float64,
    `tolls_amount` Float64,
    `improvement_surcharge` Float64,
    `total_amount` Float64,
    `congestion_surcharge` Float64,
    `airport_fee` Float64
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(tpep_pickup_datetime)
ORDER BY (VendorID, tpep_pickup_datetime, PULocationID, DOLocationID);
```

然后通过 `INSERT INTO SELECT` 从 Lakekeeper 目录 表中导入数据：

```sql theme={null}
INSERT INTO taxis 
SELECT * FROM demo.`default.taxis`;
```