SeaweedFS catalog

Integration with the SeaweedFS catalog works with Iceberg tables only.

ClickHouse supports integration with multiple catalogs (Unity, Glue, REST, Polaris, etc.). This guide will walk you through the steps to query your data using ClickHouse and the SeaweedFS catalog. SeaweedFS is an open-source distributed file and object store with an S3-compatible gateway. Its S3 Table Buckets provide both halves of an Iceberg deployment: the embedded Iceberg REST catalog serves the table metadata, and the table bucket stores the table data as Parquet files behind the same S3 endpoint:

Single service - catalog metadata and Parquet data are served by one process, with no separate metadata database
REST API compliance with the Iceberg REST catalog specification
Server-side maintenance - automatic Parquet compaction and snapshot expiration, with no external maintenance service

As this feature is experimental, you will need to enable it using: SET allow_experimental_database_iceberg = 1;

Local development setup

For local development and testing, you can run SeaweedFS and ClickHouse with Docker Compose. This approach is ideal for learning, prototyping, and development environments.

Prerequisites

Docker and Docker Compose: Ensure Docker is installed and running
Versions: SeaweedFS 4.42 or later; ClickHouse 26.8 or later (versions back to 25.8 can read and insert, but creating tables through the catalog requires 26.8)
Python with PyIceberg (optional): used below to seed sample data

Setting up the local SeaweedFS catalog

Step 1: Create a new folder in which to run the example, then create a file s3config.json with the credentials for the S3 gateway and the catalog:

{
  "identities": [
    {
      "name": "analyst",
      "credentials": [
        {
          "accessKey": "tutorialkey",
          "secretKey": "tutorialsecret"
        }
      ],
      "actions": ["Admin", "Read", "Write", "List", "Tagging"]
    }
  ]
}

Step 2: Create a file docker-compose.yml with the following configuration:

services:
  seaweedfs:
    image: chrislusf/seaweedfs:latest
    command: mini -dir=/data -s3.config=/etc/seaweedfs/s3config.json -tableBucket=analytics
    ports:
      - "8333:8333"   # S3 endpoint
      - "8181:8181"   # Iceberg REST catalog
    volumes:
      - ./s3config.json:/etc/seaweedfs/s3config.json
      - seaweedfs_data:/data
    networks:
      - iceberg_net

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    container_name: seaweedfs-clickhouse
    ports:
      - "8123:8123"
      - "9000:9000"
    depends_on:
      - seaweedfs
    networks:
      - iceberg_net

volumes:
  seaweedfs_data:

networks:
  iceberg_net:
    driver: bridge

The mini command starts the whole SeaweedFS stack in a single container. The -tableBucket=analytics flag pre-creates an S3 Tables bucket named analytics, which serves as the Iceberg warehouse. Step 3: Run the following command to start the services:

docker compose up -d

Seeding sample data

The catalog starts out empty. Create a table and append a few rows with PyIceberg (pip install pyiceberg pyarrow):

import pyarrow as pa
from pyiceberg.catalog.rest import RestCatalog

catalog = RestCatalog(
    "seaweedfs",
    uri="http://localhost:8181",
    warehouse="s3://analytics",
    credential="tutorialkey:tutorialsecret",
    **{
        "s3.endpoint": "http://localhost:8333",
        "s3.access-key-id": "tutorialkey",
        "s3.secret-access-key": "tutorialsecret",
        "s3.region": "us-east-1",
        "s3.path-style-access": "true",
    },
)

rows = pa.table({
    "id": pa.array([1, 2, 3, 4, 5, 6], pa.int64()),
    "region": ["NA", "EU", "EU", "APAC", "NA", "EU"],
    "amount": pa.array([12.5, 40.0, 7.25, 99.9, 3.5, 61.0], pa.float64()),
})

catalog.create_namespace("sales")
table = catalog.create_table("sales.orders", schema=rows.schema)
table.append(rows)

Connecting to the local SeaweedFS catalog

Connect to your ClickHouse container:

docker exec -it seaweedfs-clickhouse clickhouse-client

Then create the database connection to the SeaweedFS catalog:

SET allow_experimental_database_iceberg = 1;

CREATE DATABASE lake
ENGINE = DataLakeCatalog('http://seaweedfs:8181/v1', 'tutorialkey', 'tutorialsecret')
SETTINGS catalog_type = 'rest',
    warehouse = 's3://analytics',
    storage_endpoint = 'http://seaweedfs:8333/analytics',
    catalog_credential = 'tutorialkey:tutorialsecret',
    oauth_server_uri = 'http://seaweedfs:8181/v1/oauth/tokens'

The engine arguments carry the S3 credentials ClickHouse uses to read table data, while catalog_credential and oauth_server_uri authenticate to the catalog itself through the OAuth2 client-credentials flow. SeaweedFS accepts the same access key and secret key for both.

Querying SeaweedFS catalog tables using ClickHouse

Now that the connection is in place, you can start querying via the SeaweedFS catalog. For example:

USE lake;

SHOW TABLES;

┌─name─────────┐
│ sales.orders │
└──────────────┘

Backticks requiredBackticks are required because ClickHouse doesn’t support more than one namespace.

To query a table:

SELECT region, sum(amount) AS total
FROM `sales.orders`
GROUP BY region
ORDER BY total DESC;

┌─region─┬──total─┐
│ EU     │ 108.25 │
│ APAC   │   99.9 │
│ NA     │     16 │
└────────┴────────┘

Creating tables and writing data from ClickHouse

You can also create tables in the SeaweedFS catalog and write to them directly from ClickHouse:

SET allow_experimental_database_iceberg = 1;
SET allow_experimental_insert_into_iceberg = 1;
SET write_full_path_in_iceberg_metadata = 1;

CREATE TABLE lake.`sales.returns` (id Int64, reason String)
ENGINE = IcebergS3('http://seaweedfs:8333/analytics/sales/returns/', 'tutorialkey', 'tutorialsecret');

INSERT INTO lake.`sales.returns` VALUES (1, 'damaged'), (2, 'wrong size');

SELECT * FROM lake.`sales.returns` ORDER BY id;

┌─id─┬─reason─────┐
│  1 │ damaged    │
│  2 │ wrong size │
└────┴────────────┘

The IcebergS3 engine clause names the storage path for the new table, and write_full_path_in_iceberg_metadata makes ClickHouse register the full table location with the catalog.

Creating tables through a catalog requires ClickHouse 26.8 or later. Versions 26.4 through 26.7 write the table files before registering the namespace, which SeaweedFS rejects unless the namespace already exists in the catalog; versions before 26.4 appear to succeed, but the table files are written to object storage without being registered in the catalog.

When ClickHouse commits an insert, the SeaweedFS catalog repairs metadata the experimental writer does not yet produce: it fills in missing field IDs in manifests, rewrites bucket-relative file paths as absolute locations, and stamps a default name mapping on the table. Strict readers such as PyIceberg and Spark can then read the rows ClickHouse wrote. This requires SeaweedFS 4.42 or later.

Loading data from your Data Lake into ClickHouse

If you need to load data from the SeaweedFS catalog into ClickHouse, start by creating a local ClickHouse table:

CREATE TABLE default.orders
(
    `id` Int64,
    `region` String,
    `amount` Float64
)
ENGINE = MergeTree()
ORDER BY (region, id);

Then load the data from your SeaweedFS catalog table via an INSERT INTO SELECT:

INSERT INTO default.orders
SELECT * FROM lake.`sales.orders`;

ClickHouse guides

Use case guides

Local development setup

Prerequisites

Setting up the local SeaweedFS catalog

Seeding sample data

Connecting to the local SeaweedFS catalog

Querying SeaweedFS catalog tables using ClickHouse

Creating tables and writing data from ClickHouse

Loading data from your Data Lake into ClickHouse

​Local development setup

​Prerequisites

​Setting up the local SeaweedFS catalog

​Seeding sample data

​Connecting to the local SeaweedFS catalog

​Querying SeaweedFS catalog tables using ClickHouse

​Creating tables and writing data from ClickHouse

​Loading data from your Data Lake into ClickHouse

Local development setup

Prerequisites

Setting up the local SeaweedFS catalog

Seeding sample data

Connecting to the local SeaweedFS catalog

Querying SeaweedFS catalog tables using ClickHouse

Creating tables and writing data from ClickHouse

Loading data from your Data Lake into ClickHouse