> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# SeaweedFS catalog

> In this guide, we will walk you through the steps to query your data using ClickHouse and the SeaweedFS Iceberg catalog.

export const ExperimentalBadge = () => {
  return <a href="https://clickhouse.com/docs/reference/settings/beta-and-experimental-features#experimental-features" className="experimentalBadge">
            <div className="experimentalIcon">
            <svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
                <path strokeWidth="1.25" d="M5.5 2H10.5" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
                <path strokeWidth="1.25" d="M9.50015 2V6.19625L13.4283 12.7425C13.4738 12.8183 13.4985 12.9049 13.4996 12.9934C13.5008 13.0818 13.4785 13.169 13.435 13.246C13.3914 13.323 13.3283 13.3871 13.2519 13.4317C13.1755 13.4764 13.0886 13.4999 13.0002 13.5H3.00015C2.91164 13.5 2.8247 13.4766 2.74822 13.432C2.67174 13.3874 2.60847 13.3233 2.56487 13.2463C2.52126 13.1693 2.49889 13.082 2.50004 12.9935C2.50119 12.905 2.52582 12.8184 2.5714 12.7425L6.50015 6.19625V2" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
                <path strokeWidth="1.25" d="M4.47656 9.56754C5.30344 9.41254 6.47656 9.47942 7.99969 10.25C10.0153 11.2707 11.4216 11.0569 12.2184 10.7282" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
            </svg>
        </div>
            Experimental feature
        </a>;
};

<ExperimentalBadge />

<Note>
  Integration with the SeaweedFS catalog works with Iceberg tables only.
</Note>

ClickHouse supports integration with multiple catalogs (Unity, Glue, REST, Polaris, etc.). This guide will walk you through the steps to query your data using ClickHouse and the [SeaweedFS](https://github.com/seaweedfs/seaweedfs) catalog.

SeaweedFS is an open-source distributed file and object store with an S3-compatible gateway. Its S3 Table Buckets provide both halves of an Iceberg deployment: the embedded Iceberg REST catalog serves the table metadata, and the table bucket stores the table data as Parquet files behind the same S3 endpoint:

* **Single service** - catalog metadata and Parquet data are served by one process, with no separate metadata database
* **REST API** compliance with the Iceberg REST catalog specification
* **Server-side maintenance** - automatic Parquet compaction and snapshot expiration, with no external maintenance service

<Note>
  As this feature is experimental, you will need to enable it using:
  `SET allow_experimental_database_iceberg = 1;`
</Note>

<h2 id="local-development-setup">
  Local development setup
</h2>

For local development and testing, you can run SeaweedFS and ClickHouse with Docker Compose. This approach is ideal for learning, prototyping, and development environments.

<h3 id="local-prerequisites">
  Prerequisites
</h3>

1. **Docker and Docker Compose**: Ensure Docker is installed and running
2. **Versions**: SeaweedFS 4.42 or later; ClickHouse 26.8 or later (versions back to 25.8 can read and insert, but creating tables through the catalog requires 26.8)
3. **Python with PyIceberg** (optional): used below to seed sample data

<h3 id="setting-up-local-seaweedfs-catalog">
  Setting up the local SeaweedFS catalog
</h3>

**Step 1:** Create a new folder in which to run the example, then create a file `s3config.json` with the credentials for the S3 gateway and the catalog:

```json theme={null}
{
  "identities": [
    {
      "name": "analyst",
      "credentials": [
        {
          "accessKey": "tutorialkey",
          "secretKey": "tutorialsecret"
        }
      ],
      "actions": ["Admin", "Read", "Write", "List", "Tagging"]
    }
  ]
}
```

**Step 2:** Create a file `docker-compose.yml` with the following configuration:

```yaml theme={null}
services:
  seaweedfs:
    image: chrislusf/seaweedfs:latest
    command: mini -dir=/data -s3.config=/etc/seaweedfs/s3config.json -tableBucket=analytics
    ports:
      - "8333:8333"   # S3 endpoint
      - "8181:8181"   # Iceberg REST catalog
    volumes:
      - ./s3config.json:/etc/seaweedfs/s3config.json
      - seaweedfs_data:/data
    networks:
      - iceberg_net

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    container_name: seaweedfs-clickhouse
    ports:
      - "8123:8123"
      - "9000:9000"
    depends_on:
      - seaweedfs
    networks:
      - iceberg_net

volumes:
  seaweedfs_data:

networks:
  iceberg_net:
    driver: bridge
```

The `mini` command starts the whole SeaweedFS stack in a single container. The `-tableBucket=analytics` flag pre-creates an S3 Tables bucket named `analytics`, which serves as the Iceberg warehouse.

**Step 3:** Run the following command to start the services:

```bash theme={null}
docker compose up -d
```

<h3 id="seeding-sample-data">
  Seeding sample data
</h3>

The catalog starts out empty. Create a table and append a few rows with PyIceberg (`pip install pyiceberg pyarrow`):

```python theme={null}
import pyarrow as pa
from pyiceberg.catalog.rest import RestCatalog

catalog = RestCatalog(
    "seaweedfs",
    uri="http://localhost:8181",
    warehouse="s3://analytics",
    credential="tutorialkey:tutorialsecret",
    **{
        "s3.endpoint": "http://localhost:8333",
        "s3.access-key-id": "tutorialkey",
        "s3.secret-access-key": "tutorialsecret",
        "s3.region": "us-east-1",
        "s3.path-style-access": "true",
    },
)

rows = pa.table({
    "id": pa.array([1, 2, 3, 4, 5, 6], pa.int64()),
    "region": ["NA", "EU", "EU", "APAC", "NA", "EU"],
    "amount": pa.array([12.5, 40.0, 7.25, 99.9, 3.5, 61.0], pa.float64()),
})

catalog.create_namespace("sales")
table = catalog.create_table("sales.orders", schema=rows.schema)
table.append(rows)
```

<h3 id="connecting-to-local-seaweedfs-catalog">
  Connecting to the local SeaweedFS catalog
</h3>

Connect to your ClickHouse container:

```bash theme={null}
docker exec -it seaweedfs-clickhouse clickhouse-client
```

Then create the database connection to the SeaweedFS catalog:

```sql theme={null}
SET allow_experimental_database_iceberg = 1;

CREATE DATABASE lake
ENGINE = DataLakeCatalog('http://seaweedfs:8181/v1', 'tutorialkey', 'tutorialsecret')
SETTINGS catalog_type = 'rest',
    warehouse = 's3://analytics',
    storage_endpoint = 'http://seaweedfs:8333/analytics',
    catalog_credential = 'tutorialkey:tutorialsecret',
    oauth_server_uri = 'http://seaweedfs:8181/v1/oauth/tokens'
```

The engine arguments carry the S3 credentials ClickHouse uses to read table data, while `catalog_credential` and `oauth_server_uri` authenticate to the catalog itself through the OAuth2 client-credentials flow. SeaweedFS accepts the same access key and secret key for both.

<h2 id="querying-seaweedfs-catalog-tables-using-clickhouse">
  Querying SeaweedFS catalog tables using ClickHouse
</h2>

Now that the connection is in place, you can start querying via the SeaweedFS catalog. For example:

```sql theme={null}
USE lake;

SHOW TABLES;
```

```response theme={null}
┌─name─────────┐
│ sales.orders │
└──────────────┘
```

<Info>
  **Backticks required**

  Backticks are required because ClickHouse doesn't support more than one namespace.
</Info>

To query a table:

```sql theme={null}
SELECT region, sum(amount) AS total
FROM `sales.orders`
GROUP BY region
ORDER BY total DESC;
```

```response theme={null}
┌─region─┬──total─┐
│ EU     │ 108.25 │
│ APAC   │   99.9 │
│ NA     │     16 │
└────────┴────────┘
```

<h2 id="creating-tables-and-writing-data-from-clickhouse">
  Creating tables and writing data from ClickHouse
</h2>

You can also create tables in the SeaweedFS catalog and write to them directly from ClickHouse:

```sql theme={null}
SET allow_experimental_database_iceberg = 1;
SET allow_experimental_insert_into_iceberg = 1;
SET write_full_path_in_iceberg_metadata = 1;

CREATE TABLE lake.`sales.returns` (id Int64, reason String)
ENGINE = IcebergS3('http://seaweedfs:8333/analytics/sales/returns/', 'tutorialkey', 'tutorialsecret');

INSERT INTO lake.`sales.returns` VALUES (1, 'damaged'), (2, 'wrong size');

SELECT * FROM lake.`sales.returns` ORDER BY id;
```

```response theme={null}
┌─id─┬─reason─────┐
│  1 │ damaged    │
│  2 │ wrong size │
└────┴────────────┘
```

The `IcebergS3` engine clause names the storage path for the new table, and `write_full_path_in_iceberg_metadata` makes ClickHouse register the full table location with the catalog.

<Note>
  Creating tables through a catalog requires ClickHouse 26.8 or later. Versions 26.4 through 26.7 write the table files before registering the namespace, which SeaweedFS rejects unless the namespace already exists in the catalog; versions before 26.4 appear to succeed, but the table files are written to object storage without being registered in the catalog.
</Note>

When ClickHouse commits an insert, the SeaweedFS catalog repairs metadata the experimental writer does not yet produce: it fills in missing field IDs in manifests, rewrites bucket-relative file paths as absolute locations, and stamps a default name mapping on the table. Strict readers such as PyIceberg and Spark can then read the rows ClickHouse wrote. This requires SeaweedFS 4.42 or later.

<h2 id="loading-data-from-your-data-lake-into-clickhouse">
  Loading data from your Data Lake into ClickHouse
</h2>

If you need to load data from the SeaweedFS catalog into ClickHouse, start by creating a local ClickHouse table:

```sql theme={null}
CREATE TABLE default.orders
(
    `id` Int64,
    `region` String,
    `amount` Float64
)
ENGINE = MergeTree()
ORDER BY (region, id);
```

Then load the data from your SeaweedFS catalog table via an `INSERT INTO SELECT`:

```sql theme={null}
INSERT INTO default.orders
SELECT * FROM lake.`sales.orders`;
```