Migrate to Shared Catalog - ClickHouse Documentation

Shared Catalog is the new approach to managing metadata in ClickHouse Private instances. Instead of databases using the Replicated engine, which uses a DDL queue in ClickHouse Keeper to ensure DDL statements are applied on all replicas, the Shared database engine stores all metadata in Keeper itself, removing the need to sync DDL statements across replicas. Shared Catalog provides several benefits:

Consistent state across all replicas
Statelessness of compute nodes (enables stateless Server nodes)
Atomic CREATE TABLE ... AS SELECT
Support for UNDROP
RENAME/move TABLE between databases
Fast, reliable replica bootstrapping (decreases wake, start and provisioning times)

However, it does require a migration if your instance wasn’t set up with Shared Catalog on the first deployment. We recommend you test the migration first on a test environment before migrating production instances.

What to expect during the migration

The migration is performed on a live service. Queries continue to be served throughout: the Operator replaces the Server pods one at a time, and each Server drains its in-flight queries before exiting. Plan for reduced capacity while the roll is in progress, since one replica is unavailable at a time, and make sure clients reconnect and retry — existing connections are closed when a replica is replaced.

DDL queries will not work for the lifespan of the migration. The Operator stops replicated DDL processing before the first Server pod is replaced, and enables it again only once the migration has completed. SELECT and INSERT statements are unaffected.For instances configured through compute-compute separation this covers the parent and every child, for the whole operation — from the parent’s first Helm upgrade until the last child has completed. See Migrating instances with compute-compute separation.

Migration requirements

Before the migration can begin, verify the following prerequisites:

The queries below use clusterAllReplicas('default', ...), which covers the replicas of a single instance. If your instance is a parent configured through compute-compute separation, substitute all_groups.default for default to cover the parent and all of its children. Checking only default on a parent reports success while children are still unmigrated.

1. At least 2 Server replicas

The Operator performs a rolling upgrade to migrate databases to the new Shared database engine. Set the server.replicas helm value to 2 or greater. For instances configured through compute-compute separation this applies to the parent and to every child.

2. All databases use the Replicated engine

All databases (except system) must use the Replicated database engine. Verify with:

SELECT DISTINCT
    name,
    engine
FROM clusterAllReplicas('default', system.databases)
WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema'))
  AND (engine != 'Replicated')

The query should return no rows.

3. All tables use non-replicated table engines

No tables (except in system) should use replicated or legacy table engines. Verify with:

SELECT
    hostName(),
    name,
    engine,
    database
FROM clusterAllReplicas('default', system.tables)
WHERE ((engine LIKE '%Replicated%') OR (engine IN ('Ordinary', 'Atomic')))
  AND (database != 'system')

The query should return no rows.

4. No detached tables exist

SELECT hostName(), * FROM clusterAllReplicas('default', system.detached_tables)

The query should return no rows.

5. Metadata persistent volumes are still enabled

featureFlags.disableMetadataPersistentVolumes must be false. The migration reads the existing Replicated databases from the metadata volumes of the Server pods, so those volumes have to still be in place — setting this flag to true makes the Servers stateless and removes them. From chart version 1.1.245 onwards this flag defaults to true, so instances migrating from Database Replicated must set it explicitly:

featureFlags:
  disableMetadataPersistentVolumes: false

Keep it false for the whole migration, and also while rolling back. Moving to stateless Servers is a separate change: apply it as its own Helm upgrade once the migration is verified complete. Once all requirements are met the migration can be started. If any requirements are unmet, reach out to ClickHouse support for assistance.

Remove `ON CLUSTER` from DDL queries

Review your applications, migration scripts and automation for ON CLUSTER before you migrate, and remove the clause.

Shared Catalog keeps all metadata in Keeper and every replica reads the same catalog, so a DDL statement already applies to the whole instance and the clause has nothing left to do. Some statements are rejected outright while it is present:

Before (Replicated)
After (Shared Catalog)

ALTER DATABASE analytics ON CLUSTER default MODIFY COMMENT 'Analytics tables'

ALTER DATABASE analytics MODIFY COMMENT 'Analytics tables'

Other statements still run with the clause in place, so a spot check proves little. We advise you to remove ON CLUSTER everywhere rather than only where you have seen it fail.

If query logging is enabled, this query groups the statements that mentioned ON CLUSTER over the last seven days, and shows which users and clients sent them:

SELECT
    normalized_query_hash,
    any(query) AS example,
    groupUniqArray(user) AS users,
    groupUniqArray(client_name) AS clients,
    groupUniqArray(http_user_agent) AS user_agents,
    count() AS queries,
    max(event_time) AS last_seen
FROM clusterAllReplicas('default', system.query_log)
WHERE (type = 'QueryStart')
  AND (is_initial_query = 1)
  AND (event_time > (now() - toIntervalDay(7)))
  AND (query ILIKE '% ON CLUSTER %')
GROUP BY normalized_query_hash
ORDER BY last_seen DESC

It matches on the query text, so read the example column before acting on a row: statements that only mention ON CLUSTER in a comment or a string literal show up too. Results are also bound by how long the log is kept, so treat this as a starting point rather than an inventory. Clients that run DDL occasionally, such as schema migrations, backup tooling and ad hoc admin scripts, still need a manual review. On a parent instance, substitute all_groups.default for default so the audit covers its children as well.

Performing the migration

The steps below are performed on each instance. Choose one:

A single instance with no children — follow steps 1 to 4 in order.
An instance configured through compute-compute separation — the same steps apply to the parent and to every child, but the order across instances matters and the parent does not reach Running until every child has been migrated. Read Migrating instances with compute-compute separation before you start.

1. Enable Shared Catalog and add the migration markers

Set both feature flags in the same Helm upgrade:

featureFlags:
  enableSharedCatalog: true
  migrateToSharedCatalog: true
  disableMetadataPersistentVolumes: false

enableSharedCatalog and migrateToSharedCatalog must be enabled together, in a single upgrade. Enabling enableSharedCatalog on its own leaves ClickHouse expecting a Shared Catalog that does not exist yet, the Servers start in an error state, and the Operator cannot apply the migration. If that happens, follow Rollback and start again.

Leave featureFlags.disableMetadataPersistentVolumes set to false — see requirement 5. Enabling it here drops the metadata volumes the migration needs.

Together these two flags:

Add the shared_database_catalog.migration_from_database_replicated setting to ClickHouse, telling it to expect existing Replicated databases while running in Shared Catalog mode
Add the clickhouse.com/start-shared-catalog-migration: "true" annotation to the ClickHouseCluster Kubernetes resource, notifying the Operator to migrate this instance

Once deployed, the Operator replaces the Server pods one at a time and changes the type of all databases to Shared.

2. Wait for the instance to reach its expected state

kubectl get clickhousecluster -n <namespace> <instance-name>

Single instance
Parent with children

Wait for the Status column to read Running, then continue to step 3.

A parent instance will not reach Running at this point. It stays Degraded until every child has been migrated, which is expected.Do not roll back, and do not continue to step 3. Follow Migrating instances with compute-compute separation instead — it covers the remaining steps for the parent and for every child.

3. Remove the migration markers

Once the status is Running, set featureFlags.migrateToSharedCatalog to false in your Helm values. As all databases have been migrated to type Shared instead of Replicated, the migration settings are no longer relevant.

Do not remove the markers while any instance in the group is still unmigrated. On a parent, removing them early makes the instance report Running while replicated DDL processing stays disabled, and the Operator will not enable it again. Complete every child first.

4. Verify all databases are migrated

Confirm all non-system databases use the Shared engine:

SELECT DISTINCT
    name,
    engine
FROM clusterAllReplicas('default', system.databases)
WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema'))
  AND (engine != 'Shared')

The query should return no rows. At this point your instance is migrated to the Shared Catalog.

Migrating instances with compute-compute separation

An instance and its children share one Keeper ensemble and one catalog, so the migration is a single operation across the whole group rather than one migration per instance. The Operator only treats the migration as complete once every replica of the parent and of every child is running Shared Catalog. Two consequences shape the procedure:

The parent must be migrated first. A child cannot begin until the parent has stopped replicated DDL processing, which happens at the start of the parent’s migration.
The parent reports Degraded from the moment its own replicas are migrated until the last child completes. This is the expected intermediate state, not a failure.

ClickHouse enforces this independently of the Operator: a replica refuses to migrate away from the Replicated engine unless replicated DDL processing has been stopped on every replica in the group, and fails with Replicated DDL queries are enabled on some replicas, cannot perform migration from Database Replicated. Migrating a child before its parent, or only part of the group, therefore fails server-side as well. Work through the group in this order.

1. Migrate the parent

Apply Enable Shared Catalog and add the migration markers to the parent. Do not remove the markers. The parent will move to Degraded with the reason failed to sync tables and databases:

kubectl get clickhousecluster -n <namespace> <parent-name> \
  -o jsonpath='{.status.state}{"\t"}{.status.reason}{"\n"}'
# Degraded	failed to sync tables and databases

Any other reason means something else is wrong. Stop and contact ClickHouse support rather than continuing.

Confirm the parent’s own replicas have migrated before moving on. Connected to the parent, every row must report Shared:

SELECT hostName(), engine
FROM clusterAllReplicas('default', system.databases)
WHERE name = 'default'
ORDER BY hostName()

2. Migrate every child

Apply Enable Shared Catalog and add the migration markers to each child instance, one at a time. Do not remove the markers. Each child also moves to Degraded while it waits for the group to converge. Confirm its replicas have migrated with the same query, connected to the child.

3. Wait for the group to converge

Once the last child has migrated, the parent enables replicated DDL processing again and moves to Running. The children follow.

kubectl get clickhouseclusters -A

Wait until the parent and every child report Running. Every replica in the group should now report Shared:

SELECT hostName(), engine
FROM clusterAllReplicas('all_groups.default', system.databases)
WHERE name = 'default'
ORDER BY hostName()

4. Remove the migration markers

Only now, set featureFlags.migrateToSharedCatalog to false on the parent and on every child, then run Verify all databases are migrated against all_groups.default to confirm the whole group is migrated.

Enable stateless Servers and reclaim metadata PVCs

Once the migration is complete, you can make the Server nodes stateless by setting featureFlags.disableMetadataPersistentVolumes to true in your Helm values. Server pods will start without persistent metadata volumes and rely on the Shared Catalog and DatabaseDisk for metadata.

Applying disableMetadataPersistentVolumes: true on an existing cluster does not remove existing PersistentVolumeClaims. A StatefulSet’s spec.volumeClaimTemplates is immutable, so the Operator cannot reshape the running StatefulSet in place. Without the steps below, the PVCs remain bound and the Server Pods keep mounting them.

1. Apply the flag

Set the flag in your Helm values and upgrade the release:

featureFlags:
  enableSharedCatalog: true
  disableMetadataPersistentVolumes: true

At this point the existing Server StatefulSets still carry volumeClaimTemplates. The Operator logs an immutable-field warning and does not rewrite them.

2. Recreate each Server StatefulSet, one replica at a time

Each Server replica runs in its own StatefulSet (one Pod per StatefulSet). For each replica, delete the StatefulSet with --cascade=orphan so the running Pod keeps serving traffic while the Operator recreates the StatefulSet without volumeClaimTemplates. Once the new StatefulSet exists, its rolling update replaces the Pod so it starts without the metadata PVC. List the Server StatefulSets for the cluster:

kubectl get statefulsets \
  -l app.kubernetes.io/name=clickhouse-server,app=<instance-name>-server

For each StatefulSet, delete it while keeping the Pod running:

kubectl delete statefulset <sts-name> --cascade=orphan

Wait for the Operator to recreate the StatefulSet and for its Pod to become Ready before moving to the next replica:

kubectl rollout status statefulset/<sts-name>

Confirm the recreated StatefulSet has no volumeClaimTemplates:

kubectl get statefulset <sts-name> \
  -o jsonpath='{.spec.volumeClaimTemplates}{"\n"}'
# []

Confirm the running Pod no longer references a metadata PVC:

kubectl get pod <pod-name> \
  -o jsonpath='{range .spec.volumes[?(@.persistentVolumeClaim)]}{.persistentVolumeClaim.claimName}{"\n"}{end}'
# (empty)

Repeat for every Server replica.

3. Delete the orphaned PVCs

Once every Server Pod is Ready without a PVC, delete the leftover PersistentVolumeClaims:

# List the orphaned metadata PVCs.
kubectl get pvc -l app=<instance-name>-server

# Delete them.
kubectl delete pvc -l app=<instance-name>-server

Rollback

If any issues are encountered with the Shared Catalog, you can roll back to the previous state:

For instances configured through compute-compute separation, the catalog in Keeper is shared by the parent and all of its children. Roll back every child before the parent, and only remove the catalog once every instance in the group has reverted to the Replicated engine. Removing it while any instance is still running Shared Catalog leaves that instance without a catalog.

1. Remove the migration markers

Set featureFlags.migrateToSharedCatalog to false in your Helm values.

2. Remove the Shared Catalog settings

Set featureFlags.enableSharedCatalog to false in your Helm values. This will trigger the Operator to revert the changes.

3. Remove the Catalog in Keeper

Port-forward a connection to Keeper and start a keeper-client session (using clickhouse keeper-client).

This is destructive and affects every instance sharing this Keeper ensemble. Confirm that steps 1 and 2 have been applied to the parent and to every child, and that no instance still reports the Shared engine, before running rmr.

Once the session is started, execute the following commands:

ls '/clickhouse/catalog'
# Output: migration_from_database_replicated_completed names references replicas uuids

rmr '/clickhouse/catalog'
# You are going to recursively delete path /clickhouse/catalog Continue?
# [y/n] y

This will clean up all Shared Catalog state from Keeper.

4. Verify rollback

Confirm all non-system databases are using the Replicated engine again:

SELECT DISTINCT
    name,
    engine
FROM clusterAllReplicas('default', system.databases)
WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema'))
  AND (engine != 'Replicated')

The query should return no rows.

​What to expect during the migration

​Migration requirements

​1. At least 2 Server replicas

​2. All databases use the Replicated engine

​3. All tables use non-replicated table engines

​4. No detached tables exist

​5. Metadata persistent volumes are still enabled

​Remove ON CLUSTER from DDL queries

​Performing the migration

​1. Enable Shared Catalog and add the migration markers

​2. Wait for the instance to reach its expected state

​3. Remove the migration markers

​4. Verify all databases are migrated

​Migrating instances with compute-compute separation

​1. Migrate the parent

​2. Migrate every child

​3. Wait for the group to converge

​4. Remove the migration markers

​Enable stateless Servers and reclaim metadata PVCs

​1. Apply the flag

​2. Recreate each Server StatefulSet, one replica at a time

​3. Delete the orphaned PVCs

​Rollback

​1. Remove the migration markers

​2. Remove the Shared Catalog settings

​3. Remove the Catalog in Keeper

​4. Verify rollback

What to expect during the migration

Migration requirements

1. At least 2 Server replicas

2. All databases use the Replicated engine

3. All tables use non-replicated table engines

4. No detached tables exist

5. Metadata persistent volumes are still enabled

Remove `ON CLUSTER` from DDL queries

Performing the migration

1. Enable Shared Catalog and add the migration markers

2. Wait for the instance to reach its expected state

3. Remove the migration markers

4. Verify all databases are migrated

Migrating instances with compute-compute separation

1. Migrate the parent

2. Migrate every child

3. Wait for the group to converge

4. Remove the migration markers

Enable stateless Servers and reclaim metadata PVCs

1. Apply the flag

2. Recreate each Server StatefulSet, one replica at a time

3. Delete the orphaned PVCs

Rollback

1. Remove the migration markers

2. Remove the Shared Catalog settings

3. Remove the Catalog in Keeper

4. Verify rollback