Migrate to Shared Catalog

Shared Catalog is the new approach to managing metadata in ClickHouse Private instances. Instead of databases using the Replicated engine, which uses a DDL queue in ClickHouse Keeper to ensure DDL statements are applied on all replicas, the Shared database engine stores all metadata in Keeper itself, removing the need to sync DDL statements across replicas. Shared Catalog provides several benefits:

Consistent state across all replicas
Statelessness of compute nodes (enables stateless Server nodes)
Atomic CREATE TABLE ... AS SELECT
Support for UNDROP
RENAME/move TABLE between databases
Fast, reliable replica bootstrapping (decreases wake, start and provisioning times)

However, it does require a migration if your instance wasn’t set up with Shared Catalog on the first deployment. We recommend you test the migration first on a test environment before migrating production instances.

Migration requirements

Before the migration can begin, verify the following prerequisites:

1. At least 2 Server replicas

The Operator performs a rolling upgrade to migrate databases to the new Shared database engine. Set the server.replicas helm value to 2 or greater.

2. All databases use the Replicated engine

All databases (except system) must use the Replicated database engine. Verify with:

SELECT DISTINCT
    name,
    engine
FROM clusterAllReplicas('default', system.databases)
WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema'))
  AND (engine != 'Replicated')

The query should return no rows.

3. All tables use non-replicated table engines

No tables (except in system) should use replicated or legacy table engines. Verify with:

SELECT
    name,
    engine,
    database
FROM system.tables
WHERE ((engine LIKE '%Replicated%') OR (engine IN ('Ordinary', 'Atomic')))
  AND (database != 'system')

The query should return no rows.

4. No detached tables exist

SELECT * FROM system.detached_tables

The query should return no rows.

5. Metadata persistent volumes are still enabled

featureFlags.disableMetadataPersistentVolumes must be false. The migration reads the existing Replicated databases from the metadata volumes of the Server pods, so those volumes have to still be in place — setting this flag to true makes the Servers stateless and removes them. From chart version 1.1.245 onwards this flag defaults to true, so instances migrating from Database Replicated must set it explicitly:

featureFlags:
  disableMetadataPersistentVolumes: false

Keep it false for the whole migration, and also while rolling back. Moving to stateless Servers is a separate change: apply it as its own Helm upgrade once the migration is verified complete. Once all requirements are met the migration can be started. If any requirements are unmet, reach out to ClickHouse support for assistance.

Performing the migration

For child instances configured through compute-compute separation, migrate the parent instance first before migrating any children. Repeat the steps below for each child instance after the parent is migrated.

1. Enable Shared Catalog configuration

If you haven’t migrated to the shared catalog you need to enable both featureFlags.enableSharedCatalog & featureFlags.migrateToSharedCatalog simultaneously. Otherwise ClickHouse starts in an error state and the operator will be unable to apply the migration. If that occurs you can follow the steps on reverting the migration below and start again.

Set featureFlags.enableSharedCatalog to true in your Helm values. This enables the configuration, but doesn’t trigger the migration of databases. Deploying a ClickHouse Cluster with this flag set to true but not migrating existing replicated databases will cause the ClickHouse Cluster to fail to start.

Leave featureFlags.disableMetadataPersistentVolumes set to false in the same Helm values — see requirement 5. Enabling it here drops the metadata volumes the migration needs.

2. Add the migration markers

Set featureFlags.migrateToSharedCatalog to true in your Helm values. This:

Adds the shared_database_catalog.migration_from_database_replicated setting to ClickHouse, telling it to expect existing Replicated databases while running in Shared Catalog mode
Adds the clickhouse.com/start-shared-catalog-migration: "true" annotation to the ClickHouseCluster Kubernetes resource, notifying the Operator to migrate this cluster

Once deployed, the Operator will trigger a rolling upgrade of the Server nodes. This will also change the types of all databases to Shared.

3. Wait for the status to be Ready

kubectl get ClickHouseCluster <instance-name>

4. Remove the migration markers

Once the status is Ready, set featureFlags.migrateToSharedCatalog to false in your Helm values. As all databases have been migrated to type Shared instead of Replicated, the migration settings are no longer relevant.

5. Verify all databases are migrated

Confirm all non-system databases use the Shared engine:

SELECT DISTINCT
    name,
    engine
FROM clusterAllReplicas('default', system.databases)
WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema'))
  AND (engine != 'Shared')

The query should return no rows. At this point your instance is migrated to the Shared Catalog.

Enable stateless Servers and reclaim metadata PVCs

Once the migration is complete, you can make the Server nodes stateless by setting featureFlags.disableMetadataPersistentVolumes to true in your Helm values. Server pods will start without persistent metadata volumes and rely on the Shared Catalog and DatabaseDisk for metadata.

Applying disableMetadataPersistentVolumes: true on an existing cluster does not remove existing PersistentVolumeClaims. A StatefulSet’s spec.volumeClaimTemplates is immutable, so the Operator cannot reshape the running StatefulSet in place. Without the steps below, the PVCs remain bound and the Server Pods keep mounting them.

1. Apply the flag

Set the flag in your Helm values and upgrade the release:

featureFlags:
  enableSharedCatalog: true
  disableMetadataPersistentVolumes: true

At this point the existing Server StatefulSets still carry volumeClaimTemplates. The Operator logs an immutable-field warning and does not rewrite them.

2. Recreate each Server StatefulSet, one replica at a time

Each Server replica runs in its own StatefulSet (one Pod per StatefulSet). For each replica, delete the StatefulSet with --cascade=orphan so the running Pod keeps serving traffic while the Operator recreates the StatefulSet without volumeClaimTemplates. Once the new StatefulSet exists, its rolling update replaces the Pod so it starts without the metadata PVC. List the Server StatefulSets for the cluster:

kubectl get statefulsets \
  -l app.kubernetes.io/name=clickhouse-server,app=<instance-name>-server

For each StatefulSet, delete it while keeping the Pod running:

kubectl delete statefulset <sts-name> --cascade=orphan

Wait for the Operator to recreate the StatefulSet and for its Pod to become Ready before moving to the next replica:

kubectl rollout status statefulset/<sts-name>

Confirm the recreated StatefulSet has no volumeClaimTemplates:

kubectl get statefulset <sts-name> \
  -o jsonpath='{.spec.volumeClaimTemplates}{"\n"}'
# []

Confirm the running Pod no longer references a metadata PVC:

kubectl get pod <pod-name> \
  -o jsonpath='{range .spec.volumes[?(@.persistentVolumeClaim)]}{.persistentVolumeClaim.claimName}{"\n"}{end}'
# (empty)

Repeat for every Server replica.

3. Delete the orphaned PVCs

Once every Server Pod is Ready without a PVC, delete the leftover PersistentVolumeClaims:

# List the orphaned metadata PVCs.
kubectl get pvc -l app=<instance-name>-server

# Delete them.
kubectl delete pvc -l app=<instance-name>-server

Rollback

If any issues are encountered with the Shared Catalog, you can roll back to the previous state:

1. Remove the migration markers

Set featureFlags.migrateToSharedCatalog to false in your Helm values.

2. Remove the Shared Catalog settings

Set featureFlags.enableSharedCatalog to false in your Helm values. This will trigger the Operator to revert the changes.

3. Remove the Catalog in Keeper

Port-forward a connection to Keeper and start a keeper-client session (using clickhouse keeper-client). Once the session is started, execute the following commands:

ls '/clickhouse/catalog'
# Output: migration_from_database_replicated_completed names references replicas uuids

rmr '/clickhouse/catalog'
# You are going to recursively delete path /clickhouse/catalog Continue?
# [y/n] y

This will clean up all Shared Catalog state from Keeper.

4. Verify rollback

Confirm all non-system databases are using the Replicated engine again:

SELECT DISTINCT
    name,
    engine
FROM clusterAllReplicas('default', system.databases)
WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema'))
  AND (engine != 'Replicated')

The query should return no rows.

ClickHouse Cloud

Bring Your Own Cloud

ClickHouse Private

Migration requirements

1. At least 2 Server replicas

2. All databases use the Replicated engine

3. All tables use non-replicated table engines

4. No detached tables exist

5. Metadata persistent volumes are still enabled

Performing the migration

1. Enable Shared Catalog configuration

2. Add the migration markers

3. Wait for the status to be Ready

4. Remove the migration markers

5. Verify all databases are migrated

Enable stateless Servers and reclaim metadata PVCs

1. Apply the flag

2. Recreate each Server StatefulSet, one replica at a time

3. Delete the orphaned PVCs

Rollback

1. Remove the migration markers

2. Remove the Shared Catalog settings

3. Remove the Catalog in Keeper

4. Verify rollback

​Migration requirements

​1. At least 2 Server replicas

​2. All databases use the Replicated engine

​3. All tables use non-replicated table engines

​4. No detached tables exist

​5. Metadata persistent volumes are still enabled

​Performing the migration

​1. Enable Shared Catalog configuration

​2. Add the migration markers

​3. Wait for the status to be Ready

​4. Remove the migration markers

​5. Verify all databases are migrated

​Enable stateless Servers and reclaim metadata PVCs

​1. Apply the flag

​2. Recreate each Server StatefulSet, one replica at a time

​3. Delete the orphaned PVCs

​Rollback

​1. Remove the migration markers

​2. Remove the Shared Catalog settings

​3. Remove the Catalog in Keeper

​4. Verify rollback

Migration requirements

1. At least 2 Server replicas

2. All databases use the Replicated engine

3. All tables use non-replicated table engines

4. No detached tables exist

5. Metadata persistent volumes are still enabled

Performing the migration

1. Enable Shared Catalog configuration

2. Add the migration markers

3. Wait for the status to be Ready

4. Remove the migration markers

5. Verify all databases are migrated

Enable stateless Servers and reclaim metadata PVCs

1. Apply the flag

2. Recreate each Server StatefulSet, one replica at a time

3. Delete the orphaned PVCs

Rollback

1. Remove the migration markers

2. Remove the Shared Catalog settings

3. Remove the Catalog in Keeper

4. Verify rollback