Upgrade ClickHouse Private

Best practice: Always run the upgrade on a staging or test cluster first before applying it to production. A dedicated test cluster that mirrors production is strongly recommended as part of your standard upgrade process.

​

Pre-Upgrade Checklist

1. Verify the cluster is healthy — confirm all server and keeper pods are running and ready, and the ClickhouseCluster CR status shows no errors:
   kubectl get clickhousecluster -A
   
   Do not proceed if any pods are crashlooping or the cluster is degraded.
2. (Highly recommended for production) Take a full backup of your cluster — see Back Up and Restore ClickHouse. Verify the backup completes successfully (status = BACKUP_DONE in system.backups) before proceeding.
3. Ensure the previous version is available for rollback — verify that the previous artifact versions are still accessible in your internal ECR. Having the images and charts present is sufficient to roll back quickly if needed.
4. Copy all images for the new version to your internal ECR. Use the Component Versions reference for the target release.
5. Run preflight checks. Preflight checks validate additional requirements and checks for deprecated feature flags. For more details see the How To: Run Preflight Checks page.

​

Upgrade Procedure

​

1. Upgrade the Operator Helm Chart

helm upgrade clickhouse-operator oci://$YOUR_ECR/helm/clickhouse-operator-helm \
  --version <new-version> \
  -f values.yaml \
  -n clickhouse-operator-system

Note: Upgrading the operator may trigger a rolling restart of all ClickHouse clusters it manages. If you have multiple clusters and want to validate the upgrade on one before allowing the operator to reconcile others, see Protecting Clusters During Upgrade below.

​

2. Validate the Upgraded Operator

kubectl rollout status deployment -n clickhouse-operator-system
kubectl get deployments clickhouse-operator-clickhouse-operator-helm -n clickhouse-operator-system
kubectl logs -n clickhouse-operator-system deployment/clickhouse-operator-clickhouse-operator-helm

kubectl get clickhousecluster -A

​

3. Upgrade the onprem-clickhouse-cluster Helm Chart

helm upgrade <cluster-release-name> oci://$YOUR_ECR/helm/onprem-clickhouse-cluster \
  --version <new-chart-version> \
  -n <namespace> \
  -f values.yaml \
  --set server.version=<new-server-version> \
  --set keeper.version=<new-keeper-version>

​

4. Validate the Upgraded Cluster

kubectl get clickhousecluster -n <namespace>
kubectl get pods -n <namespace>

​

Protecting Clusters During Upgrade

kubectl patch clickhousecluster c-default-xx-01 -n ns-default-xx-01 \
  --type='merge' \
  -p '{"metadata": {"annotations": {"clickhouse.com/skip-reconcile": "true"}}}'

kubectl patch clickhousecluster c-default-xx-01 -n ns-default-xx-01 \
  --type='json' \
  -p='[{"op": "remove", "path": "/metadata/annotations/clickhouse.com~1skip-reconcile"}]'

Skip ClickhouseCluster reconcile req ... because it has clickhouse.com/skip-reconcile annotation

​

Rollback

• Operator rollback: redeploy the previous operator helm chart version.
• Cluster rollback: redeploy the previous onprem-clickhouse-cluster helm chart version with the previous server and keeper image versions.
• Data recovery: if data issues are suspected, restore from the backup taken in the pre-upgrade checklist — see Back Up and Restore ClickHouse.