Upgrade ClickHouse Private

Best practice: Always run the upgrade on a staging or test cluster first before applying it to production. A dedicated test cluster that mirrors production is strongly recommended as part of your standard upgrade process.

​

Pre-Upgrade Checklist

1. Verify the cluster is healthy — confirm all server and keeper pods are running and ready, and the ClickhouseCluster CR status shows no errors:
   kubectl get clickhousecluster -A
   
   Do not proceed if any pods are crashlooping or the cluster is degraded.
2. (Highly recommended for production) Take a full backup of your cluster — see Back Up and Restore ClickHouse. Verify the backup completes successfully (status = BACKUP_DONE in system.backups) before proceeding.
3. Ensure the previous version is available for rollback — verify that the previous artifact versions are still accessible in your internal ECR. Having the images and charts present is sufficient to roll back quickly if needed.
4. Copy all images for the new version to your internal ECR. Use the Component Versions reference for the target release.
5. Run preflight checks. Preflight checks validate additional requirements and checks for deprecated feature flags. For more details see the How To: Run Preflight Checks page.

​

Upgrade Procedure

​

1. Upgrade the Operator Helm Chart

helm upgrade clickhouse-operator oci://$YOUR_ECR/helm/clickhouse-operator-helm \
  --version <new-version> \
  -f values.yaml \
  -n clickhouse-operator-system

Note: Upgrading the operator may trigger a rolling restart of all ClickHouse clusters it manages. If you have multiple clusters and want to validate the upgrade on one before allowing the operator to reconcile others, see Protecting Clusters During Upgrade below.

​

2. Validate the Upgraded Operator

kubectl rollout status deployment -n clickhouse-operator-system
kubectl get deployments clickhouse-operator-clickhouse-operator-helm -n clickhouse-operator-system
kubectl logs -n clickhouse-operator-system deployment/clickhouse-operator-clickhouse-operator-helm

kubectl get clickhousecluster -A

​

3. Upgrade the onprem-clickhouse-cluster Helm Chart

helm upgrade <cluster-release-name> oci://$YOUR_ECR/helm/onprem-clickhouse-cluster \
  --version <new-chart-version> \
  -n <namespace> \
  -f values.yaml

Note: If your values.yaml pins server.image.tag or keeper.image.tag, those values override the versions the chart ships with and the upgrade will not move ClickHouse. Remove them unless you intentionally need a specific version. FIPS deployments are the exception — they select the -fips variant by tag and must keep the tags set, updated to the new release.

​

4. Validate the Upgraded Cluster

kubectl get clickhousecluster -n <namespace>
kubectl get pods -n <namespace>

​

Protecting Clusters During Upgrade

kubectl patch clickhousecluster c-default-xx-01 -n ns-default-xx-01 \
  --type='merge' \
  -p '{"metadata": {"annotations": {"clickhouse.com/skip-reconcile": "true"}}}'

kubectl patch clickhousecluster c-default-xx-01 -n ns-default-xx-01 \
  --type='json' \
  -p='[{"op": "remove", "path": "/metadata/annotations/clickhouse.com~1skip-reconcile"}]'

Skip ClickhouseCluster reconcile req ... because it has clickhouse.com/skip-reconcile annotation

​

Rollback

• Operator rollback: redeploy the previous operator helm chart version.
• Cluster rollback: redeploy the previous onprem-clickhouse-cluster helm chart version, which restores the server and keeper versions that shipped with it.
• Data recovery: if data issues are suspected, restore from the backup taken in the pre-upgrade checklist — see Back Up and Restore ClickHouse.