Troubleshooting Guide

​

Operator Issues

​

CR changes not being applied

1. The operator itself is crashlooping.
2. ClickHouse pods (keeper or server) are crashlooping, which blocks reconciliation.
3. A clickhouse.com/skip-reconcile annotation is present on the CR, which instructs the operator to skip reconciliation entirely.

1. Check operator logs for crashes or errors:
   kubectl logs -f -n clickhouse-operator-system <operator-pod-name>
   
2. Check keeper and server pods for crashloops:
   kubectl get pods -n <namespace>
   kubectl logs <pod-name> -n <namespace>
   
3. Check for the skip-reconcile annotation on the CR:
   kubectl get clickhousecluster <cluster-name> -n <namespace> -o yaml | grep skip-reconcile
   
   If present and no longer needed, remove it:
   kubectl annotate clickhousecluster <cluster-name> -n <namespace> clickhouse.com/skip-reconcile-
   

​

How to drop a server replica

1. Add the skip-reconcile annotation to prevent the operator from interfering:
   kubectl annotate clickhousecluster <cluster-name> -n <namespace> clickhouse.com/skip-reconcile=remove-replica
   
   Confirm the operator has noticed by checking logs for:
   Skip ClickhouseCluster reconcile req ... because it has clickhouse.com/skip-reconcile annotation"
   
2. Remove the replica from the ReplicaStateMap. First, inspect the current map:
   kubectl get clickhousecluster <cluster-name> -n <namespace> -o json | jq .status.replicaStateMap
   
   Then edit the status subresource to remove the target replica entry:
   EDITOR=vim kubectl edit clickhousecluster <cluster-name> -n <namespace> --subresource=status
   
3. Delete the StatefulSet for the replica:
   kubectl delete sts <replica-statefulset-name> -n <namespace>
   
   Wait for the pod to terminate after deleting the StatefulSet.
4. Remove the skip-reconcile annotation:
   kubectl annotate clickhousecluster <cluster-name> -n <namespace> clickhouse.com/skip-reconcile-
   
   The operator will launch a new replica and clean up the removed replica from ClickHouse.
5. Verify by logging in to the ClickHouse cluster and confirming the old replica has been removed. If any leases the replica holds have not expired, the operator will retry removal. Cleanup should complete within 5 minutes.

​

Server pod hanging on termination

NAME                            READY   STATUS        RESTARTS   AGE
c-navy-wl-64-server-gbwmanx-0   1/2     Terminating   0          3m25s
c-navy-wl-64-server-pxb9mde-0   1/2     Terminating   0          3m25s

1. Check the PreStop hook log for details on what the hook is waiting for:
   kubectl exec <pod-name> -- cat /var/log/clickhouse-server/prestop.log
   
2. Check server logs for any received signal log messages that indicate the shutdown process state.

​

Multiple ClickHouseClusters in the same namespace

1. Check how many ClickHouseCluster resources exist in the namespace:
   kubectl get clickhouseclusters -n <namespace>
   
2. If more than one exists, move the additional clusters to their own namespaces. Each ClickHouseCluster must be the only instance in its namespace.
3. The onprem-clickhouse-cluster Helm chart creates a ResourceQuota by default that prevents this situation. Verify the quota is in place:
   kubectl get resourcequota -n <namespace>
   
   If the quota is missing, ensure resourceQuota.enabled is set to true (the default) in your Helm values.

​

CR not in healthy Running state

1. Check cluster status across all namespaces:
   kubectl get clickhouseclusters -A
   
2. Describe the problem pod for detailed status and events:
   kubectl describe pod <pod-name> -n <namespace>
   
3. Check logs from the previously terminated container:
   kubectl logs -p <pod-name> -n <namespace>
   
4. Check namespace events for scheduling or resource issues:
   kubectl get events -n <namespace>
   

​

ClickHouse Server Issues

​

Crashlooping server pods

1. Check the ClickHouse server pod logs for the crash reason:
   kubectl logs <pod-name> -n <namespace>
   kubectl logs -p <pod-name> -n <namespace>
   
2. If the crash is due to memory pressure or Kubernetes-initiated termination, check Kubernetes events:
   kubectl describe pod <pod-name> -n <namespace>
   kubectl get events -n <namespace>
   

​

Data loss/corruption (ClickHouseBrokenPartDetectedOnSelect)

1. Examine logs for the POTENTIALLY_BROKEN_DATA_PART exception. If not found in logs, also check system.errors.
2. Understand what data parts are lost. Query system.replicas to find affected tables:
   SELECT
       database,
       table,
       lost_part_count AS value
   FROM system.replicas
   WHERE value > 0
   
3. Find logs related to lost parts. Check /var/log/clickhouse-server/ on the pod (use zgrep for archived logs), or query system.text_log:
   SELECT
       hostName(),
       event_time,
       logger_name,
       message
   FROM clusterAllReplicas(default, system.text_log)
   WHERE message_format_string = 'Part {} is lost forever.'
   ORDER BY hostName(), event_time
   
   Add a predicate for event_time range to speed up the query.
4. Understand the history of lost parts. Pick a lost part name and find all related logs:
   SELECT
       event_time,
       message
   FROM system.text_log
   WHERE message LIKE '%<part name>%' AND hostName() = '<host where lost forever log was created>'
   ORDER BY event_time ASC
   
   Focus on log messages before Part * is lost forever. Messages after that point are irrelevant (any “found” part is actually an empty replacement).
5. Check for false positives:
   • Check if the table has TTL and the lost part should have been dropped anyway.
   • Check system.query_log for TRUNCATE or DROP PARTITION queries that should have dropped the lost parts.
6. Additional investigation:
   • If the part was detached as broken, determine why it was broken.
   • If you see The specified key does not exist, search all logs with the blob name to find when and why it was removed. Also check log messages about zero-copy locks.
7. Contact ClickHouse support with your findings.

​

Table replicas read-only (ClickHouseTableReplicasReadOnly)

1. Check if there are still read-only tables in the cluster:
   SELECT
     dateDiff('second', readonly_start_time, now()) AS readonly_duration_seconds,
     database,
     table,
     hostname()
   FROM clusterAllReplicas(default, system.replicas)
   WHERE is_readonly = 1
   
2. Investigate the cause. Having read-only tables typically indicates that StorageSharedMergeTree::shutdown was run but the storage object was kept alive. Search text logs using the table name as the logger name.
3. Try restarting the replica for each affected table:
   SYSTEM RESTART REPLICA <database>.<table>
   
   You can get the table names from the query in step 1. Sometimes the problem is trivial and a restart resolves it.
4. If the issue persists, check keeper logs for potential connectivity or coordination issues.

​

Replica already exists (ClickHouseReplicaAlreadyExists)

​

Cannot write to file descriptor (ClickHouseCannotWriteToFileDescriptor)

1. Check for the known partial_merge issue first. There is a known bug in tracking cache disk usage when the join_algorithm = 'partial_merge' query setting is specified. Check if this setting is in use.
2. Connect to the pod:
   kubectl exec -n <namespace> -it <pod> -- /bin/bash
   
3. Check actual disk size:
   df -h
   
4. Check required cache disk size in ClickHouse:
   SELECT path, max_size FROM system.filesystem_cache_settings
   
   Note that multiple caches (e.g., s3diskWithCache, diskPlainRewritableForSystemTablesWithCache) may share the same path (/mnt/clickhouse-cache/sharedS3DiskCache).
5. Compare actual vs. required:
   • If the actual disk size is smaller than required, the issue is a misconfiguration. Contact ClickHouse support.
   • If the disk size is sufficient, it is likely a bug in cache disk usage tracking. Investigate via system.filesystem_cache.

​

Storage and Infrastructure Issues

​

Zone details could not be found for any PV

1. Add allowedTopologies to your StorageClass to ensure volumes are created with the correct node affinities:
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: local-nvme-sc
   allowVolumeExpansion: true
   parameters:
     path: /nvme/disk
   provisioner: rancher.io/local-path
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   allowedTopologies:
     - matchLabelExpressions:
         - key: topology.kubernetes.io/zone
           values:
             - <your_cluster_name>-keeper
             - <your_host_1>
             - <your_host_2>
             - <your_host_3>
         - key: directpv.min.io/zone
           values:
             - default
   
2. If non-standard labels are used for topology, configure the operator’s additionalZoneLabelRegexes property. For example, when using the Helm chart, set the operator.additionalZoneLabelRegexes Helm value to a regex matching your labels (e.g., directpv.*zone).

​

NVMe disk questions

s3diskWithCache:
    type: cache
    disk: s3disk
    path: /mnt/clickhouse-cache/sharedS3DiskCache
    max_size:
    '@from_env': CONFIG_DISK_CACHE_SIZE
    cache_on_write_operations: 1

1. Ensure the disks are attached to the instances used for ClickHouse server Kubernetes pods.
2. Alter the launch template to reflect changes in disk architecture. You can still create a RAID disk and format as ext4 or xfs, but use the appropriate tools (e.g., lsblk) to list devices.
3. If the disks are not mounted at /nvme/disk, set the hostPathBaseDirectory in the ClickHouseCluster Helm chart to the actual mount point.

​

Configuration Issues

​

loadBalancerType field behavior

​

Parent instance stuck in Degraded during Shared Catalog migration

kubectl get clickhousecluster -n <namespace> <parent-name> \
  -o jsonpath='{.status.state}{"\t"}{.status.reason}{"\n"}'
# Degraded	failed to sync tables and databases

1. Check which instances in the group are still on the Replicated engine:
   SELECT hostName(), engine
   FROM clusterAllReplicas('all_groups.default', system.databases)
   WHERE name = 'default'
   ORDER BY hostName()
   
2. Migrate every child that has not been migrated, following Migrating instances with compute-compute separation. The parent moves to Running once the last child completes.
3. Do not remove featureFlags.migrateToSharedCatalog from the parent while any child is unmigrated. If it has already been removed, restore it to true so the Operator can finish the migration.

​

DDL with ON CLUSTER fails on Shared Catalog

Code: 80. DB::Exception: There was an error on [<replica>:9000]: Code: 80.
DB::Exception: It's not initial query. ON CLUSTER is not allowed for Shared Catalog.
(INCORRECT_QUERY)

-- Before
ALTER DATABASE analytics ON CLUSTER default MODIFY COMMENT 'Analytics tables'

-- After
ALTER DATABASE analytics MODIFY COMMENT 'Analytics tables'

​

Diagnostic Commands Quick Reference

​

Cluster status

kubectl get clickhouseclusters -A

​

Operator logs

kubectl logs -f -n clickhouse-operator-system <operator-pod-name>

​

Pod events and details

kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace>

​

Previous container logs (after a crash)

kubectl logs -p <pod-name> -n <namespace>

​

Replica state map

kubectl get clickhousecluster <cluster-name> -n <namespace> -o json | jq .status.replicaStateMap

​

Replication queue size per table (SQL)

SELECT
    concat(database, '.', table),
    count()
FROM system.replication_queue
GROUP BY
    database,
    table

SELECT
    concat(database, '.', table),
    queue_size
FROM system.replicas

​

Replication queue oldest entry per table (SQL)

SELECT
    concat(database, '.', table),
    min(create_time)
FROM system.replication_queue
GROUP BY
    database,
    table