Troubleshooting Guide - ClickHouse Documentation

This guide consolidates common issues, their symptoms, likely causes, and resolution steps for ClickHouse Private deployments. Each entry follows a symptom, cause, resolution structure.

Operator Issues

CR changes not being applied

Symptom: Changes made to the ClickHouseCluster CR are not being reconciled or applied to the cluster for an extended period. Cause: One of three common reasons:

The operator itself is crashlooping.
ClickHouse pods (keeper or server) are crashlooping, which blocks reconciliation.
A clickhouse.com/skip-reconcile annotation is present on the CR, which instructs the operator to skip reconciliation entirely.

Resolution:

Check operator logs for crashes or errors:

kubectl logs -f -n clickhouse-operator-system <operator-pod-name>

Check keeper and server pods for crashloops:

kubectl get pods -n <namespace>
kubectl logs <pod-name> -n <namespace>

Check for the skip-reconcile annotation on the CR:

kubectl get clickhousecluster <cluster-name> -n <namespace> -o yaml | grep skip-reconcile

If present and no longer needed, remove it:

kubectl annotate clickhousecluster <cluster-name> -n <namespace> clickhouse.com/skip-reconcile-

How to drop a server replica

Symptom: You need to remove a specific replica from the cluster without scaling in (reducing the desired replica count). Cause: A replica is unhealthy, stuck, or otherwise needs to be manually removed. Resolution:

Add the skip-reconcile annotation to prevent the operator from interfering:

kubectl annotate clickhousecluster <cluster-name> -n <namespace> clickhouse.com/skip-reconcile=remove-replica

Confirm the operator has noticed by checking logs for:

Skip ClickhouseCluster reconcile req ... because it has clickhouse.com/skip-reconcile annotation"

Remove the replica from the ReplicaStateMap. First, inspect the current map:

kubectl get clickhousecluster <cluster-name> -n <namespace> -o json | jq .status.replicaStateMap

Then edit the status subresource to remove the target replica entry:

EDITOR=vim kubectl edit clickhousecluster <cluster-name> -n <namespace> --subresource=status

Delete the StatefulSet for the replica:
kubectl delete sts <replica-statefulset-name> -n <namespace>
Wait for the pod to terminate after deleting the StatefulSet.
Remove the skip-reconcile annotation:
kubectl annotate clickhousecluster <cluster-name> -n <namespace> clickhouse.com/skip-reconcile-
The operator will launch a new replica and clean up the removed replica from ClickHouse.
Verify by logging in to the ClickHouse cluster and confirming the old replica has been removed. If any leases the replica holds have not expired, the operator will retry removal. Cleanup should complete within 5 minutes.

Server pod hanging on termination

Symptom: Server pods remain in Terminating status for an extended period:

NAME                            READY   STATUS        RESTARTS   AGE
c-navy-wl-64-server-gbwmanx-0   1/2     Terminating   0          3m25s
c-navy-wl-64-server-pxb9mde-0   1/2     Terminating   0          3m25s

Cause: Before terminating, server pods run a PreStop hook that drains connections and waits for running requests to complete. If long-running queries or connections are active, the pod will remain in a terminating state. Resolution:

Check the PreStop hook log for details on what the hook is waiting for:

kubectl exec <pod-name> -- cat /var/log/clickhouse-server/prestop.log

Check server logs for any received signal log messages that indicate the shutdown process state.

Multiple ClickHouseClusters in the same namespace

Symptom: Unexpected reconciliation behavior, resources being modified or deleted unexpectedly, or the operator oscillating between cluster states. Cause: More than one ClickHouseCluster custom resource exists in the same namespace. The operator assumes a single ClickHouseCluster per namespace and cannot correctly reconcile when multiple instances share a namespace. Resolution:

Check how many ClickHouseCluster resources exist in the namespace:
kubectl get clickhouseclusters -n <namespace>
If more than one exists, move the additional clusters to their own namespaces. Each ClickHouseCluster must be the only instance in its namespace.
The onprem-clickhouse-cluster Helm chart creates a ResourceQuota by default that prevents this situation. Verify the quota is in place:
kubectl get resourcequota -n <namespace>
If the quota is missing, ensure resourceQuota.enabled is set to true (the default) in your Helm values.

CR not in healthy Running state

Symptom: The ClickHouseCluster CR shows a state other than Running, or pods are restarting. Cause: Various issues can prevent a healthy state — crashlooping pods, resource constraints, misconfigurations, or scheduling failures. Resolution:

Check cluster status across all namespaces:
kubectl get clickhouseclusters -A
Describe the problem pod for detailed status and events:
kubectl describe pod <pod-name> -n <namespace>
Check logs from the previously terminated container:
kubectl logs -p <pod-name> -n <namespace>
Check namespace events for scheduling or resource issues:
kubectl get events -n <namespace>

ClickHouse Server Issues

Crashlooping server pods

Symptom: ClickHouse server pods are in a CrashLoopBackOff state. Cause: The ClickHouse server process is crashing on startup or shortly after. This could be due to configuration errors, corrupt data, memory pressure, or OOM kills by Kubernetes. Resolution:

Check the ClickHouse server pod logs for the crash reason:

kubectl logs <pod-name> -n <namespace>
kubectl logs -p <pod-name> -n <namespace>

If the crash is due to memory pressure or Kubernetes-initiated termination, check Kubernetes events:
kubectl describe pod <pod-name> -n <namespace> kubectl get events -n <namespace>

Data loss/corruption (ClickHouseBrokenPartDetectedOnSelect)

Symptom: The ClickHouseBrokenPartDetectedOnSelect alert fires, or SELECT queries fail with POTENTIALLY_BROKEN_DATA_PART errors. Cause: An SMT data part read failed with a non-retriable error. The POTENTIALLY_BROKEN_DATA_PART exception is thrown when a data part is found to be broken during a SELECT operation. Resolution:

Examine logs for the POTENTIALLY_BROKEN_DATA_PART exception. If not found in logs, also check system.errors.

Understand what data parts are lost. Query system.replicas to find affected tables:

SELECT
    database,
    table,
    lost_part_count AS value
FROM system.replicas
WHERE value > 0

Find logs related to lost parts. Check /var/log/clickhouse-server/ on the pod (use zgrep for archived logs), or query system.text_log:

SELECT
    hostName(),
    event_time,
    logger_name,
    message
FROM clusterAllReplicas(default, system.text_log)
WHERE message_format_string = 'Part {} is lost forever.'
ORDER BY hostName(), event_time

Add a predicate for event_time range to speed up the query.

Understand the history of lost parts. Pick a lost part name and find all related logs:
SELECT event_time, message FROM system.text_log WHERE message LIKE '%<part name>%' AND hostName() = '<host where lost forever log was created>' ORDER BY event_time ASC
Focus on log messages before Part * is lost forever. Messages after that point are irrelevant (any “found” part is actually an empty replacement).
Check for false positives:
- Check if the table has TTL and the lost part should have been dropped anyway.
- Check system.query_log for TRUNCATE or DROP PARTITION queries that should have dropped the lost parts.
Additional investigation:
- If the part was detached as broken, determine why it was broken.
- If you see The specified key does not exist, search all logs with the blob name to find when and why it was removed. Also check log messages about zero-copy locks.
Contact ClickHouse support with your findings.

Table replicas read-only (ClickHouseTableReplicasReadOnly)

Symptom: The ClickHouseTableReplicasReadOnly alert fires, or writes to tables fail because they are in read-only mode. Cause: A table has been in read-only mode for at least one hour. This excludes tables in *_broken_replicated_tables and *_broken_tables databases. It could be caused by a DROP operation that went badly, or a keeper connectivity issue. Resolution:

Check if there are still read-only tables in the cluster:

SELECT
  dateDiff('second', readonly_start_time, now()) AS readonly_duration_seconds,
  database,
  table,
  hostname()
FROM clusterAllReplicas(default, system.replicas)
WHERE is_readonly = 1

Investigate the cause. Having read-only tables typically indicates that StorageSharedMergeTree::shutdown was run but the storage object was kept alive. Search text logs using the table name as the logger name.
Try restarting the replica for each affected table:
SYSTEM RESTART REPLICA <database>.<table>
You can get the table names from the query in step 1. Sometimes the problem is trivial and a restart resolves it.
If the issue persists, check keeper logs for potential connectivity or coordination issues.

Replica already exists (ClickHouseReplicaAlreadyExists)

Symptom: The ClickHouseReplicaAlreadyExists alert fires, or replica creation fails with a REPLICA_ALREADY_EXISTS error. Cause: A replicated table (SMT or RMT) could not be created because an existing replica is already associated with the ZooKeeper path. This is unlikely to be caused by user error (explicit UUID reuse is now prohibited via database_replicated_allow_explicit_uuid). This is likely a bug in the Replicated database or Shared Catalog. Resolution: Contact ClickHouse support. This is likely a bug that requires investigation by the engineering team.

Cannot write to file descriptor (ClickHouseCannotWriteToFileDescriptor)

Symptom: The ClickHouseCannotWriteToFileDescriptor alert fires, or errors such as CANNOT_WRITE_TO_FILE_DESCRIPTOR or no space left on device appear in logs. Cause: The cache disk is full. The exception is thrown when there is not enough space for a new cache entry or for external data processing (e.g., external aggregation, external joins). This may be due to a misconfiguration where the disk was created with less space than requested in the CR config. Resolution:

Check for the known partial_merge issue first. There is a known bug in tracking cache disk usage when the join_algorithm = 'partial_merge' query setting is specified. Check if this setting is in use.

Connect to the pod:

kubectl exec -n <namespace> -it <pod> -- /bin/bash

Check actual disk size:
df -h
Check required cache disk size in ClickHouse:
SELECT path, max_size FROM system.filesystem_cache_settings
Note that multiple caches (e.g., s3diskWithCache, diskPlainRewritableForSystemTablesWithCache) may share the same path (/mnt/clickhouse-cache/sharedS3DiskCache).
Compare actual vs. required:
- If the actual disk size is smaller than required, the issue is a misconfiguration. Contact ClickHouse support.
- If the disk size is sufficient, it is likely a bug in cache disk usage tracking. Investigate via system.filesystem_cache.

Storage and Infrastructure Issues

Zone details could not be found for any PV

Symptom: The ClickHouse operator logs show the error zone details couldn't be found for any PV, and reconciliation cannot complete. Cause: The operator expects topology-aware node affinities to be automatically populated on PersistentVolumes using the topology.kubernetes.io/zone label. Normally this is handled by topology-aware volume provisioners (e.g., AWS EBS). However, for certain cloud providers (e.g., IBM) or on-premise environments, this label is not automatically populated or non-standard labels are used. Resolution:

Add allowedTopologies to your StorageClass to ensure volumes are created with the correct node affinities:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-nvme-sc
allowVolumeExpansion: true
parameters:
  path: /nvme/disk
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values:
          - <your_cluster_name>-keeper
          - <your_host_1>
          - <your_host_2>
          - <your_host_3>
      - key: directpv.min.io/zone
        values:
          - default

If non-standard labels are used for topology, configure the operator’s additionalZoneLabelRegexes property. For example, when using the Helm chart, set the operator.additionalZoneLabelRegexes Helm value to a regex matching your labels (e.g., directpv.*zone).

NVMe disk questions

Symptom: Uncertainty about whether NVMe-attached instances are required, or questions about using alternative storage for the cache disk. Cause: The NVMe disk serves as a cache volume for data coming from S3 (or equivalent object storage). The operator mounts it in ClickHouse server pods and configures it as an S3 cache disk:

s3diskWithCache:
    type: cache
    disk: s3disk
    path: /mnt/clickhouse-cache/sharedS3DiskCache
    max_size:
    '@from_env': CONFIG_DISK_CACHE_SIZE
    cache_on_write_operations: 1

With SharedMergeTree, ClickHouse does not store data locally. The cache exists solely to optimize query performance. Resolution: It is possible to use other storage devices (e.g., AWS EBS) instead of NVMe. If doing so, adjust the following:

Ensure the disks are attached to the instances used for ClickHouse server Kubernetes pods.
Alter the launch template to reflect changes in disk architecture. You can still create a RAID disk and format as ext4 or xfs, but use the appropriate tools (e.g., lsblk) to list devices.
If the disks are not mounted at /nvme/disk, set the hostPathBaseDirectory in the ClickHouseCluster Helm chart to the actual mount point.

Note that alternative storage choices may impact query performance.

Configuration Issues

loadBalancerType field behavior

Symptom: Setting the loadBalancerType field on the ClickHouseCluster CRD does not create load balancer annotations on the Service. Cause: The loadBalancerType field does not add load balancer annotations to the Service of the ClickHouse cluster. The ClickHouse operator does not support creating a load balancer for the ClickHouse cluster. This field only serves a purpose in ClickHouse Cloud, where it is used to help create users for the Cloud SQL Console. Resolution: No action is needed unless you are running in ClickHouse Cloud. If you need a load balancer for your ClickHouse cluster, configure it separately through standard Kubernetes Service annotations and configurations outside of the ClickHouseCluster CRD.

Diagnostic Commands Quick Reference

Cluster status

kubectl get clickhouseclusters -A

Operator logs

kubectl logs -f -n clickhouse-operator-system <operator-pod-name>

Pod events and details

kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace>

Previous container logs (after a crash)

kubectl logs -p <pod-name> -n <namespace>

Replica state map

kubectl get clickhousecluster <cluster-name> -n <namespace> -o json | jq .status.replicaStateMap

Replication queue size per table (SQL)

SELECT
    concat(database, '.', table),
    count()
FROM system.replication_queue
GROUP BY
    database,
    table

Alternative using system.replicas:

SELECT
    concat(database, '.', table),
    queue_size
FROM system.replicas

Replication queue oldest entry per table (SQL)

Alert if the oldest entry is older than 1 day.

SELECT
    concat(database, '.', table),
    min(create_time)
FROM system.replication_queue
GROUP BY
    database,
    table

​Operator Issues

​CR changes not being applied

​How to drop a server replica

​Server pod hanging on termination

​Multiple ClickHouseClusters in the same namespace

​CR not in healthy Running state

​ClickHouse Server Issues

​Crashlooping server pods

​Data loss/corruption (ClickHouseBrokenPartDetectedOnSelect)

​Table replicas read-only (ClickHouseTableReplicasReadOnly)

​Replica already exists (ClickHouseReplicaAlreadyExists)

​Cannot write to file descriptor (ClickHouseCannotWriteToFileDescriptor)

​Storage and Infrastructure Issues

​Zone details could not be found for any PV

​NVMe disk questions

​Configuration Issues

​loadBalancerType field behavior

​Diagnostic Commands Quick Reference

​Cluster status

​Operator logs

​Pod events and details

​Previous container logs (after a crash)

​Replica state map

​Replication queue size per table (SQL)

​Replication queue oldest entry per table (SQL)