ClickHouseCluster and
KeeperCluster objects it manages. These events trace what the operator did
during reconciliation — where resource changes failed, when a cluster became
ready, why scaling was blocked — and surface failures that never reach a log a
user normally reads. They complement the metrics
by attaching a human-readable history directly to the custom resource.
The clickhouse-controller reports events on ClickHouseCluster objects and the
keeper-controller reports them on KeeperCluster objects. The resource
lifecycle failure events also reference the owned object they concern (a
StatefulSet, Service, ConfigMap, Secret, PodDisruptionBudget,
PersistentVolumeClaim, or version-probe Job); other events reference only the
cluster itself.
The Kubernetes API server expires events after a TTL — one hour by default
(
--event-ttl). Events are therefore a short-lived signal for recent activity,
not a durable audit trail.Viewing events
The quickest view iskubectl describe on the custom resource, which lists the
most recent events at the bottom:
events resource and filter by the involved object or by type:
ClickHouseCluster event (clickhouse-controller) from a KeeperCluster event
(keeper-controller).
Event reasons reference
The operator emits a fixed set of reasons, grouped by what they describe.Normal
events report expected progress; Warning events report a failure or a state a
user should act on.
Resource lifecycle
Emitted on bothClickHouseCluster and KeeperCluster when the operator fails to
apply an owned resource during reconciliation.
Cluster readiness
Emitted on both kinds when the cluster crosses a readiness boundary.Scaling
Emitted onKeeperCluster as the operator changes the replica count.
HorizontalScaleBlocked is the event to watch when a Keeper scale request seems
to do nothing: the operator deliberately holds the change and keeps the existing
quorum rather than risk a split. The event message states the constraint that
blocked the change.External secret
Emitted onClickHouseCluster when the cluster references an external Secret that
the operator cannot use. See the External Secret feature in the
configuration guide.
Version checks
Emitted by the version checks forClickHouseCluster and KeeperCluster.
VersionProbeFailed is specific to the ClickHouse version-probe Job.
ClickHouse server warnings
This last reason is distinct: it does not describe the operator’s own actions. On
each ready replica the operator periodically queries the server’s
system.warnings table and
republishes every row as a Warning event on the cluster, prefixed with the
replica it came from. That turns ClickHouse’s own configuration and runtime
warnings — obsolete settings, low limits, unsafe options — into events you can see
with kubectl without opening a clickhouse-client session against each replica.
Events, metrics, and conditions
The operator exposes three observability surfaces; use each for what it does best:- Events (this guide) — recent, human-readable, attached to the object. Best
for “what just happened to this cluster” and interactive troubleshooting with
kubectl describe. They expire. status.conditionson the custom resource — the current, persistent truth (ready, external secret valid, scale allowed, version in sync). Best for scripts and GitOps health gates. Read them withkubectl get clickhousecluster <name> -o jsonpath='{.status.conditions}'.- Metrics — durable and numeric. Best for dashboards and for alerting on a sustained reconcile error rate.
Warning event and a False condition often describe the same problem from two
angles: the event captures the moment and the message, the condition reflects the
state until it clears.
Troubleshooting with events
A few common signals and where they point:FailedCreate/FailedUpdaterepeating — the operator cannot apply a resource. The event message carries the API error (admission rejection, quota, invalid spec). Reconciliation retries, so a transient cause clears on its own; a persistent one needs a spec or cluster fix.ClusterNotReadywithout a matchingClusterReady— the cluster is not recovering. The event message names the not-ready shards or the quorum problem; check the pods behind them.HorizontalScaleBlocked— an intended scale is being held for safety. Read the message for the exact constraint before forcing anything.ExternalSecretNotFound/ExternalSecretInvalid— fix the Secret name or its keys; the matchingExternalSecretValidcondition flips toTrueonce the operator can use it.ClickHouseWarning— the problem is inside ClickHouse, not the operator. Treat the message as you would a row fromsystem.warnings.
Related guides
- Monitoring the operator — metrics and health probes, the durable counterpart to events.
- Scaling — what
HorizontalScaleBlockedprotects and how Keeper quorum bounds scaling. - Configuration — the External Secret feature behind the external-secret events.