> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure Alerting for ClickHouse

Set up Prometheus scrape targets and alerting for ClickHouse Server, Keeper, and the operator.

## Set Up Prometheus Scrape Targets for ClickHouse Server

ClickHouse server pods include `prometheus.io/*` annotations that expose basic metrics. For the full set of metrics (including `ClickHouse_CustomMetrics_*`), configure the `:8123/metrics` endpoint as a Prometheus scrape target on each server pod via a PodMonitor or equivalent.

This endpoint **requires authentication**. Authenticate with a dedicated user that has read-only privileges.

## Set Up PodMonitor for Keeper

Create a PodMonitor targeting the `:8001/metrics` endpoint on Keeper pods to capture Keeper metrics.

## Alert Rules

For the full list of recommended alert definitions (Prometheus expressions, thresholds, and actions), see [reference/metrics-and-alerts.md](/docs/cloud/clickhouse-private/reference/metrics-and-alerts).

Key alerts include:

| Alert                          | Metric                                                 | Purpose                                                 |
| ------------------------------ | ------------------------------------------------------ | ------------------------------------------------------- |
| Operator not reconciling       | `last_cluster_reconcile`                               | Operator has not reconciled within expected window      |
| Operator reconciliation errors | `controller_runtime_reconcile_errors_total`            | Error rate exceeds 5% of reconciliation attempts        |
| Broken detached parts          | `ClickHouse_CustomMetric_NumberOfBrokenDetachedParts`  | More than 100 broken detached parts for over 60 minutes |
| Data loss                      | `ClickHouse_CustomMetric_LostPartCount`                | Lost parts detected (potential data loss)               |
| Server stuck shutting down     | `ClickHouseMetrics_IsServerShuttingDown`               | Server has been shutting down for over an hour          |
| Table replicas read only       | `ClickHouse_CustomMetric_TableReadOnlyDurationSeconds` | Table in read-only mode for over an hour                |

## Standard Kubernetes Alerts

In addition to ClickHouse-specific alerts, add standard Kubernetes alerts for:

* Crashlooping pods
* Unschedulable pods
* Other infrastructure-related issues particular to your environment