> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Alerts with ClickHouse Observability

> Alerts with ClickHouse Observability

ClickHouse Observability includes built-in support for alerting, enabling teams to detect and respond to issues in real time across logs, metrics, and traces.

Alerts can be created directly in the user interface and integrate with popular notification systems like Slack and PagerDuty.

Alerting works seamlessly across your observability data, helping you track system health, catch performance regressions, and monitor key business events.

## Types of alerts

You can create alerts from a **saved search**, directly in **Chart Explorer**, or from a **chart on a saved dashboard**. Search alerts count matching events, while chart alerts evaluate the values calculated by a chart.

### Search alerts

Search alerts allow you to trigger notifications based on the results of a saved search. They help you detect when specific events or patterns occur more (or less) frequently than expected.

An alert is triggered when the count of matching results within a defined time window either exceeds or falls below a specified threshold.

To create a search alert:

For an alert to be created for a search, the search must be saved. You can either create the alert for an existing saved search or save the search during the alert creation process. In the example below, we assume the search isn't saved.

**1. Open alert creation dialog**

Start by entering a [search](https://clickhouse.com/docs/clickstack/observability/features/search.md) and clicking the `Alerts` button in the top-right corner of the `Search` page.

![Search view with the Alerts button highlighted](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-alerts-search.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=e5d48a624dabb589accfeec06d8462d9)

**2. Create the alert**

From the alert creation panel, you can:

* Assign a name to the saved search associated with the alert.
* Choose a comparison operator, a matching-event count, and a time window. For example, trigger when at least 10 matching events appear within 5 minutes.
* Specify a `grouped by` value. This allows the search to be subject to an aggregation, e.g., `ServiceName`, thus allowing multiple alerts to be triggered off the same search.
* Choose a notification destination, such as email or a webhook. You can add a new webhook directly from this view. See [Adding a webhook](#add-webhook) for details.

Before saving, the interface visualizes the threshold condition so you can confirm it will behave as expected.

![Search alert configuration with a static threshold, ServiceName grouping, and webhook destination](https://mintcdn.com/private-7c7dfe99/FqnhjwVWZILRZzw7/images/clickstack/observability-search-alert.png?fit=max\&auto=format\&n=FqnhjwVWZILRZzw7\&q=85\&s=6f9a67970ecea98beca59fe6ba2fef9d)

Note that multiple alerts can be added to a search. If the above process is repeated, you will see the current alerts as tabs at the top of the edit alert dialog, with each alert assigned a number.

![New Alert tab beside an existing alert for a saved search](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-multiple-search-alerts.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=845dad7d679dd957cdc2e62ed464f18c)

### Chart Explorer alerts

Create an alert directly in [Chart Explorer](https://clickhouse.com/docs/clickstack/observability/features/dashboards/overview.md#navigate-chart-explorer) to monitor a chart without adding it to a dashboard.

**1. Build the chart**

Select **Chart explorer** in the sidebar and use the **Time Series** chart type. Choose a data source and configure the aggregation, filters, and grouping you want to monitor. For example, count error logs and group by `ServiceName` to evaluate each service separately.

Select **Run** and check the results before configuring the alert.

**2. Configure the alert**

Select **Add alert**. For a **Static threshold** alert, configure:

* **Trigger when value**: choose the comparison operator and threshold.
* **over**: choose the evaluation time window, such as 5 minutes.
* **Name**: give the alert a descriptive name.
* **Send to**: choose an email or webhook destination. To create a webhook, see [Adding a webhook](#add-webhook).

You can also add tags and a note to provide context for the alert.

![Chart Explorer with a log count chart and the alert panel showing threshold, name, notification destination, and Create alert controls](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-chart-explorer-alert.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=6c3498e471b05a78d43cd6293c315d1e)

**3. Create and manage the alert**

Select **Create alert** to save it. You do not need to save a dashboard first.

Open **Alerts** in the sidebar and select the alert to view its details, including its condition, notification destination, chart, and evaluation history.

![Chart Explorer alert detail page showing the alert status, threshold, evaluation interval, notification destination, chart, and evaluation history](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-chart-explorer-alert-details.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=f5c8d962b2524059b1d8af42a5149e00)

### Dashboard chart alerts

Dashboard alerts extend alerting to charts.

You can create a chart-based alert directly from a saved dashboard, powered by full SQL aggregations and ClickHouse functions for advanced calculations.

When a metric crosses a defined threshold, an alert triggers automatically, allowing you to monitor KPIs, latencies, or other key metrics over time.

> [!NOTE]
>
> For an alert to be created for a visualization on a dashboard, the dashboard must be saved.

To add a dashboard alert:

Alerts can be created during the chart creation process, when adding a chart to a dashboard, or added to existing charts. In the example below, we assume the chart already exists on the dashboard.

**1. Open the chart edit dialog**

Click the **Add alert** bell icon in the chart toolbar to open the chart edit dialog.

![Dashboard chart with the Add alert button highlighted](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-dashboard-add-alert.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=5487f5b3a99f4e8b7c8234ad36e7d6aa)

**2. Add the alert**

Select **Add alert**.

![Chart editor with the Add alert button highlighted](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-chart-editor-add-alert.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=8904f34f2273d03f497f66216df5c5c1)

**3. Define the alert conditions**

Define the condition (`>=`, `>`, `<=`, `<`, `=`, `!=`, `<= x >=`, `> or <`), threshold, duration, and webhook. The duration here will also dictate how often the alert is triggered.

![Chart editor with the alert conditions panel open](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-create-chart-alert.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=a975fe1355227038bf2e6fe6ebe2136c)

You can add a new webhook directly from this view. See [Adding a webhook](#add-webhook) for details.

## Adding a webhook

During alert creation, you can either use an existing webhook or create one. Once created, the webhook will be available for reuse across other alerts.

A webhook can be created for different service types, including Slack and PagerDuty, as well as generic targets.

In the alert panel, select **Add New Incoming Webhook** to create a notification destination.

![Alert panel with Add New Incoming Webhook highlighted](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-add-new-webhook.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=5242db6f9b8a4b6e6c768d4fd1a21767)

This opens the webhook creation dialog, where you can create a new webhook:

![Add New Webhook dialog with service type, name, URL, and description fields](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-add-webhook-dialog.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=2fbb79430799a35a4242e62cf402e415)

A webhook name is required, while descriptions are optional. Other settings that must be completed depend on the service type.

See [Service type integrations](#integrations) for supported notification services.

### Service type integrations

Alerts integrate out of the box with the following service types:

* **Slack**: send notifications directly to a channel via either a webhook or API.
* **PagerDuty**: route incidents for on-call teams via the PagerDuty API.
* **incident.io**: send alerts to incident.io.
* **Generic**: connect alerts to any custom system or workflow via a generic webhook.

> [!NOTE]
>
> **ClickHouse Cloud only integrations**
>
> The Slack API and PagerDuty integrations are only supported in ClickHouse Cloud.

Depending on the service type, you will need to provide different details. Specifically:

**Slack (Webhook URL)**

* Webhook URL. For example: `https://hooks.slack.com/services/<unique_path>`. See the [Slack documentation](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/) for further details.

**Slack (API)**

* Slack bot token. See the [Slack documentation](https://docs.slack.dev/authentication/tokens/#bot/) for further details.

**PagerDuty API**

* PagerDuty integration key. See the [PagerDuty documentation](https://support.pagerduty.com/main/docs/api-access-keys) for further details.

**Generic**

* Webhook URL
* Webhook headers (optional)
* Webhook body (optional). The body currently supports the template variables `{{title}}`, `{{body}}`, and `{{link}}`.

## Managing alerts

Select **Alerts** in the left sidebar to manage your alerts. You can search by name and filter by state, source, or type.

![Alert management page with Alerts highlighted in the sidebar](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-manage-alerts.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=0db07f9c7ba12f21b71f45a87c5a7756)

This view also displays the alert evaluation history. Alerts are evaluated on a recurring time interval (defined by the period/duration set during alert creation). During each evaluation, the service queries your data to check whether the alert condition is met:

* **Red bar**: The threshold condition was met during this evaluation and the alert fired (notification sent)
* **Green bar**: The alert was evaluated but the threshold condition wasn't met (no notification sent)

Each evaluation is independent - the alert checks the data for that time window and fires only if the condition is true at that moment.

In the example above, **Alert 1** has an **OK** status, with green bars showing recent evaluations where the threshold condition was not met.

Select an alert to open its detail page, including its condition, notification destination, and evaluation history.

### Deleting an alert

To remove a chart alert, open the chart editor and select **Edit alert**. Click the trash icon in the top-right corner of the alert panel.

![Chart editor with Edit alert and the alert deletion button highlighted](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-remove-chart-alert.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=40019d763d3e486249b91fb1524d208d)

## SQL-based chart alerts

SQL-based chart alerts let you write arbitrary ClickHouse SQL to define alert conditions. This gives you full control over filtering, aggregation, and math — anything you can express in SQL can become an alert.

### Supported chart types

SQL-based alerts are supported on three chart display types:

| Chart type | Behavior |
| - | - |
| **Line** | Time-series alert. The query must produce time-bucketed rows. Each bucket is evaluated independently against the threshold. |
| **Stacked Bar** | Time-series alert. Same behavior as Line. |
| **Number** | Single-value alert. The query returns a single numeric result which is compared against the threshold once per evaluation. |

Other SQL-based chart types (Table, Pie, Heatmap, etc.) do not support alerts.

### Creating a SQL alert

To create an alert on a SQL-based chart:

**1. Create or open a SQL-based chart on a dashboard**

From a saved dashboard, either [create a new chart with the **SQL** chart mode](https://clickhouse.com/docs/clickstack/observability/features/dashboards/sql-visualizations.md), or open an existing SQL-based chart for editing.

Choose **Line**, **Stacked Bar**, or **Number** as the display type.

![SQL chart editor with SQL and Add alert highlighted](https://mintcdn.com/private-7c7dfe99/FqnhjwVWZILRZzw7/images/clickstack/observability-sql-chart-alert.png?fit=max\&auto=format\&n=FqnhjwVWZILRZzw7\&q=85\&s=e7a0add9a251cab19d337c91d9818707)

**2. Add the alert**

Select **Add alert** from the alert section of the chart editor. Configure:

* **Threshold type**: `>=` (greater than or equal), `>` (greater than), `<=` (less than or equal), `<` (less than), `=` (equal), `!=` (not equal), `≤ x ≤` (between), or `< or >` (outside)
* **Threshold value**: The numeric value to compare against
* **Interval**: How often the alert is evaluated (5m, 15m, 30m, 1h, 6h, 12h, or 1d). This also defines the time window for each evaluation.
* **Webhook**: The notification channel to use when the alert fires. See [Adding a webhook](#add-webhook).

![SQL chart editor with alert threshold and notification settings open](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-raw-sql-alert.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=88788259bf8553407686fc1077da3a6d)

> [!WARNING]
>
> **Alert Time Range**
>
> Typically, alert queries are executed once per interval. However, if one or more intervals are skipped due to errors or slow queries, the following execution will use a time range that includes the missed intervals. In this case, the query's interval parameters would still be set to the alert's configured period, but the time range parameters would reflect the longer time range.

**3. Save the dashboard**

Save the dashboard to activate the alert. The alert will begin evaluating on the configured interval.

### How query results are interpreted

The alert system inspects the columns returned by your SQL query to determine what to compare against the threshold.

* **Value column**: The **last numeric column** in your `SELECT` clause is used as the alert value. If your query returns multiple numeric columns (e.g., `count, avg_latency, p99_latency`), only the last one (`p99_latency`) is compared to the threshold.
* **Timestamp column**: For time-series charts (Line and Stacked Bar), the system identifies the Date/DateTime column in your results as the time bucket (i.e. the x-axis on a time-series chart). The value column for each time bucket is evaluated against the threshold independently, and if the value for any time bucket breaches the configured threshold, the alert will trigger.
* **Group columns**: Any non-numeric, non-timestamp columns (e.g., `ServiceName`, `Environment`) are treated as grouping dimensions. When groups are present, each unique combination of group values is tracked and alerted on separately. The service will send an alert for each group with a value that breaches the configured threshold. Groups are only available for time-series charts.

### Query parameters and macros

SQL alert queries support template parameters and macros that are automatically replaced at evaluation time. These are the same parameters and macros available when [building a SQL-based chart](https://clickhouse.com/docs/clickstack/observability/features/dashboards/sql-visualizations.md).

#### Required and Recommended Parameters

Queries used for line or stacked bar chart alerts **must** include an interval parameter or macro (`{intervalSeconds:Int64}`, `{intervalMilliseconds:Int64}`, `$__timeInterval(col)`, or `$__timeInterval_ms(col)`). During alert execution, it will be replaced with the alert's configured period.

Queries used for alerts **should** include a time range filter (`{startDateMilliseconds:Int64}` and `{endDateMilliseconds:Int64}`, or `$__timeFilter(col)`, etc.). Regardless of whether a time range filter is present in the query, the alert query will run on the alert's configured period. If there is no time range filter, then the query will read the entire time range available in the source table during each execution.

> [!WARNING]
>
> **Alert Time Range**
>
> Typically, alert queries are executed once per interval. However, if one or more intervals are skipped due to errors or slow queries, the following execution will use a time range that includes the missed intervals. In this case, the query's interval parameters would still be set to the alert's configured period, but the time range parameters would reflect the longer time range.

### Example alert queries

#### Error rate per service (time-series)

Alert when any service has an error rate of at least 5%, with more than 10 requests in the alert period to avoid noisy alerts on low-traffic services.

```sql theme={null}
WITH error_rates AS (
  SELECT
    $__timeInterval(Timestamp) as ts,
    ServiceName,
    countIf (SpanKind = 'Server') as request_count,
    countIf (
      SpanKind = 'Server'
      and StatusCode = 'Error'
    ) as error_count,
    error_count / request_count * 100 AS error_percent
  FROM $__sourceTable
  WHERE $__timeFilter(Timestamp)
  GROUP BY ts, ServiceName
)
SELECT ts, ServiceName, error_percent
FROM error_rates
WHERE request_count > 10
```

**Display type**: Line or Stacked Bar
**Threshold**: `>= 5` (fires when error rate reaches 5%)

In this query, `ServiceName` is a non-numeric, non-timestamp column, so each service is tracked as a separate alert group. The alert fires independently per service.

#### Anomaly detection with lagging average (time-series)

Alert on excess error counts that exceed a rolling average by more than two standard deviations. This catches spikes relative to recent baseline behavior rather than a fixed threshold.

```sql theme={null}
WITH buckets AS (
  SELECT
    $__timeInterval(Timestamp) AS ts,
    count() AS bucket_count
  FROM $__sourceTable
  WHERE TimestampTime >= fromUnixTimestamp64Milli({startDateMilliseconds:Int64})
        - toIntervalSecond($__interval_s * 30) -- Fetch 30 intervals back
    AND TimestampTime < fromUnixTimestamp64Milli({endDateMilliseconds:Int64})
    AND SeverityText = 'error'
  GROUP BY ts
  ORDER BY ts
  WITH FILL
    FROM toDateTime(fromUnixTimestamp64Milli({startDateMilliseconds:Int64}))
    TO toDateTime(fromUnixTimestamp64Milli({endDateMilliseconds:Int64}))
    STEP toIntervalSecond($__interval_s)
),

anomaly_detection AS (
  SELECT
    ts,
    bucket_count,
    avg(bucket_count) OVER (
      ORDER BY ts ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING
    ) AS previous_30_avg,
    stddevPop(bucket_count) OVER (
      ORDER BY ts ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING
    ) AS previous_30_stddev,
    greatest(
      bucket_count - (previous_30_avg + 2 * previous_30_stddev), 0
    ) AS excess_error_count
  FROM buckets
)

SELECT ts, excess_error_count
FROM anomaly_detection
WHERE ts >= fromUnixTimestamp64Milli({startDateMilliseconds:Int64})
  AND ts < fromUnixTimestamp64Milli({endDateMilliseconds:Int64})
```

**Display type**: Line
**Threshold**: `> 0` (fires when excess errors above the rolling baseline are detected)

Note that the query fetches 30 intervals *before* the start of the date range to seed the rolling window calculations, then filters the final output to only the evaluation window.

## Common alert scenarios

Here are a few common alert scenarios you can configure:

**Errors:** We recommend setting up alerts for the default
`All Error Events` and `HTTP Status >= 400` saved searches to be notified when
excess errors occur.

**Slow Operations:** You can set up a search for slow operations (e.g.,
`duration:>5000`) and then alert when there are too many slow operations
occurring.

**User Events:** You can also set up alerts for customer-facing teams to be
notified when new users sign up or a critical user action is performed.