Prometheus protocols and PromQL

Expose ClickHouse server metrics

If you’re using ClickHouse Cloud, you can expose metrics to Prometheus using the Prometheus Integration.

Configure a dedicated port when a Prometheus server needs to scrape ClickHouse’s own metrics:

<prometheus>
    <port>9363</port>
    <endpoint>/metrics</endpoint>
    <metrics>true</metrics>
    <asynchronous_metrics>true</asynchronous_metrics>
    <events>true</events>
    <errors>true</errors>
    <histograms>true</histograms>
    <dimensional_metrics>true</dimensional_metrics>
</prometheus>

Section <prometheus.handlers> can be used to make more extended handlers on the same port. This section is similar to <http_handlers> but works for prometheus protocols:

<prometheus>
    <port>9363</port>
    <handlers>
        <my_rule_1>
            <url>/metrics</url>
            <handler>
                <type>expose_metrics</type>
                <metrics>true</metrics>
                <asynchronous_metrics>true</asynchronous_metrics>
                <events>true</events>
                <errors>true</errors>
                <histograms>true</histograms>
                <dimensional_metrics>true</dimensional_metrics>
                <labels>
                    <environment>production</environment>
                    <shard from_env="SHARD_NAME"></shard>
                </labels>
            </handler>
        </my_rule_1>
    </handlers>
</prometheus>

Settings:

Name	Default	Description
`port`	none	Port that serves ClickHouse metrics.
`endpoint`	`/metrics`	HTTP endpoint for scraping metrics. It starts with `/`. Should not be used with the `<handlers>` section.
`url` / `headers` / `method`	none	Filters used to find a matching handler for a request. Similar to the fields with the same names in the `<http_handlers>` section.
`info`	true	Exposes the `ClickHouse_Info` gauge with server identity labels (`name`, `version`, `version_describe`, `version_major`, `version_minor`, `version_patch`).
`metrics`	true	Exposes metrics from `system.metrics`.
`asynchronous_metrics`	true	Exposes metrics from `system.asynchronous_metrics`.
`events`	true	Exposes metrics from `system.events`.
`errors`	true	Exposes error counts from `system.errors`.
`histograms`	true	Exposes metrics from `system.histogram_metrics`.
`dimensional_metrics`	true	Exposes metrics from `system.dimensional_metrics`.
`labels`	none	Constant labels added to every exposed metric. Each child element defines one label: the element name is the label name (which must match `[a-zA-Z_][a-zA-Z0-9_]*`) and the element value is the label value. Label values support standard config substitutions such as the `from_env` attribute. A label name is rejected when it starts with `__` (reserved by Prometheus), or when it would collide with a label this endpoint already writes for one of its enabled sections. The reserved set therefore follows the endpoint’s active export surface: `le` when `histograms` is enabled; the `ClickHouse_Info` labels (`name`, `version`, `version_describe`, `version_major`, `version_minor`, `version_patch`) when `info` is enabled; and any label used by an exposed histogram or dimensional metric family (for example, `group`, `direction`, or `operation_type`) when `histograms` or `dimensional_metrics` is enabled. Because it depends on what the endpoint actually exposes, a name can be valid on one endpoint but rejected on another.

Check the endpoint:

curl http://127.0.0.1:9363/metrics

Prometheus HTTP API and PromQL

ClickHouse implements the Prometheus HTTP API over a TimeSeries table. One handler serves remote write, remote read, instant PromQL queries, and range PromQL queries.

Prerequisites

Enable the allow_experimental_time_series_table setting for the user that creates and accesses the table:

SET allow_experimental_time_series_table = 1;

Create a database and a TimeSeries table:

CREATE DATABASE prometheus;
CREATE TABLE prometheus.metrics ENGINE = TimeSeries;

For HTTP API requests, enable allow_experimental_time_series_table in the profile of the API user.

Configure the Prometheus API

Configure one prefix-routed handler on the main ClickHouse HTTP port:

<http_handlers>
    <defaults/>
    <rule>
        <url_prefix>/prometheus/api/v1</url_prefix>
        <handler>
            <type>prometheus_api_v1</type>
        </handler>
    </rule>
</http_handlers>

<defaults/> preserves the built-in handlers for endpoints such as /ping and for SQL requests. The prefix above exposes these endpoints through one handler:

Endpoint	Purpose
`/prometheus/api/v1/write`	Prometheus remote write
`/prometheus/api/v1/read`	Prometheus remote read
`/prometheus/api/v1/query`	Instant PromQL queries
`/prometheus/api/v1/query_range`	Range PromQL queries

The example omits database and table from the handler. Each request must provide the table query parameter. It can also provide database, use a qualified table name such as prometheus.metrics, or omit the database to use default. This allows one handler to serve multiple TimeSeries tables. To use one fixed table for every request, configure it in the handler:

<handler>
    <type>prometheus_api_v1</type>
    <database>prometheus</database>
    <table>metrics</table>
</handler>

A table configured in the handler cannot be overridden by request parameters. Routing and handler settings:

Name	Default	Description
`url_prefix`	none	Rule filter that matches every request path that starts with the configured prefix.
`table`	none	The name of a `TimeSeries` table. When omitted, the request must provide the `table` query parameter. The configured name can include a database.
`database`	none	The database containing the table. A request can provide it as a query parameter. When omitted, ClickHouse uses a database from a qualified `table` value or falls back to `default`.

Ingest metrics with remote write

ClickHouse supports the Prometheus remote-write protocol. Configure Prometheus to write to the handler:

remote_write:
  - url: https://clickhouse.example.com:8443/prometheus/api/v1/write?database=prometheus&table=metrics
    basic_auth:
      username: default
      password: <password>

Prometheus sends samples to the prometheus.metrics table.

Query with PromQL

Use the instant-query endpoint to evaluate a PromQL expression at one point in time:

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/query" \
  --data-urlencode "query=rate(http_requests_total[5m])" \
  --data-urlencode "database=prometheus" \
  --data-urlencode "table=metrics"

Use the range-query endpoint to evaluate an expression over a time range:

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/query_range" \
  --data-urlencode "query=rate(http_requests_total[5m])" \
  --data-urlencode "start=2026-08-15T12:00:00Z" \
  --data-urlencode "end=2026-08-15T13:00:00Z" \
  --data-urlencode "step=60s" \
  --data-urlencode "database=prometheus" \
  --data-urlencode "table=metrics"

See the supported PromQL features for the function and aggregation operator list used by the HTTP API, the promql dialect, and the table functions.

Grafana

Configure a Prometheus data source with the base URL ending before /api/v1:

apiVersion: 1
datasources:
  - name: ClickHouse Prometheus
    type: prometheus
    access: proxy
    url: https://clickhouse.example.com:8443/prometheus
    basicAuth: true
    basicAuthUser: default
    jsonData:
      httpMethod: GET
      customQueryParameters: database=prometheus&table=metrics
    secureJsonData:
      basicAuthPassword: <password>

Grafana appends /api/v1/query or /api/v1/query_range to this base URL and adds customQueryParameters to each request.

Only the query endpoints /api/v1/query and /api/v1/query_range are implemented. The metadata endpoints a Grafana Prometheus datasource uses for label browsing, template variables, and query-builder autocomplete (/api/v1/series, /api/v1/labels, /api/v1/label/<name>/values) are not implemented and return an error. Write PromQL expressions in code mode instead of the query builder.

SQL entry points

ClickHouse uses the same PromQL converter for the HTTP API, the promql dialect, and the prometheusQuery and prometheusQueryRange table functions. Run PromQL directly with clickhouse-client:

clickhouse-client \
  --dialect promql \
  --promql_database prometheus \
  --promql_table metrics \
  --query 'rate(http_requests_total[5m])'

Use the table functions to embed PromQL in a SQL query:

SELECT *
FROM prometheusQuery(
    prometheus.metrics,
    'rate(http_requests_total[5m])',
    now()
);

Read metrics with remote read

ClickHouse supports the Prometheus remote-read protocol at /prometheus/api/v1/read. Configure a Prometheus server to read from the same TimeSeries table:

remote_read:
  - url: https://clickhouse.example.com:8443/prometheus/api/v1/read?database=prometheus&table=metrics
    basic_auth:
      username: default
      password: <password>

Fundamentals

ClickHouse concepts

Best practices

Expose ClickHouse server metrics

Prometheus HTTP API and PromQL

Prerequisites

Configure the Prometheus API

Ingest metrics with remote write

Query with PromQL

Grafana

SQL entry points

Read metrics with remote read

​Expose ClickHouse server metrics

​Prometheus HTTP API and PromQL

​Prerequisites

​Configure the Prometheus API

​Ingest metrics with remote write

​Query with PromQL

​Grafana

​SQL entry points

​Read metrics with remote read

Expose ClickHouse server metrics

Prometheus HTTP API and PromQL

Prerequisites

Configure the Prometheus API

Ingest metrics with remote write

Query with PromQL

Grafana

SQL entry points

Read metrics with remote read