Monitoring Kafka metrics with ClickHouse Observability

Collect Kafka broker metrics with the OpenTelemetry JMX Metric Gatherer and send them to ClickHouse Observability. Use the demo dataset to try the integration without a Kafka deployment.

Prerequisites

A ClickHouse Observability service
Docker with Docker Compose and curl
For your own metrics, a Kafka broker with remote JMX enabled and reachable from the gatherer

Connection details

Fully managed
Self-managed

Open ClickHouse Observability and go to Team Settings → Data → Managed Telemetry. Copy your OTLP HTTP endpoint and API key. You can use Copy connection details to copy the configuration, or Reveal key to view the masked key.Set these variables in the terminal where you will run the JMX gatherer or demo collector:

export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_ENDPOINT:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY'

Replace the placeholders with the exact endpoint, including its port, and API key from your service. Use the base HTTP endpoint without a /v1/metrics suffix; the exporter appends the signal path.

Use your collector’s reachable OTLP HTTP endpoint (for example, http://YOUR_COLLECTOR_HOST:4318) and authentication settings. If your collector does not require authentication, omit the JMX gatherer’s -Dotel.exporter.otlp.headers argument and the demo collector’s headers block. See the self-managed getting-started guide for setup guidance.

export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY'

The endpoint must be reachable from the gatherer or collector container. For a destination collector running on this computer, use http://host.docker.internal:4318. The Docker commands below include the host mapping.

Integration with an existing Kafka deployment

Check the broker's JMX endpoint

Use the remote JMX endpoint configured for your broker. For a broker exposing JMX on port 9999:

export KAFKA_JMX_URL='service:jmx:rmi:///jndi/rmi://YOUR_KAFKA_HOST:9999/jmxrmi'
export KAFKA_BROKER_ID='broker-0'

Replace the hostname and broker ID with your own values. The RMI hostname advertised by the broker must also be reachable from the gatherer. If the broker uses separate registry and RMI ports, allow both through the network. The gatherer reads JMX; it does not need the Kafka client port 9092.For containers on a shared Docker network, use the broker’s container hostname. For a broker on the Docker host, use host.docker.internal and configure the broker to advertise a hostname the container can reach. Keep JMX access restricted to your monitoring network and retain your broker’s authentication and TLS settings.

Download the JMX Metric Gatherer

curl -fL -o opentelemetry-jmx-metrics.jar \
  https://github.com/open-telemetry/opentelemetry-java-contrib/releases/download/v1.32.0/opentelemetry-jmx-metrics.jar

Configure the gatherer

Save this as compose.yaml beside the downloaded JAR:

services:
  kafka-jmx-gatherer:
    image: eclipse-temurin:11-jre
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      - KAFKA_JMX_URL
      - KAFKA_BROKER_ID
      - OTEL_EXPORTER_OTLP_ENDPOINT
      - OTEL_EXPORTER_OTLP_AUTHORIZATION
    volumes:
      - ./opentelemetry-jmx-metrics.jar:/app/opentelemetry-jmx-metrics.jar:ro
    command:
      - sh
      - -c
      - >-
        exec java
        -Dotel.jmx.service.url="$${KAFKA_JMX_URL}"
        -Dotel.jmx.target.system=kafka
        -Dotel.jmx.interval.milliseconds=10000
        -Dotel.metrics.exporter=otlp
        -Dotel.exporter.otlp.protocol=http/protobuf
        -Dotel.exporter.otlp.endpoint="$${OTEL_EXPORTER_OTLP_ENDPOINT}"
        -Dotel.exporter.otlp.headers="authorization=$${OTEL_EXPORTER_OTLP_AUTHORIZATION}"
        -Dotel.resource.attributes="service.name=kafka-service,kafka.broker.id=$${KAFKA_BROKER_ID}"
        -jar /app/opentelemetry-jmx-metrics.jar

If the broker is on an existing Docker network, attach kafka-jmx-gatherer to that network in your Compose configuration. For authenticated JMX, add otel.jmx.username and otel.jmx.password properties; for TLS, configure the appropriate trust store. See the gatherer’s authentication and TLS configuration.

Start collection

Set the connection variables and broker variables in this terminal, then run from the directory containing compose.yaml:

docker compose up kafka-jmx-gatherer

Keep the terminal open and check for JMX connection or OTLP export errors. The gatherer collects every 10 seconds. Allow at least two collection intervals before checking your service. Stop it with Ctrl+C when finished testing.For multiple brokers, run one gatherer per broker with its own JMX URL and kafka.broker.id.

Verify metrics

Open Chart Explorer, select your Metrics source, filter for ServiceName:kafka-service, and select Last 15 minutes. Look for kafka.partition.count, kafka.partition.under_replicated, or kafka.message.count. Traffic-related metrics need broker activity to produce meaningful values.

Demo dataset

The sample contains 6,282 gauge and sum points sampled every five minutes across roughly 29 hours. It preserves the original dataset’s traffic surges, request latency, consumer lag, and partition health patterns. No Kafka deployment or direct database access is needed. Set the connection variables before starting.

Download the sample metrics

Download the sample bundle (about 53 KB) and extract kafka-metrics-demo.json into your working directory.

Configure the demo collector

Save this as kafka-metrics-demo.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
processors:
  resource/demo:
    attributes:
      - key: service.name
        value: kafka-demo
        action: upsert
  transform/demo_timestamps:
    error_mode: propagate
    metric_statements:
      - context: datapoint
        statements:
          - set(cache["interval_ns"], time_unix_nano - start_time_unix_nano)
          - set(time, Now() - Duration("2m") - (Time("2025-11-06T22:14:00Z", "%Y-%m-%dT%H:%M:%SZ") - time))
          - set(start_time_unix_nano, time_unix_nano - cache["interval_ns"])
  batch:
    timeout: 5s
    send_batch_size: 500
    send_batch_max_size: 500
exporters:
  otlp_http/managed:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [resource/demo, transform/demo_timestamps, batch]
      exporters: [otlp_http/managed]

The demo-only transform refreshes timestamps while preserving metric values and the interval between each point’s start and end timestamps. Omit it for live Kafka metrics. The batch limit keeps uploads small enough for the destination collector.

Run the demo collector

From the directory containing the configuration, run the collector and leave this terminal open:

docker run --rm --name kafka-metrics-demo-collector \
  --add-host=host.docker.internal:host-gateway \
  --memory=512m \
  -p 127.0.0.1:14318:4318 \
  -e OTEL_EXPORTER_OTLP_ENDPOINT \
  -e OTEL_EXPORTER_OTLP_AUTHORIZATION \
  -v "$(pwd)/kafka-metrics-demo.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  otel/opentelemetry-collector-contrib:0.161.0 \
  --config=/etc/otelcol-contrib/config.yaml

Send the sample

In another terminal, run this command from the download directory:

curl --fail-with-body -sS http://localhost:14318/v1/metrics \
  -H 'Content-Type: application/json' \
  --data-binary @kafka-metrics-demo.json

A response such as {} or {"partialSuccess":{}} indicates the local collector accepted the request. Check the collector terminal for export errors and allow a few seconds for forwarding. Stop it with Ctrl+C after verifying the data. Sending the sample again can produce duplicate points.

Verify the metrics

Import the dashboard in the next section, filter for ServiceName:kafka-demo, and select Last 2 days. The sample includes both gauge and sum metrics; no direct database access is needed.

Dashboards and visualization

Import the dashboard

Open Dashboards and click Import in the upper-right corner.
Upload kafka-metrics-dashboard.json.
Map the charts to your Metrics source and click Finish Import.

View the dashboard

For the sample, filter for ServiceName:kafka-demo and select Last 2 days. The dashboard shows throughput, consumer lag, request latency, queue depth, and offline partitions.

The dashboard uses the demo’s metric names, including precomputed rates and consumer lag. The broker JMX gatherer exposes a different set of metrics, so adjust charts to match your live data. Broker JMX alone does not provide consumer-group lag.

Troubleshooting

JMX connection fails

Check the gatherer output with docker compose logs kafka-jmx-gatherer. Confirm JMX is enabled, both the registry and advertised RMI endpoint are reachable, and the configured JMX credentials and trust store match the broker. Containers must share the intended network when using container hostnames.

No metrics appearing

Check the gatherer or demo collector output for connection, authentication, or export errors.
Verify the OTLP HTTP endpoint and authentication against your service’s connection details.
Check the Metrics source, service-name filter, and time range. For the demo, use Last 2 days.
For an existing collector, confirm its metrics pipeline includes the OTLP receiver and a metrics exporter.
Verify live metric names in Chart Explorer before using the demo dashboard.

JMX port conflicts in Kafka client commands

If you run Kafka client commands inside the broker container, unset the inherited broker JMX settings for that command so the client does not try to bind the same port:

env -u JMX_PORT -u KAFKA_JMX_OPTS kafka-topics --list --bootstrap-server YOUR_KAFKA_HOST:9092

Next steps

Set up alerts for partition health or request latency.
Create additional dashboards for your broker metrics.
See Sending OpenTelemetry data for collector deployment guidance.

Introduction

Features

Administration

Guides

Integration partners

Self-managed

Monitoring Kafka metrics with ClickHouse Observability

Prerequisites

Connection details

Integration with an existing Kafka deployment

Demo dataset

Dashboards and visualization

Troubleshooting

JMX connection fails

No metrics appearing

JMX port conflicts in Kafka client commands

Next steps

​Prerequisites

​Connection details

​Integration with an existing Kafka deployment

​Demo dataset

​Dashboards and visualization

​Troubleshooting

​JMX connection fails

​No metrics appearing

​JMX port conflicts in Kafka client commands

​Next steps

Prerequisites

Connection details

Integration with an existing Kafka deployment

Demo dataset

Dashboards and visualization

Troubleshooting

JMX connection fails

No metrics appearing

JMX port conflicts in Kafka client commands

Next steps