Monitoring Kafka logs with ClickHouse Observability

Collect Kafka broker logs with the OpenTelemetry filelog receiver and send them to ClickHouse Observability. Use the demo dataset to try the integration without a Kafka deployment.

Prerequisites

A ClickHouse Observability service
Docker and curl
For your own logs, access to Kafka broker log files matching the format below

Connection details

Fully managed
Self-managed

Open ClickHouse Observability and go to Team Settings → Data → Managed Telemetry. Copy your OTLP HTTP endpoint and API key. You can use Copy connection details to copy the configuration, or Reveal key to view the masked key.For the OTLP exporter examples below, set these variables in the terminal where you will run the collector:

export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_ENDPOINT:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY'

Replace the placeholders with the exact endpoint, including its port, and API key from your service. Use the base HTTP endpoint without a /v1/logs suffix; the exporter appends the signal path.

Use your collector’s reachable OTLP HTTP endpoint (for example, http://YOUR_COLLECTOR_HOST:4318) and authentication settings. If your collector does not require authentication, omit the exporter’s headers block from the configurations below. See the self-managed getting-started guide for setup guidance.

export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY'

The endpoint must be reachable from the collector container. For a destination collector running on this computer, use http://host.docker.internal:4318. The Docker commands below include the host mapping.

Integration with existing Kafka

Verify Kafka logging configuration

Locate your broker logs. Common locations include $KAFKA_HOME/logs and /var/log/kafka, depending on your installation and logging configuration:

ls "$KAFKA_HOME/logs/"
ls /var/log/kafka/

Common files are server.log (broker events), controller.log (controller events), and state-change.log (partition and replica transitions). Separate files exist only when configured in your logging appenders.The parser in this guide expects this Log4j text format:

[2026-03-09 14:23:45,123] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

If your broker logs only to stdout, configure a file appender in the broker’s logging configuration and share that directory with the collector. The filelog receiver does not read container stdout directly. Adjust the parser if your logging pattern differs.

Configure the collector

Save this as kafka-logs-monitoring.yaml:

receivers:
  filelog/kafka:
    include:
      - /var/log/kafka/server.log
      - /var/log/kafka/controller.log
      - /var/log/kafka/state-change.log
    start_at: end
    force_flush_period: 1s
    multiline:
      line_start_pattern: '^\[\d{4}-\d{2}-\d{2}'
    operators:
      - type: regex_parser
        regex: '(?s)^\[(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P<severity>\w+) (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%d %H:%M:%S,%L'
          location: UTC
        severity:
          parse_from: attributes.severity
      - type: move
        from: attributes.message
        to: body
      - type: add
        field: attributes.source
        value: kafka
      - type: add
        field: resource["service.name"]
        value: kafka-service
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 400
    spike_limit_mib: 100
  transform/kafka:
    error_mode: propagate
    log_statements:
      - context: log
        statements:
          - set(log.severity_text, ToLowerCase(log.severity_text))
  batch:
    timeout: 5s
    send_batch_size: 500
    send_batch_max_size: 500
exporters:
  otlp_http/observability:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
service:
  pipelines:
    logs/kafka:
      receivers: [filelog/kafka]
      processors: [memory_limiter, transform/kafka, batch]
      exporters: [otlp_http/observability]

Set location to the timezone used by your broker logs; the example assumes UTC. Multiline entries retain their complete stack traces. The collector reads new entries with start_at: end and normalizes severity text to lowercase.

Run the collector

Set the connection variables, then run from the configuration directory. Adjust the host log path if needed. For group-readable files, add --group-add LOG_GROUP_ID using the numeric group shown by ls -ln /var/log/kafka.

docker run --rm --name kafka-logs-collector \
  --add-host=host.docker.internal:host-gateway \
  --memory=512m \
  -e OTEL_EXPORTER_OTLP_ENDPOINT \
  -e OTEL_EXPORTER_OTLP_AUTHORIZATION \
  -v "$(pwd)/kafka-logs-monitoring.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  -v /var/log/kafka:/var/log/kafka:ro \
  otel/opentelemetry-collector-contrib:0.161.0 \
  --config=/etc/otelcol-contrib/config.yaml

Leave the terminal open and generate broker activity that writes new log entries. Stop with Ctrl+C when finished testing.

Verify logs

Open Search, select Logs, filter for ServiceName:kafka-service, and select Last 15 minutes. You should see broker messages with parsed timestamps and severity.

Demo dataset

The sample contains 6,376 Kafka broker log entries spanning approximately 24 hours. The collector refreshes their timestamps while preserving messages and stack traces.

Download the sample

curl -fL -o server.log \
  https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/kafka/server.log

Configure the demo collector

Save this as kafka-logs-demo.yaml:

receivers:
  filelog/kafka:
    include:
      - /var/log/kafka/server.log
    start_at: beginning
    force_flush_period: 1s
    multiline:
      line_start_pattern: '^\[\d{4}-\d{2}-\d{2}'
    operators:
      - type: regex_parser
        regex: '(?s)^\[(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P<severity>\w+) (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%d %H:%M:%S,%L'
          location: UTC
        severity:
          parse_from: attributes.severity
      - type: move
        from: attributes.message
        to: body
      - type: add
        field: attributes.source
        value: kafka-demo
      - type: add
        field: resource["service.name"]
        value: kafka-demo
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 400
    spike_limit_mib: 100
  transform/kafka:
    error_mode: propagate
    log_statements:
      - context: log
        statements:
          - set(log.severity_text, ToLowerCase(log.severity_text))
  transform/demo_timestamps:
    error_mode: propagate
    log_statements:
      - context: log
        statements:
          - set(log.time, Now() - Duration("2m") - (Time("2026-03-09T23:59:58.180Z", "%Y-%m-%dT%H:%M:%S.%LZ") - log.time))
  batch:
    timeout: 5s
    send_batch_size: 500
    send_batch_max_size: 500
exporters:
  otlp_http/observability:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
service:
  pipelines:
    logs/kafka:
      receivers: [filelog/kafka]
      processors: [memory_limiter, transform/kafka, transform/demo_timestamps, batch]
      exporters: [otlp_http/observability]

Run the demo collector

Set the connection variables, then run from the directory containing both files:

docker run --rm --name kafka-logs-demo-collector \
  --add-host=host.docker.internal:host-gateway \
  --memory=512m \
  -e OTEL_EXPORTER_OTLP_ENDPOINT \
  -e OTEL_EXPORTER_OTLP_AUTHORIZATION \
  -v "$(pwd)/kafka-logs-demo.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  -v "$(pwd)/server.log:/var/log/kafka/server.log:ro" \
  otel/opentelemetry-collector-contrib:0.161.0 \
  --config=/etc/otelcol-contrib/config.yaml

Leave the terminal open while verifying ingestion. Stop with Ctrl+C when finished. Running the demo again re-ingests the sample with refreshed timestamps.

Verify demo logs

Open Search, select Logs, filter for ServiceName:kafka-demo, and select Last 2 days.

Dashboards and visualization

Import the dashboard

Open Dashboards and click Import in the upper-right corner.
Upload kafka-logs-dashboard.json.
Map the charts to your Logs source and click Finish Import.

View the dashboard

For the sample, optionally filter for ServiceName:kafka-demo and select Last 2 days. For your own logs, use ServiceName:kafka-service and a time range covering your records.

Troubleshooting

No logs appearing

Check the collector output for file permission, parsing, authentication, or export errors. Confirm your broker writes to the mounted directory and the filenames match the receiver’s include list. The live configuration reads new entries only.

Logs not parsing correctly

Compare a broker log entry with the expected format:

tail -n 20 /var/log/kafka/server.log

Adjust the regex and multiline start pattern if your logging format differs. Set the timestamp parser’s location to your broker’s logging timezone.

Next steps

Set up alerts for broker failures or replication errors.
Combine with Kafka metrics.
Create additional dashboards for controller events or partition reassignment.
See Sending OpenTelemetry data for deployment guidance.

Introduction

Features

Administration

Guides

Integration partners

Self-managed

Monitoring Kafka logs with ClickHouse Observability

Prerequisites

Connection details

Integration with existing Kafka

Demo dataset

Dashboards and visualization

Troubleshooting

No logs appearing

Logs not parsing correctly

Next steps

​Prerequisites

​Connection details

​Integration with existing Kafka

​Demo dataset

​Dashboards and visualization

​Troubleshooting

​No logs appearing

​Logs not parsing correctly

​Next steps

Prerequisites

Connection details

Integration with existing Kafka

Demo dataset

Dashboards and visualization

Troubleshooting

No logs appearing

Logs not parsing correctly

Next steps