Monitoring Kubernetes with ClickHouse Observability

Collect container logs, node and cluster metrics, and Kubernetes events with two OpenTelemetry collectors: a DaemonSet on each node and a single deployment for cluster-wide data.

Prerequisites

A ClickHouse Observability service
A Kubernetes cluster with Linux nodes and permission to create workloads, ClusterRoles, and host-path mounts
kubectl configured for the intended cluster and Helm 3 installed
Network access from the collectors to your service’s OTLP HTTP endpoint, the Kubernetes API, and kubelets

These collectors gather infrastructure data. Instrument your applications separately to send application metrics and traces; see OpenTelemetry integrations.

Connection details

Fully managed
Self-managed

Open ClickHouse Observability and go to Team Settings → Data → Managed Telemetry. Copy the OTLP HTTP endpoint and API key.Set these variables in your terminal:

export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_ENDPOINT:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY'

Use the exact endpoint, including its port, without a /v1/logs or /v1/metrics suffix.

Use your collector’s OTLP HTTP endpoint reachable from the cluster and its authentication settings. If authentication is disabled, set OTEL_EXPORTER_OTLP_AUTHORIZATION to an empty string and omit the exporter’s headers block below. See the self-managed setup guide.

export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY'

Deploy the collectors

Create the namespace and connection settings

Check that kubectl points to the intended cluster:

kubectl config current-context
kubectl create namespace observability --dry-run=client -o yaml | kubectl apply -f -

kubectl create configmap observability-connection -n observability \
  --from-literal=endpoint="$OTEL_EXPORTER_OTLP_ENDPOINT" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic observability-connection -n observability \
  --from-literal=authorization="$OTEL_EXPORTER_OTLP_AUTHORIZATION" \
  --dry-run=client -o yaml | kubectl apply -f -

Both collectors use these settings in the observability namespace.

Save the shared configuration

Save this as k8s_common.yaml:

image:
  repository: otel/opentelemetry-collector-contrib
  tag: "0.161.0"
command:
  name: otelcol-contrib
resources:
  limits:
    memory: 512Mi
presets:
  kubernetesAttributes:
    enabled: true
extraEnvs:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    valueFrom:
      configMapKeyRef:
        name: observability-connection
        key: endpoint
  - name: OTEL_EXPORTER_OTLP_AUTHORIZATION
    valueFrom:
      secretKeyRef:
        name: observability-connection
        key: authorization
service:
  enabled: false
ports:
  otlp:
    enabled: false
  otlp-http:
    enabled: false
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false
config:
  receivers:
    otlp: null
    jaeger: null
    zipkin: null
    prometheus: null
  exporters:
    debug: null
    otlp_http/observability:
      endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
      headers:
        authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
      compression: gzip
  processors:
    batch:
      timeout: 5s
      send_batch_size: 500
      send_batch_max_size: 500
  service:
    pipelines:
      logs:
        receivers: []
        exporters: [otlp_http/observability]
      metrics:
        receivers: []
        exporters: [otlp_http/observability]
      traces: null

The chart presets add the receivers, Kubernetes metadata processor, RBAC rules, and mounts needed by each collector. The shared configuration forwards logs and metrics to your service and disables unused application-ingestion receivers.

Configure node collection

Save this as k8s_daemonset.yaml:

mode: daemonset
presets:
  logsCollection:
    enabled: true
    storeCheckpoints: true
  hostMetrics:
    enabled: true
  kubeletMetrics:
    enabled: true
config:
  receivers:
    kubeletstats:
      insecure_skip_verify: true

The DaemonSet reads container logs and collects host and kubelet metrics. Checkpoints preserve log read positions across collector restarts. This preset runs the collector as root to write checkpoints on the node; your cluster’s pod security settings must permit its host mounts.The example skips kubelet certificate verification for clusters whose kubelet certificates are not trusted by the collector. If your cluster provides trusted kubelet certificates, set insecure_skip_verify to false and configure the appropriate CA.

Configure cluster collection

Save this as k8s_deployment.yaml:

mode: deployment
replicaCount: 1
presets:
  kubernetesEvents:
    enabled: true
  clusterMetrics:
    enabled: true

Keep one replica to avoid collecting duplicate cluster metrics and events.

Install the Helm releases

Run from the directory containing the three configuration files. These examples use OpenTelemetry Collector chart 0.175.1 and Collector Contrib 0.161.0:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry

helm upgrade --install observability-nodes open-telemetry/opentelemetry-collector \
  --version 0.175.1 --namespace observability \
  -f k8s_common.yaml -f k8s_daemonset.yaml --wait

helm upgrade --install observability-cluster open-telemetry/opentelemetry-collector \
  --version 0.175.1 --namespace observability \
  -f k8s_common.yaml -f k8s_deployment.yaml --wait

Check the collectors:

kubectl get pods -n observability
kubectl logs -n observability -l app.kubernetes.io/instance=observability-nodes --tail=50
kubectl logs -n observability -l app.kubernetes.io/instance=observability-cluster --tail=50

If your nodes have scheduling taints, add appropriate tolerations to k8s_daemonset.yaml for the nodes you intend to monitor.

Verify the data

Allow a few collection intervals, then open ClickHouse Observability:

In Search, select your Logs source and Last 15 minutes. Expand a container log and check attributes such as k8s.namespace.name, k8s.pod.name, and k8s.node.name.
In Chart Explorer, select your Metrics source and look for metrics beginning with k8s. or system..
Check your Logs source for Kubernetes events as they occur. Event volume depends on cluster activity.

Application metadata

For instrumented applications, add pod metadata through the Kubernetes downward API. Merge this env list into your application container’s existing configuration and preserve any existing resource attributes:

env:
  - name: OTEL_SERVICE_NAME
    value: my-app
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_UID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: k8s.pod.name=$(POD_NAME),k8s.pod.uid=$(POD_UID),k8s.namespace.name=$(POD_NAMESPACE),k8s.node.name=$(NODE_NAME)

The Kubernetes metadata processor enriches collected container logs with workload metadata. These application variables attach pod identity to telemetry emitted by your application’s OpenTelemetry SDK.

Troubleshooting

Collectors do not start

Inspect pod events with kubectl describe pod -n observability POD_NAME. Check image pulls, scheduling constraints, pod security restrictions, and that the ConfigMap and Secret exist in the same namespace as the releases.

No logs or metrics appear

Check collector logs for authentication or export errors. Confirm the destination endpoint is reachable from the cluster. For kubelet errors, check connectivity to port 10250, certificate settings, and the generated RBAC permissions. Container log collection requires access to the node’s /var/log/pods directory; restricted or serverless node environments may not support these host mounts.

Connection settings changed

After updating the ConfigMap or Secret, restart the collectors to reload their environment variables:

kubectl rollout restart daemonset -n observability -l app.kubernetes.io/instance=observability-nodes
kubectl rollout restart deployment -n observability -l app.kubernetes.io/instance=observability-cluster

Next steps

Build dashboards for cluster and workload health.
Set up alerts.
See the OpenTelemetry Helm chart for additional deployment settings.

Introduction

Features

Administration

Guides

Integration partners

Self-managed

Monitoring Kubernetes with ClickHouse Observability

Prerequisites

Connection details

Deploy the collectors

Application metadata

Troubleshooting

Collectors do not start

No logs or metrics appear

Connection settings changed

Next steps

​Prerequisites

​Connection details

​Deploy the collectors

​Application metadata

​Troubleshooting

​Collectors do not start

​No logs or metrics appear

​Connection settings changed

​Next steps

Prerequisites

Connection details

Deploy the collectors

Application metadata

Troubleshooting

Collectors do not start

No logs or metrics appear

Connection settings changed

Next steps