> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring Kubernetes with ClickHouse Observability

> Collect Kubernetes container logs, infrastructure metrics, and events with OpenTelemetry and send them to ClickHouse Observability.

Collect container logs, node and cluster metrics, and Kubernetes events with two OpenTelemetry collectors: a DaemonSet on each node and a single deployment for cluster-wide data.

## Prerequisites

* A ClickHouse Observability service
* A Kubernetes cluster with Linux nodes and permission to create workloads, ClusterRoles, and host-path mounts
* `kubectl` configured for the intended cluster and Helm 3 installed
* Network access from the collectors to your service's OTLP HTTP endpoint, the Kubernetes API, and kubelets

These collectors gather infrastructure data. Instrument your applications separately to send application metrics and traces; see [OpenTelemetry integrations](https://clickhouse.com/docs/clickstack/observability/ingesting-data/opentelemetry.md).

## Connection details

> [tab] Fully managed
>
> [Beta offering](https://clickhouse.com/docs/reference/settings/beta-and-experimental-features.md#beta-features)
>
> Open ClickHouse Observability and go to **Team Settings → Data → Managed Telemetry**. Copy the OTLP HTTP endpoint and API key.
>
> ![Managed Telemetry connection details](https://mintcdn.com/private-7c7dfe99/Zk1feNn6fCQBpold/images/clickstack/getting-started/observability-managed-telemetry.png?fit=max\&auto=format\&n=Zk1feNn6fCQBpold\&q=85\&s=e63a2ac2e6ab996d3fc360ebe0f487a5)
>
> Set these variables in your terminal:
>
> ```bash theme={null}
> export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_ENDPOINT:4318'
> export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY'
> ```
>
> Use the exact endpoint, including its port, without a `/v1/logs` or `/v1/metrics` suffix.

> [tab] Self-managed
>
> Use your collector's OTLP HTTP endpoint reachable from the cluster and its authentication settings. If authentication is disabled, set `OTEL_EXPORTER_OTLP_AUTHORIZATION` to an empty string and omit the exporter's `headers` block below. See the [self-managed setup guide](https://clickhouse.com/docs/clickstack/self-configured/getting-started/managed.md).
>
> ```bash theme={null}
> export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318'
> export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY'
> ```

## Deploy the collectors

**1. Create the namespace and connection settings**

Check that `kubectl` points to the intended cluster:

```bash theme={null}
kubectl config current-context
kubectl create namespace observability --dry-run=client -o yaml | kubectl apply -f -

kubectl create configmap observability-connection -n observability \
  --from-literal=endpoint="$OTEL_EXPORTER_OTLP_ENDPOINT" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic observability-connection -n observability \
  --from-literal=authorization="$OTEL_EXPORTER_OTLP_AUTHORIZATION" \
  --dry-run=client -o yaml | kubectl apply -f -
```

Both collectors use these settings in the `observability` namespace.

**2. Save the shared configuration**

Save this as `k8s_common.yaml`:

```yaml theme={null}
image:
  repository: otel/opentelemetry-collector-contrib
  tag: "0.161.0"
command:
  name: otelcol-contrib
resources:
  limits:
    memory: 512Mi
presets:
  kubernetesAttributes:
    enabled: true
extraEnvs:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    valueFrom:
      configMapKeyRef:
        name: observability-connection
        key: endpoint
  - name: OTEL_EXPORTER_OTLP_AUTHORIZATION
    valueFrom:
      secretKeyRef:
        name: observability-connection
        key: authorization
service:
  enabled: false
ports:
  otlp:
    enabled: false
  otlp-http:
    enabled: false
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false
config:
  receivers:
    otlp: null
    jaeger: null
    zipkin: null
    prometheus: null
  exporters:
    debug: null
    otlp_http/observability:
      endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
      headers:
        authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
      compression: gzip
  processors:
    batch:
      timeout: 5s
      send_batch_size: 500
      send_batch_max_size: 500
  service:
    pipelines:
      logs:
        receivers: []
        exporters: [otlp_http/observability]
      metrics:
        receivers: []
        exporters: [otlp_http/observability]
      traces: null
```

The chart presets add the receivers, Kubernetes metadata processor, RBAC rules, and mounts needed by each collector. The shared configuration forwards logs and metrics to your service and disables unused application-ingestion receivers.

**3. Configure node collection**

Save this as `k8s_daemonset.yaml`:

```yaml theme={null}
mode: daemonset
presets:
  logsCollection:
    enabled: true
    storeCheckpoints: true
  hostMetrics:
    enabled: true
  kubeletMetrics:
    enabled: true
config:
  receivers:
    kubeletstats:
      insecure_skip_verify: true
```

The DaemonSet reads container logs and collects host and kubelet metrics. Checkpoints preserve log read positions across collector restarts. This preset runs the collector as root to write checkpoints on the node; your cluster's pod security settings must permit its host mounts.

The example skips kubelet certificate verification for clusters whose kubelet certificates are not trusted by the collector. If your cluster provides trusted kubelet certificates, set `insecure_skip_verify` to `false` and configure the appropriate CA.

**4. Configure cluster collection**

Save this as `k8s_deployment.yaml`:

```yaml theme={null}
mode: deployment
replicaCount: 1
presets:
  kubernetesEvents:
    enabled: true
  clusterMetrics:
    enabled: true
```

Keep one replica to avoid collecting duplicate cluster metrics and events.

**5. Install the Helm releases**

Run from the directory containing the three configuration files. These examples use OpenTelemetry Collector chart `0.175.1` and Collector Contrib `0.161.0`:

```bash theme={null}
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry

helm upgrade --install observability-nodes open-telemetry/opentelemetry-collector \
  --version 0.175.1 --namespace observability \
  -f k8s_common.yaml -f k8s_daemonset.yaml --wait

helm upgrade --install observability-cluster open-telemetry/opentelemetry-collector \
  --version 0.175.1 --namespace observability \
  -f k8s_common.yaml -f k8s_deployment.yaml --wait
```

Check the collectors:

```bash theme={null}
kubectl get pods -n observability
kubectl logs -n observability -l app.kubernetes.io/instance=observability-nodes --tail=50
kubectl logs -n observability -l app.kubernetes.io/instance=observability-cluster --tail=50
```

If your nodes have scheduling taints, add appropriate `tolerations` to `k8s_daemonset.yaml` for the nodes you intend to monitor.

**6. Verify the data**

Allow a few collection intervals, then open ClickHouse Observability:

* In **Search**, select your **Logs** source and **Last 15 minutes**. Expand a container log and check attributes such as `k8s.namespace.name`, `k8s.pod.name`, and `k8s.node.name`.
* In **Chart Explorer**, select your **Metrics** source and look for metrics beginning with `k8s.` or `system.`.
* Check your **Logs** source for Kubernetes events as they occur. Event volume depends on cluster activity.

## Application metadata

For instrumented applications, add pod metadata through the Kubernetes downward API. Merge this `env` list into your application container's existing configuration and preserve any existing resource attributes:

```yaml theme={null}
env:
  - name: OTEL_SERVICE_NAME
    value: my-app
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_UID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: k8s.pod.name=$(POD_NAME),k8s.pod.uid=$(POD_UID),k8s.namespace.name=$(POD_NAMESPACE),k8s.node.name=$(NODE_NAME)
```

The Kubernetes metadata processor enriches collected container logs with workload metadata. These application variables attach pod identity to telemetry emitted by your application's OpenTelemetry SDK.

## Troubleshooting

### Collectors do not start

Inspect pod events with `kubectl describe pod -n observability POD_NAME`. Check image pulls, scheduling constraints, pod security restrictions, and that the ConfigMap and Secret exist in the same namespace as the releases.

### No logs or metrics appear

Check collector logs for authentication or export errors. Confirm the destination endpoint is reachable from the cluster. For kubelet errors, check connectivity to port `10250`, certificate settings, and the generated RBAC permissions. Container log collection requires access to the node's `/var/log/pods` directory; restricted or serverless node environments may not support these host mounts.

### Connection settings changed

After updating the ConfigMap or Secret, restart the collectors to reload their environment variables:

```bash theme={null}
kubectl rollout restart daemonset -n observability -l app.kubernetes.io/instance=observability-nodes
kubectl rollout restart deployment -n observability -l app.kubernetes.io/instance=observability-cluster
```

## Next steps

* Build [dashboards](https://clickhouse.com/docs/clickstack/observability/features/dashboards/overview.md) for cluster and workload health.
* Set up [alerts](https://clickhouse.com/docs/clickstack/observability/features/alerts.md).
* See the [OpenTelemetry Helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/opentelemetry-collector-0.175.1/charts/opentelemetry-collector) for additional deployment settings.