Monitoring Nginx traces with ClickHouse Observability

Capture distributed traces from Nginx with the OpenTelemetry Nginx module and send them to ClickHouse Observability. Use the demo dataset to try the integration without installing Nginx.

Prerequisites

A ClickHouse Observability service
For your own traces, an Nginx installation with the OpenTelemetry module and permission to update its configuration; the Docker example below uses nginx:1.27-otel
For the demo dataset, curl and Docker

Connection details

Fully managed
Self-managed

Open ClickHouse Observability and go to Team Settings → Data → Managed Telemetry. Copy your endpoint and API key using Copy connection details, or select Reveal key to view the masked key.The Nginx module exports over OTLP/gRPC, normally on port 4317. The JSON demo uses OTLP/HTTP, normally on port 4318. Use the endpoint for the matching protocol, including https:// when TLS is required:

export OTEL_EXPORTER_OTLP_GRPC_ENDPOINT='https://YOUR_OTEL_HOST:4317'
export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_HOST:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY'

Replace the placeholders with your service’s connection details. Use the base HTTP endpoint without /v1/traces; the collector exporter adds that path.

Use your collector’s OTLP endpoints and authentication settings. If your collector does not require authentication, omit the Nginx header authorization directive and the demo collector’s headers block. See the self-managed getting-started guide.

export OTEL_EXPORTER_OTLP_GRPC_ENDPOINT='http://YOUR_COLLECTOR_HOST:4317'
export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY'

The gRPC endpoint must be reachable from Nginx. With Docker Desktop, use host.docker.internal to reach a collector on the host. For containers on a shared Docker network, use the collector’s container name. Use http:// for a local endpoint without TLS.

Integration with existing Nginx

Install the OpenTelemetry Nginx module

The nginx:1.27-otel image includes ngx_otel_module.so. For an installation outside Docker, follow the installation instructions linked from the Nginx OpenTelemetry module documentation.

Configure Nginx tracing

The following Docker example uses a template so the container’s startup script substitutes the connection environment variables. Save it as nginx.conf.template. The return 200 location provides a simple smoke test; for your application, retain your existing server and routing configuration and add the tracing directives shown here.

load_module modules/ngx_otel_module.so;

events {
    worker_connections 1024;
}

http {
    otel_exporter {
        endpoint ${OTEL_EXPORTER_OTLP_GRPC_ENDPOINT};
        header authorization "${OTEL_EXPORTER_OTLP_AUTHORIZATION}";
    }

    otel_service_name "nginx-proxy";
    otel_trace on;

    server {
        listen 80;

        location / {
            otel_trace_context propagate;
            otel_span_name "$request_method $uri";
            otel_span_attr http.status_code $status;
            otel_span_attr http.request.method $request_method;
            otel_span_attr http.route $uri;
            return 200 "Nginx tracing is enabled\n";
        }
    }
}

Save this as compose.yaml in the same directory:

services:
  nginx:
    image: nginx:1.27-otel
    ports:
      - "8081:80"
    environment:
      OTEL_EXPORTER_OTLP_GRPC_ENDPOINT: ${OTEL_EXPORTER_OTLP_GRPC_ENDPOINT}
      OTEL_EXPORTER_OTLP_AUTHORIZATION: ${OTEL_EXPORTER_OTLP_AUTHORIZATION}
      NGINX_ENVSUBST_OUTPUT_DIR: /etc/nginx
      NGINX_ENVSUBST_FILTER: ^OTEL_EXPORTER_OTLP_
    volumes:
      - ./nginx.conf.template:/etc/nginx/templates/nginx.conf.template:ro

The substitution filter preserves Nginx variables such as $request_method and $uri. Nginx does not automatically substitute shell environment variables in a directly mounted nginx.conf. Outside Docker, replace the connection placeholders with their actual values before validating the configuration, and restrict access to the file containing the key.

Validate and start Nginx

Set the connection variables in the same terminal, then run:

docker compose run --rm nginx nginx -t
docker compose up -d nginx
curl http://localhost:8081/

Each request creates a span with the method, path, status code, and duration. For an existing deployment, generate requests to your application’s URL. After changing the template or environment variables, run docker compose up -d --force-recreate nginx to apply them.

Verify traces in the UI

Open Search, select Traces, and filter for ServiceName:nginx-proxy. Select a time range covering your requests and allow a few seconds for export. Open a trace to inspect its spans and HTTP attributes.

Demo dataset

The sample contains 1,000 Nginx spans across nine endpoints, with successful requests, HTTP errors, and varied latencies. Set the connection variables before sending it.

Download the sample dataset

curl -fLO https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/nginx-traces-sample.json

Configure the demo collector

Save the following as nginx-traces-demo.yaml in the download directory:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64
  transform/demo_timestamps:
    error_mode: propagate
    trace_statements:
      - context: span
        statements:
          - set(cache["duration_ns"], end_time_unix_nano - start_time_unix_nano)
          - >-
            set(start_time,
              Now() - Duration("2m") -
              (Time("2025-10-27T19:36:05Z", "%Y-%m-%dT%H:%M:%SZ") - start_time)
            )
          - set(end_time_unix_nano, start_time_unix_nano + cache["duration_ns"])
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp_http/observability:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: "${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}"

service:
  pipelines:
    traces/nginx-demo:
      receivers: [otlp]
      processors: [memory_limiter, transform/demo_timestamps, batch]
      exporters: [otlp_http/observability]

The demo-only transform/demo_timestamps processor shifts this sample into roughly the past 24 hours, ending two minutes before ingestion, while preserving span durations and the traffic pattern. Omit this processor for your own Nginx traces so their original timestamps are preserved. Sending the sample again can produce duplicate spans.

Run the demo collector

For a destination collector running on this computer, set OTEL_EXPORTER_OTLP_ENDPOINT to http://host.docker.internal:4318. The command below includes the Docker host mapping.

Run OpenTelemetry Collector Contrib 0.161.0 from the directory containing the configuration, in the terminal where you set the connection variables. Keep this terminal open:

docker run --rm --name nginx-traces-demo-collector \
  --add-host=host.docker.internal:host-gateway \
  --memory=512m \
  -p 127.0.0.1:14318:4318 \
  -e OTEL_EXPORTER_OTLP_ENDPOINT \
  -e OTEL_EXPORTER_OTLP_AUTHORIZATION \
  -v "$(pwd)/nginx-traces-demo.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  otel/opentelemetry-collector-contrib:0.161.0 \
  --config=/etc/otelcol-contrib/config.yaml

The local receiver uses port 14318 to avoid conflicting with an existing collector on port 4318. The collector forwards the traces to your configured endpoint using the authentication settings above.

Send the traces

In another terminal, from the download directory, send the original sample to the local collector:

curl --fail-with-body -sS http://localhost:14318/v1/traces \
  -H "Content-Type: application/json" \
  --data-binary @nginx-traces-sample.json

An empty success response such as {} or {"partialSuccess":{}} indicates the local collector accepted the request. Check the collector’s output for export errors and allow a few seconds for the batch to reach your service. Stop the demo collector with Ctrl+C when finished.

Verify the demo traces

Open Search, select Traces, filter for ServiceName:nginx-proxy, and select Last 2 days to include the full shifted dataset. Open a trace to inspect the HTTP attributes.

Dashboards and visualization

Download the dashboard configuration

Import the dashboard

Open Dashboards and click Import in the upper-right corner.
Upload nginx-traces-dashboard.json.
Map the charts to your Traces source and click Finish Import.

View the dashboard

Filter for ServiceName:nginx-proxy and select a time range covering your traces. For the demo, select Last 2 days.

Troubleshooting

Nginx fails to start

Run docker compose run --rm nginx nginx -t to validate the generated configuration. An unknown otel_* directive or missing module file means the module is not installed or loaded. Check that you are using the -otel image and that the template is mounted at the documented path.

No traces appearing

Verify the gRPC endpoint and any authentication required by your destination collector. Managed services use the Bearer prefix. Do not use the HTTP endpoint for the Nginx module.
Confirm the endpoint is reachable from the Nginx container and that its scheme matches the destination’s TLS configuration.
Generate requests; an idle Nginx instance produces no request spans.
Check Traces, the service-name filter, and the selected time range. For the demo, confirm transform/demo_timestamps is included in the collector pipeline and select Last 2 days.
For the demo, check the collector terminal for authentication or export errors. Verify its HTTP endpoint and authorization value against your connection details.
Inspect Nginx errors with docker compose logs nginx. Outside Docker, inspect your Nginx error log.

Next steps

Set up alerts for errors or latency thresholds.
Create additional dashboards.
For a collector between Nginx and your destination, see Sending OpenTelemetry data.

Introduction

Features

Administration

Guides

Integration partners

Self-managed

Monitoring Nginx traces with ClickHouse Observability

Prerequisites

Connection details

Integration with existing Nginx

Demo dataset

Dashboards and visualization

Troubleshooting

Nginx fails to start

No traces appearing

Next steps

​Prerequisites

​Connection details

​Integration with existing Nginx

​Demo dataset

​Dashboards and visualization

​Troubleshooting

​Nginx fails to start

​No traces appearing

​Next steps

Prerequisites

Connection details

Integration with existing Nginx

Demo dataset

Dashboards and visualization

Troubleshooting

Nginx fails to start

No traces appearing

Next steps