> ## Documentation Index > Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring Nginx traces with ClickHouse Observability > Collect, search, and visualize Nginx traces in ClickHouse Observability with managed or self-managed ingestion. Capture distributed traces from Nginx with the OpenTelemetry Nginx module and send them to ClickHouse Observability. Use the [demo dataset](#demo-dataset) to try the integration without installing Nginx. ## Prerequisites * A ClickHouse Observability service * For your own traces, an Nginx installation with the OpenTelemetry module and permission to update its configuration; the Docker example below uses `nginx:1.27-otel` * For the demo dataset, `curl` and Docker ### Connection details > [tab] Fully managed > > [Beta offering](https://clickhouse.com/docs/reference/settings/beta-and-experimental-features.md#beta-features) > > Open ClickHouse Observability and go to **Team Settings → Data → Managed Telemetry**. Copy your endpoint and API key using **Copy connection details**, or select **Reveal key** to view the masked key. > > ![Managed Telemetry connection details](https://mintcdn.com/private-7c7dfe99/Zk1feNn6fCQBpold/images/clickstack/getting-started/observability-managed-telemetry.png?fit=max\&auto=format\&n=Zk1feNn6fCQBpold\&q=85\&s=e63a2ac2e6ab996d3fc360ebe0f487a5) > > The Nginx module exports over **OTLP/gRPC**, normally on port **4317**. The JSON demo uses **OTLP/HTTP**, normally on port **4318**. Use the endpoint for the matching protocol, including `https://` when TLS is required: > > ```bash theme={null} > export OTEL_EXPORTER_OTLP_GRPC_ENDPOINT='https://YOUR_OTEL_HOST:4317' > export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_HOST:4318' > export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY' > ``` > > Replace the placeholders with your service's connection details. Use the base HTTP endpoint without `/v1/traces`; the collector exporter adds that path. > [tab] Self-managed > > Use your collector's OTLP endpoints and authentication settings. If your collector does not require authentication, omit the Nginx `header authorization` directive and the demo collector's `headers` block. See the [self-managed getting-started guide](https://clickhouse.com/docs/clickstack/self-configured/getting-started/managed.md). > > ```bash theme={null} > export OTEL_EXPORTER_OTLP_GRPC_ENDPOINT='http://YOUR_COLLECTOR_HOST:4317' > export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318' > export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY' > ``` The gRPC endpoint must be reachable from Nginx. With Docker Desktop, use `host.docker.internal` to reach a collector on the host. For containers on a shared Docker network, use the collector's container name. Use `http://` for a local endpoint without TLS. ## Integration with existing Nginx **1. Install the OpenTelemetry Nginx module** The `nginx:1.27-otel` image includes `ngx_otel_module.so`. For an installation outside Docker, follow the installation instructions linked from the [Nginx OpenTelemetry module documentation](https://nginx.org/en/docs/ngx_otel_module.html). **2. Configure Nginx tracing** The following Docker example uses a template so the container's startup script substitutes the connection environment variables. Save it as `nginx.conf.template`. The `return 200` location provides a simple smoke test; for your application, retain your existing server and routing configuration and add the tracing directives shown here. ```nginx theme={null} load_module modules/ngx_otel_module.so; events { worker_connections 1024; } http { otel_exporter { endpoint ${OTEL_EXPORTER_OTLP_GRPC_ENDPOINT}; header authorization "${OTEL_EXPORTER_OTLP_AUTHORIZATION}"; } otel_service_name "nginx-proxy"; otel_trace on; server { listen 80; location / { otel_trace_context propagate; otel_span_name "$request_method $uri"; otel_span_attr http.status_code $status; otel_span_attr http.request.method $request_method; otel_span_attr http.route $uri; return 200 "Nginx tracing is enabled\n"; } } } ``` Save this as `compose.yaml` in the same directory: ```yaml theme={null} services: nginx: image: nginx:1.27-otel ports: - "8081:80" environment: OTEL_EXPORTER_OTLP_GRPC_ENDPOINT: ${OTEL_EXPORTER_OTLP_GRPC_ENDPOINT} OTEL_EXPORTER_OTLP_AUTHORIZATION: ${OTEL_EXPORTER_OTLP_AUTHORIZATION} NGINX_ENVSUBST_OUTPUT_DIR: /etc/nginx NGINX_ENVSUBST_FILTER: ^OTEL_EXPORTER_OTLP_ volumes: - ./nginx.conf.template:/etc/nginx/templates/nginx.conf.template:ro ``` The substitution filter preserves Nginx variables such as `$request_method` and `$uri`. Nginx does not automatically substitute shell environment variables in a directly mounted `nginx.conf`. Outside Docker, replace the connection placeholders with their actual values before validating the configuration, and restrict access to the file containing the key. **3. Validate and start Nginx** Set the [connection variables](#connection-details) in the same terminal, then run: ```bash theme={null} docker compose run --rm nginx nginx -t docker compose up -d nginx curl http://localhost:8081/ ``` Each request creates a span with the method, path, status code, and duration. For an existing deployment, generate requests to your application's URL. After changing the template or environment variables, run `docker compose up -d --force-recreate nginx` to apply them. **4. Verify traces in the UI** Open **Search**, select **Traces**, and filter for `ServiceName:nginx-proxy`. Select a time range covering your requests and allow a few seconds for export. Open a trace to inspect its spans and HTTP attributes. ![Nginx traces in the search view](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-nginx-demo-traces.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=69b26136229b973f382c74f25c8b4c5e) ## Demo dataset The sample contains 1,000 Nginx spans across nine endpoints, with successful requests, HTTP errors, and varied latencies. Set the [connection variables](#connection-details) before sending it. **1. Download the sample dataset** ```bash theme={null} curl -fLO https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/nginx-traces-sample.json ``` **2. Configure the demo collector** Save the following as `nginx-traces-demo.yaml` in the download directory: ```yaml theme={null} receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 processors: memory_limiter: check_interval: 1s limit_mib: 256 spike_limit_mib: 64 transform/demo_timestamps: error_mode: propagate trace_statements: - context: span statements: - set(cache["duration_ns"], end_time_unix_nano - start_time_unix_nano) - >- set(start_time, Now() - Duration("2m") - (Time("2025-10-27T19:36:05Z", "%Y-%m-%dT%H:%M:%SZ") - start_time) ) - set(end_time_unix_nano, start_time_unix_nano + cache["duration_ns"]) batch: timeout: 5s send_batch_size: 1024 exporters: otlp_http/observability: endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT} headers: authorization: "${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}" service: pipelines: traces/nginx-demo: receivers: [otlp] processors: [memory_limiter, transform/demo_timestamps, batch] exporters: [otlp_http/observability] ``` > [!NOTE] > > The demo-only `transform/demo_timestamps` processor shifts this sample into roughly the past 24 hours, ending two minutes before ingestion, while preserving span durations and the traffic pattern. Omit this processor for your own Nginx traces so their original timestamps are preserved. Sending the sample again can produce duplicate spans. **3. Run the demo collector** > [!NOTE] > > For a destination collector running on this computer, set `OTEL_EXPORTER_OTLP_ENDPOINT` to `http://host.docker.internal:4318`. The command below includes the Docker host mapping. Run OpenTelemetry Collector Contrib `0.161.0` from the directory containing the configuration, in the terminal where you set the connection variables. Keep this terminal open: ```bash theme={null} docker run --rm --name nginx-traces-demo-collector \ --add-host=host.docker.internal:host-gateway \ --memory=512m \ -p 127.0.0.1:14318:4318 \ -e OTEL_EXPORTER_OTLP_ENDPOINT \ -e OTEL_EXPORTER_OTLP_AUTHORIZATION \ -v "$(pwd)/nginx-traces-demo.yaml:/etc/otelcol-contrib/config.yaml:ro" \ otel/opentelemetry-collector-contrib:0.161.0 \ --config=/etc/otelcol-contrib/config.yaml ``` The local receiver uses port `14318` to avoid conflicting with an existing collector on port `4318`. The collector forwards the traces to your configured endpoint using the authentication settings above. **4. Send the traces** In another terminal, from the download directory, send the original sample to the local collector: ```bash theme={null} curl --fail-with-body -sS http://localhost:14318/v1/traces \ -H "Content-Type: application/json" \ --data-binary @nginx-traces-sample.json ``` An empty success response such as `{}` or `{"partialSuccess":{}}` indicates the local collector accepted the request. Check the collector's output for export errors and allow a few seconds for the batch to reach your service. Stop the demo collector with **Ctrl+C** when finished. **5. Verify the demo traces** Open **Search**, select **Traces**, filter for `ServiceName:nginx-proxy`, and select **Last 2 days** to include the full shifted dataset. Open a trace to inspect the HTTP attributes. ![Example Nginx traces in the search view](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-nginx-demo-traces.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=69b26136229b973f382c74f25c8b4c5e) ## Dashboards and visualization **1. Download the dashboard configuration** [Download the dashboard configuration file](https://clickhouse-docs-assets.s3.us-east-1.amazonaws.com/examples/nginx-traces-dashboard.json). **2. Import the dashboard** 1. Open **Dashboards** and click **Import** in the upper-right corner. 2. Upload `nginx-traces-dashboard.json`. 3. Map the charts to your **Traces** source and click **Finish Import**. ![Dashboards page with the Import button highlighted](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-import-dashboard.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=16bcb7e6365ca0f6decac5c693e228eb) **3. View the dashboard** Filter for `ServiceName:nginx-proxy` and select a time range covering your traces. For the demo, select **Last 2 days**. ![Example Nginx traces dashboard](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-nginx-traces-dashboard.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=89c69f3c307067d67d7411405528b859) ## Troubleshooting ### Nginx fails to start Run `docker compose run --rm nginx nginx -t` to validate the generated configuration. An unknown `otel_*` directive or missing module file means the module is not installed or loaded. Check that you are using the `-otel` image and that the template is mounted at the documented path. ### No traces appearing * Verify the gRPC endpoint and any authentication required by your destination collector. Managed services use the `Bearer` prefix. Do not use the HTTP endpoint for the Nginx module. * Confirm the endpoint is reachable from the Nginx container and that its scheme matches the destination's TLS configuration. * Generate requests; an idle Nginx instance produces no request spans. * Check **Traces**, the service-name filter, and the selected time range. For the demo, confirm `transform/demo_timestamps` is included in the collector pipeline and select **Last 2 days**. * For the demo, check the collector terminal for authentication or export errors. Verify its HTTP endpoint and authorization value against your connection details. * Inspect Nginx errors with `docker compose logs nginx`. Outside Docker, inspect your Nginx error log. ## Next steps * Set up [alerts](https://clickhouse.com/docs/clickstack/observability/features/alerts.md) for errors or latency thresholds. * Create additional [dashboards](https://clickhouse.com/docs/clickstack/observability/features/dashboards/overview.md). * For a collector between Nginx and your destination, see [Sending OpenTelemetry data](https://clickhouse.com/docs/clickstack/observability/ingesting-data/opentelemetry.md).