> ## Documentation Index > Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring Redis logs with ClickHouse Observability > Collect, search, and visualize Redis server logs in ClickHouse Observability using an OpenTelemetry collector. Collect Redis server logs with an OpenTelemetry collector and send them to ClickHouse Observability. Use the [demo dataset](#demo-dataset) to try the integration without installing Redis. ## Prerequisites * A ClickHouse Observability service * Docker to run the collector examples, or an existing OpenTelemetry Collector Contrib deployment * For your own logs, an existing Redis installation and access to its log files ### Connection details > [tab] Fully managed > > [Beta offering](https://clickhouse.com/docs/reference/settings/beta-and-experimental-features.md#beta-features) > > Open ClickHouse Observability and go to **Team Settings → Data → Managed Telemetry**. Copy your OTLP HTTP endpoint and API key. You can use **Copy connection details** to copy the configuration, or **Reveal key** to view the masked key. > > ![Managed Telemetry connection details](https://mintcdn.com/private-7c7dfe99/Zk1feNn6fCQBpold/images/clickstack/getting-started/observability-managed-telemetry.png?fit=max\&auto=format\&n=Zk1feNn6fCQBpold\&q=85\&s=e63a2ac2e6ab996d3fc360ebe0f487a5) > > For the OTLP exporter examples below, set these variables in the terminal where you will run the collector: > > ```bash theme={null} > export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_ENDPOINT:4318' > export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY' > ``` > > Replace the placeholders with the exact endpoint, including its port, and API key from your service. Use the base HTTP endpoint without a `/v1/logs` suffix; the exporter appends the signal path. > [tab] Self-managed > > Use your collector's reachable OTLP HTTP endpoint (for example, `http://YOUR_COLLECTOR_HOST:4318`) and authentication settings. If your collector does not require authentication, omit the exporter's `headers` block from the configurations below. See the [self-managed getting-started guide](https://clickhouse.com/docs/clickstack/self-configured/getting-started/managed.md) for setup guidance. > > ```bash theme={null} > export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318' > export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY' > ``` The endpoint must be reachable from the collector container. For a destination collector running on this computer, use `http://host.docker.internal:4318`. The Docker commands below include the host mapping. ## Integration with existing Redis **1. Verify Redis logging configuration** Check where Redis writes its logs: ```bash theme={null} redis-cli CONFIG GET logfile ``` Common Redis log locations: * **Linux (apt/yum)**: `/var/log/redis/redis-server.log` * **macOS (Homebrew)**: `/usr/local/var/log/redis.log` or `/opt/homebrew/var/log/redis.log` * **Docker**: Often logged to stdout, but can be configured to write to `/data/redis.log` If Redis is logging to stdout, configure it to write to a file by updating `redis.conf`: ```bash theme={null} # Log to file instead of stdout logfile /var/log/redis/redis-server.log # Set log level (options: debug, verbose, notice, warning) loglevel notice ``` Ensure the log directory exists and Redis can write to it. After changing the configuration, restart Redis using your installation’s service name: ```bash theme={null} # For systemd sudo systemctl restart redis # For Docker docker restart ``` **2. Configure the collector** Save this as `redis-monitoring.yaml`: ```yaml theme={null} receivers: filelog/redis: include: - /var/log/redis/redis-server.log start_at: end operators: - type: regex_parser regex: '^(?P\d+):(?P\w+) (?P\d{2} \w+ \d{4} \d{2}:\d{2}:\d{2}\.\d+) (?P[.\-*#]) (?P.*)$' parse_from: body parse_to: attributes - type: time_parser parse_from: attributes.timestamp layout_type: gotime layout: '02 Jan 2006 15:04:05.000' location: UTC - type: add field: attributes.source value: redis - type: add field: resource["service.name"] value: redis-service processors: memory_limiter: check_interval: 1s limit_mib: 256 spike_limit_mib: 64 transform/redis: error_mode: propagate log_statements: - set(log.severity_text, "debug") where log.attributes["log_level"] == "." or log.attributes["log_level"] == "-" - set(log.severity_number, SEVERITY_NUMBER_DEBUG) where log.severity_text == "debug" - set(log.severity_text, "info") where log.attributes["log_level"] == "*" - set(log.severity_number, SEVERITY_NUMBER_INFO) where log.severity_text == "info" - set(log.severity_text, "warn") where log.attributes["log_level"] == "#" - set(log.severity_number, SEVERITY_NUMBER_WARN) where log.severity_text == "warn" batch: timeout: 5s send_batch_size: 1024 exporters: otlp_http/observability: endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT} headers: authorization: "${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}" service: pipelines: logs/redis: receivers: [filelog/redis] processors: [memory_limiter, transform/redis, batch] exporters: [otlp_http/observability] ``` This configuration parses the process ID, role, timestamp, log level, and message. It sets `service.name` to `redis-service` and maps Redis log levels to `SeverityText` for the dashboard. Redis notice messages (`*`) map to `info`, warnings (`#`) to `warn`, and debug/verbose messages (`.` and `-`) to `debug`. Redis log timestamps do not include a timezone. Set `location` to your Redis server's timezone, such as `America/Chicago`; the example uses `UTC`. Original timestamps and the log body are preserved. With `start_at: end`, only new entries are read when the collector first opens the file. **3. Run the collector** Set the [connection variables](#connection-details), then run from the directory containing `redis-monitoring.yaml`. Adjust the host log-directory path and the receiver's `include` filename to match your installation: ```bash theme={null} docker run --rm --name redis-monitoring-collector \ --add-host=host.docker.internal:host-gateway \ --memory=512m \ -e OTEL_EXPORTER_OTLP_ENDPOINT \ -e OTEL_EXPORTER_OTLP_AUTHORIZATION \ -v "$(pwd)/redis-monitoring.yaml:/etc/otelcol-contrib/config.yaml:ro" \ -v /var/log/redis:/var/log/redis:ro \ otel/opentelemetry-collector-contrib:0.161.0 \ --config=/etc/otelcol-contrib/config.yaml ``` Keep this terminal open and generate Redis activity that writes new log entries. The collector must be able to read the mounted files. **4. Verify logs** Open **Search**, select **Logs**, and filter for `ServiceName:redis-service`. Select a time range covering your new logs. Expand an entry to inspect `SeverityText` and the parsed fields under `LogAttributes`. ![Redis logs in the search view](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-redis-demo-logs.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=b85108fa65fdb9c7108d10b98486e5ff) ## Demo dataset The sample contains 10,000 Redis server log entries. Set the [connection variables](#connection-details) before starting. **1. Download the sample dataset** ```bash theme={null} curl -fLO https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/redis/redis-server.log ``` **2. Configure the demo collector** Save this as `redis-demo.yaml` in the download directory: ```yaml theme={null} receivers: filelog/redis: include: - /var/log/redis-demo/redis-server.log start_at: beginning operators: - type: regex_parser regex: '^(?P\d+):(?P\w+) (?P\d{2} \w+ \d{4} \d{2}:\d{2}:\d{2}\.\d+) (?P[.\-*#]) (?P.*)$' parse_from: body parse_to: attributes - type: time_parser parse_from: attributes.timestamp layout_type: gotime layout: '02 Jan 2006 15:04:05.000' location: UTC - type: add field: attributes.source value: redis-demo - type: add field: resource["service.name"] value: redis-demo processors: memory_limiter: check_interval: 1s limit_mib: 256 spike_limit_mib: 64 transform/redis: error_mode: propagate log_statements: - set(log.severity_text, "debug") where log.attributes["log_level"] == "." or log.attributes["log_level"] == "-" - set(log.severity_number, SEVERITY_NUMBER_DEBUG) where log.severity_text == "debug" - set(log.severity_text, "info") where log.attributes["log_level"] == "*" - set(log.severity_number, SEVERITY_NUMBER_INFO) where log.severity_text == "info" - set(log.severity_text, "warn") where log.attributes["log_level"] == "#" - set(log.severity_number, SEVERITY_NUMBER_WARN) where log.severity_text == "warn" transform/demo_timestamps: error_mode: propagate log_statements: - >- set(log.time, Now() - Duration("2m") - (Time("2025-10-28T14:59:22Z", "%Y-%m-%dT%H:%M:%SZ") - log.time) ) batch: timeout: 5s send_batch_size: 1024 exporters: otlp_http/observability: endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT} headers: authorization: "${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}" service: pipelines: logs/redis: receivers: [filelog/redis] processors: [memory_limiter, transform/redis, transform/demo_timestamps, batch] exporters: [otlp_http/observability] ``` > [!NOTE] > > The demo-only `transform/demo_timestamps` processor shifts the sample into roughly the past 24 hours, ending two minutes before ingestion, while retaining its traffic pattern. The original timestamp attribute and log body stay unchanged. Omit this processor for your own Redis logs. **3. Run the demo collector** From the directory containing `redis-server.log` and `redis-demo.yaml`, run: ```bash theme={null} docker run --rm --name redis-demo-collector \ --add-host=host.docker.internal:host-gateway \ --memory=512m \ -e OTEL_EXPORTER_OTLP_ENDPOINT \ -e OTEL_EXPORTER_OTLP_AUTHORIZATION \ -v "$(pwd)/redis-demo.yaml:/etc/otelcol-contrib/config.yaml:ro" \ -v "$(pwd)/redis-server.log:/var/log/redis-demo/redis-server.log:ro" \ otel/opentelemetry-collector-contrib:0.161.0 \ --config=/etc/otelcol-contrib/config.yaml ``` Leave the terminal open while you verify ingestion. Stop the collector with **Ctrl+C** when finished. Running the demo again re-ingests the sample with refreshed timestamps because this example does not persist file offsets. **4. Verify the demo logs** Open **Search**, select **Logs**, filter for `ServiceName:redis-demo`, and select **Last 2 days**. Expand a log entry to inspect the parsed attributes and severity. ![Example Redis logs in the search view](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-redis-demo-logs.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=b85108fa65fdb9c7108d10b98486e5ff) ## Dashboards and visualization **1. [Download the dashboard configuration](https://clickhouse-docs-assets.s3.us-east-1.amazonaws.com/examples/redis-logs-dashboard.json)** **2. Import the dashboard** 1. Open **Dashboards** and click **Import** in the upper-right corner. 2. Upload `redis-logs-dashboard.json`. 3. Map the charts to your **Logs** source and click **Finish Import**. ![Dashboards page with the Import button highlighted](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-import-dashboard.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=16bcb7e6365ca0f6decac5c693e228eb) **3. View the dashboard** Filter for `ServiceName:redis-demo` for the sample, or `ServiceName:redis-service` for your own logs. For the demo, select **Last 2 days**. The dashboard shows log volume and severity. Its **Request rate** chart counts log entries, not Redis commands. ![Example Redis logs dashboard](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-redis-logs-dashboard.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=9d17ab762e46651d0d0c1d5f4b98b19e) ## Troubleshooting ### Collector fails to start Check the collector terminal for configuration errors. Verify the YAML file is mounted at the documented path and the endpoint and any required authentication variables are set in the terminal where you run Docker. ### No logs appearing * Check the collector output for authentication or export errors and verify your connection details. * Run `redis-cli CONFIG GET logfile` to check that Redis writes to a file. Ensure the collector can read it and that its mounted path matches the receiver's `include` path. * With `start_at: end`, generate new log entries after starting the collector. * Check the **Logs** source, service-name filter, and time range. For the demo, include `transform/demo_timestamps` in the pipeline and select **Last 2 days**. ### Logs not parsing correctly The receiver expects Redis's standard `pid:role timestamp level message` format. Check your log file and adjust the regex if your format differs. Set the time parser's `location` to the Redis server's timezone. ## Next steps * Set up [alerts](https://clickhouse.com/docs/clickstack/observability/features/alerts.md) for warnings or unusual log volume. * Create additional [dashboards](https://clickhouse.com/docs/clickstack/observability/features/dashboards/overview.md). * See [Sending OpenTelemetry data](https://clickhouse.com/docs/clickstack/observability/ingesting-data/opentelemetry.md) for collector configuration guidance.