> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring Systemd logs with ClickHouse Observability

> Collect systemd journal logs and send them to ClickHouse Observability with OpenTelemetry.

Collect systemd journal logs with the OpenTelemetry journald receiver and send them to ClickHouse Observability. Use the [demo dataset](#demo-dataset) to try the integration without a Linux host.

## Prerequisites

* A ClickHouse Observability service
* For your own logs, a Linux host running systemd, with `journalctl` and permission to read its journal
* `curl`, and Docker for the demo

### Connection details

> [tab] Fully managed
>
> [Beta offering](https://clickhouse.com/docs/reference/settings/beta-and-experimental-features.md#beta-features)
>
> Open ClickHouse Observability and go to **Team Settings → Data → Managed Telemetry**. Copy your OTLP HTTP endpoint and API key. You can use **Copy connection details** to copy the configuration, or **Reveal key** to view the masked key.
>
> ![Managed Telemetry connection details](https://mintcdn.com/private-7c7dfe99/Zk1feNn6fCQBpold/images/clickstack/getting-started/observability-managed-telemetry.png?fit=max\&auto=format\&n=Zk1feNn6fCQBpold\&q=85\&s=e63a2ac2e6ab996d3fc360ebe0f487a5)
>
> For the OTLP exporter examples below, set these variables in the terminal where you will run the collector:
>
> ```bash theme={null}
> export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_ENDPOINT:4318'
> export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY'
> ```
>
> Replace the placeholders with the exact endpoint, including its port, and API key from your service. Use the base HTTP endpoint without a `/v1/logs` suffix; the exporter appends the signal path.

> [tab] Self-managed
>
> Use your collector's reachable OTLP HTTP endpoint (for example, `http://YOUR_COLLECTOR_HOST:4318`) and authentication settings. If your collector does not require authentication, omit the exporter's `headers` block from the configurations below. See the [self-managed getting-started guide](https://clickhouse.com/docs/clickstack/self-configured/getting-started/managed.md) for setup guidance.
>
> ```bash theme={null}
> export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318'
> export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY'
> ```

The endpoint must be reachable from the monitored host or demo collector. For a destination collector running on the Docker host, use `http://host.docker.internal:4318`.

## Integration with existing systems

**1. Verify the journal**

On the Linux host you want to monitor:

```bash theme={null}
systemctl --version
sudo journalctl -n 20
sudo journalctl --disk-usage
```

The receiver uses the host's `journalctl` to read available journals. Both persistent and volatile journal storage are supported; you do not need to change storage settings.

**2. Download the collector**

Run these commands on the monitored Linux host:

```bash theme={null}
case "$(uname -m)" in
  x86_64) arch=amd64 ;;
  aarch64|arm64) arch=arm64 ;;
  *) echo "Unsupported architecture"; exit 1 ;;
esac
curl -fL -o otelcol-contrib.tar.gz \
  "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.161.0/otelcol-contrib_0.161.0_linux_${arch}.tar.gz"
tar -xzf otelcol-contrib.tar.gz otelcol-contrib
```

The collector runs directly on the host so it can use the installed `journalctl`. The standard Contrib Docker image does not include that binary.

**3. Configure the collector**

Save this as `systemd-monitoring.yaml`:

```yaml theme={null}
receivers:
  journald:
    start_at: end
    priority: info
    convert_message_bytes: true
processors:
  transform/journald:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - merge_maps(log.attributes, log.body, "upsert")
          - set(resource.attributes["host.name"], log.attributes["_HOSTNAME"])
          - set(resource.attributes["service.name"], "systemd-logs")
          - set(log.attributes["unit"], log.attributes["SYSLOG_IDENTIFIER"])
          - set(log.attributes["unit"], log.attributes["_SYSTEMD_UNIT"]) where log.attributes["_SYSTEMD_UNIT"] != nil
          - set(log.attributes["priority"], log.attributes["PRIORITY"])
          - set(log.body, log.attributes["MESSAGE"])
          - set(log.severity_text, "info")
          - set(log.severity_number, SEVERITY_NUMBER_INFO)
          - set(log.severity_text, "warn") where Int(log.attributes["PRIORITY"]) == 4
          - set(log.severity_number, SEVERITY_NUMBER_WARN) where log.severity_text == "warn"
          - set(log.severity_text, "error") where Int(log.attributes["PRIORITY"]) == 3
          - set(log.severity_number, SEVERITY_NUMBER_ERROR) where log.severity_text == "error"
          - set(log.severity_text, "fatal") where Int(log.attributes["PRIORITY"]) < 3
          - set(log.severity_number, SEVERITY_NUMBER_FATAL) where log.severity_text == "fatal"
  batch:
    timeout: 5s
    send_batch_size: 500
    send_batch_max_size: 500
exporters:
  otlp_http/observability:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
service:
  pipelines:
    logs:
      receivers: [journald]
      processors: [transform/journald, batch]
      exporters: [otlp_http/observability]
```

Journal fields are retained as log attributes, the message becomes the log body, and the hostname becomes a resource attribute. The receiver collects all units at priority `info` or higher. To restrict collection, add a `units` list under `journald` using actual unit names from your host, such as `ssh.service` or `nginx.service`.

**4. Run the collector**

Set the [connection variables](#connection-details), then run from the download directory. This command uses administrator permissions to read the system journal and preserves only the two connection variables:

```bash theme={null}
sudo --preserve-env=OTEL_EXPORTER_OTLP_ENDPOINT,OTEL_EXPORTER_OTLP_AUTHORIZATION \
  ./otelcol-contrib --config=systemd-monitoring.yaml
```

Keep the terminal open. In another terminal, generate a journal entry:

```bash theme={null}
echo 'ClickHouse Observability journal ingestion test' | systemd-cat -t observability-test -p info
```

Stop the collector with **Ctrl+C** when finished testing. For ongoing collection, run it as a systemd service with journal access and configure [cursor storage](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.161.0/receiver/journaldreceiver#cursor-tracking) to resume after restarts.

**5. Verify logs**

Open **Search**, select **Logs**, filter for `ServiceName:systemd-logs`, and select **Last 15 minutes**. Look for the test message and attributes such as `unit` and `priority`.

## Demo dataset

The sample contains 1,334 text log entries spanning approximately 48 hours, including SSH activity, scheduled jobs, and service events. It uses the `filelog` receiver, so no systemd or `journalctl` installation is needed. The collector refreshes timestamps while preserving the original bodies and spacing between events.

**1. Download the sample**

```bash theme={null}
curl -fL -o systemd-demo.log \
  https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/systemd/systemd-demo.log
```

**2. Configure the demo collector**

Save this as `systemd-demo.yaml`:

```yaml theme={null}
receivers:
  filelog/systemd:
    include:
      - /var/log/demo/systemd-demo.log
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%Y-%m-%dT%H:%M:%S%z'
      - type: add
        field: attributes.source
        value: systemd-demo
      - type: add
        field: resource["service.name"]
        value: systemd-demo
processors:
  transform/demo_timestamps:
    error_mode: propagate
    log_statements:
      - context: log
        statements:
          - set(resource.attributes["host.name"], log.attributes["hostname"])
          - set(log.time, Now() - Duration("2m") - (Time("2025-11-16T23:45:00Z", "%Y-%m-%dT%H:%M:%SZ") - log.time))
  batch:
    timeout: 5s
    send_batch_size: 500
    send_batch_max_size: 500
exporters:
  otlp_http/observability:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
service:
  pipelines:
    logs:
      receivers: [filelog/systemd]
      processors: [transform/demo_timestamps, batch]
      exporters: [otlp_http/observability]
```

The text sample has no journal priority field, so the demo does not assign severity from the message text.

**3. Run the demo collector**

Set the [connection variables](#connection-details), then run from the directory containing both files:

```bash theme={null}
docker run --rm --name systemd-demo-collector \
  --add-host=host.docker.internal:host-gateway \
  --memory=512m \
  -e OTEL_EXPORTER_OTLP_ENDPOINT \
  -e OTEL_EXPORTER_OTLP_AUTHORIZATION \
  -v "$(pwd)/systemd-demo.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  -v "$(pwd)/systemd-demo.log:/var/log/demo/systemd-demo.log:ro" \
  otel/opentelemetry-collector-contrib:0.161.0 \
  --config=/etc/otelcol-contrib/config.yaml
```

Leave the terminal open while verifying ingestion. Stop with **Ctrl+C** when finished. Running the demo again re-ingests the sample with refreshed timestamps.

**4. Verify demo logs**

Open **Search**, select **Logs**, filter for `ServiceName:systemd-demo`, and select **Last 2 days**. You should see the sample entries with `unit`, `hostname`, and `message` attributes.

![Systemd demo logs in ClickHouse Observability Search](https://mintcdn.com/private-7c7dfe99/FqnhjwVWZILRZzw7/images/clickstack/observability-systemd-demo-logs.png?fit=max\&auto=format\&n=FqnhjwVWZILRZzw7\&q=85\&s=5439882a6d579dc3b9dc3fd0e153fdb9)

## Dashboards and visualization

**1. [Download the dashboard configuration](https://clickhouse-docs-assets.s3.us-east-1.amazonaws.com/examples/systemd-logs-dashboard.json)**

**2. Import the dashboard**

1. Open **Dashboards** and click **Import** in the upper-right corner.
2. Upload `systemd-logs-dashboard.json`.
3. Map the charts to your **Logs** source and click **Finish Import**.

![Import dashboard button](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-import-dashboard.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=16bcb7e6365ca0f6decac5c693e228eb)

**3. View the dashboard**

Select **Last 2 days** for the sample and optionally filter for `ServiceName:systemd-demo`. For live logs, use `ServiceName:systemd-logs` and a time range covering your records. The dashboard includes log volume, units, SSH events, and service failures.

![Systemd logs dashboard in ClickHouse Observability](https://mintcdn.com/private-7c7dfe99/FqnhjwVWZILRZzw7/images/clickstack/observability-systemd-logs-dashboard.png?fit=max\&auto=format\&n=FqnhjwVWZILRZzw7\&q=85\&s=836abf3ccc4d4dfd7b4650c88c512ef5)

For live journal logs, change the SSH authentication chart's `Body LIKE '%sshd%'` condition to `LogAttributes['SYSLOG_IDENTIFIER'] = 'sshd'`. The live receiver stores the message in `Body` and the process identifier separately.

## Troubleshooting

### journalctl not found

Run the live collector on the monitored Linux host with `journalctl` installed and available in its execution path. The demo uses `filelog` and does not require it.

### No logs appearing

Check the collector output for journal access, authentication, or export errors. Confirm `sudo journalctl -n 20` returns entries on the monitored host. If you configured a unit filter, confirm it matches your service names. Generate a new entry after starting collection.

In ClickHouse Observability, check the **Logs** source, service-name filter, and time range. For the demo, select **Last 2 days** and confirm the sample file is mounted at the configured path.

## Next steps

* Set up [alerts](https://clickhouse.com/docs/clickstack/observability/features/alerts.md) for service failures or authentication failures.
* Create additional [dashboards](https://clickhouse.com/docs/clickstack/observability/features/dashboards/overview.md) for your services.
* See [Sending OpenTelemetry data](https://clickhouse.com/docs/clickstack/observability/ingesting-data/opentelemetry.md) for collector deployment guidance.