Monitoring Systemd logs with ClickHouse Observability

Collect systemd journal logs with the OpenTelemetry journald receiver and send them to ClickHouse Observability. Use the demo dataset to try the integration without a Linux host.

Prerequisites

A ClickHouse Observability service
For your own logs, a Linux host running systemd, with journalctl and permission to read its journal
curl, and Docker for the demo

Connection details

Fully managed
Self-managed

Open ClickHouse Observability and go to Team Settings → Data → Managed Telemetry. Copy your OTLP HTTP endpoint and API key. You can use Copy connection details to copy the configuration, or Reveal key to view the masked key.For the OTLP exporter examples below, set these variables in the terminal where you will run the collector:

export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_ENDPOINT:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY'

Replace the placeholders with the exact endpoint, including its port, and API key from your service. Use the base HTTP endpoint without a /v1/logs suffix; the exporter appends the signal path.

Use your collector’s reachable OTLP HTTP endpoint (for example, http://YOUR_COLLECTOR_HOST:4318) and authentication settings. If your collector does not require authentication, omit the exporter’s headers block from the configurations below. See the self-managed getting-started guide for setup guidance.

export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318'
export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY'

The endpoint must be reachable from the monitored host or demo collector. For a destination collector running on the Docker host, use http://host.docker.internal:4318.

Integration with existing systems

Verify the journal

On the Linux host you want to monitor:

systemctl --version
sudo journalctl -n 20
sudo journalctl --disk-usage

The receiver uses the host’s journalctl to read available journals. Both persistent and volatile journal storage are supported; you do not need to change storage settings.

Download the collector

Run these commands on the monitored Linux host:

case "$(uname -m)" in
  x86_64) arch=amd64 ;;
  aarch64|arm64) arch=arm64 ;;
  *) echo "Unsupported architecture"; exit 1 ;;
esac
curl -fL -o otelcol-contrib.tar.gz \
  "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.161.0/otelcol-contrib_0.161.0_linux_${arch}.tar.gz"
tar -xzf otelcol-contrib.tar.gz otelcol-contrib

The collector runs directly on the host so it can use the installed journalctl. The standard Contrib Docker image does not include that binary.

Configure the collector

Save this as systemd-monitoring.yaml:

receivers:
  journald:
    start_at: end
    priority: info
    convert_message_bytes: true
processors:
  transform/journald:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - merge_maps(log.attributes, log.body, "upsert")
          - set(resource.attributes["host.name"], log.attributes["_HOSTNAME"])
          - set(resource.attributes["service.name"], "systemd-logs")
          - set(log.attributes["unit"], log.attributes["SYSLOG_IDENTIFIER"])
          - set(log.attributes["unit"], log.attributes["_SYSTEMD_UNIT"]) where log.attributes["_SYSTEMD_UNIT"] != nil
          - set(log.attributes["priority"], log.attributes["PRIORITY"])
          - set(log.body, log.attributes["MESSAGE"])
          - set(log.severity_text, "info")
          - set(log.severity_number, SEVERITY_NUMBER_INFO)
          - set(log.severity_text, "warn") where Int(log.attributes["PRIORITY"]) == 4
          - set(log.severity_number, SEVERITY_NUMBER_WARN) where log.severity_text == "warn"
          - set(log.severity_text, "error") where Int(log.attributes["PRIORITY"]) == 3
          - set(log.severity_number, SEVERITY_NUMBER_ERROR) where log.severity_text == "error"
          - set(log.severity_text, "fatal") where Int(log.attributes["PRIORITY"]) < 3
          - set(log.severity_number, SEVERITY_NUMBER_FATAL) where log.severity_text == "fatal"
  batch:
    timeout: 5s
    send_batch_size: 500
    send_batch_max_size: 500
exporters:
  otlp_http/observability:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
service:
  pipelines:
    logs:
      receivers: [journald]
      processors: [transform/journald, batch]
      exporters: [otlp_http/observability]

Journal fields are retained as log attributes, the message becomes the log body, and the hostname becomes a resource attribute. The receiver collects all units at priority info or higher. To restrict collection, add a units list under journald using actual unit names from your host, such as ssh.service or nginx.service.

Run the collector

Set the connection variables, then run from the download directory. This command uses administrator permissions to read the system journal and preserves only the two connection variables:

sudo --preserve-env=OTEL_EXPORTER_OTLP_ENDPOINT,OTEL_EXPORTER_OTLP_AUTHORIZATION \
  ./otelcol-contrib --config=systemd-monitoring.yaml

Keep the terminal open. In another terminal, generate a journal entry:

echo 'ClickHouse Observability journal ingestion test' | systemd-cat -t observability-test -p info

Stop the collector with Ctrl+C when finished testing. For ongoing collection, run it as a systemd service with journal access and configure cursor storage to resume after restarts.

Verify logs

Open Search, select Logs, filter for ServiceName:systemd-logs, and select Last 15 minutes. Look for the test message and attributes such as unit and priority.

Demo dataset

The sample contains 1,334 text log entries spanning approximately 48 hours, including SSH activity, scheduled jobs, and service events. It uses the filelog receiver, so no systemd or journalctl installation is needed. The collector refreshes timestamps while preserving the original bodies and spacing between events.

Download the sample

curl -fL -o systemd-demo.log \
  https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/systemd/systemd-demo.log

Configure the demo collector

Save this as systemd-demo.yaml:

receivers:
  filelog/systemd:
    include:
      - /var/log/demo/systemd-demo.log
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%Y-%m-%dT%H:%M:%S%z'
      - type: add
        field: attributes.source
        value: systemd-demo
      - type: add
        field: resource["service.name"]
        value: systemd-demo
processors:
  transform/demo_timestamps:
    error_mode: propagate
    log_statements:
      - context: log
        statements:
          - set(resource.attributes["host.name"], log.attributes["hostname"])
          - set(log.time, Now() - Duration("2m") - (Time("2025-11-16T23:45:00Z", "%Y-%m-%dT%H:%M:%SZ") - log.time))
  batch:
    timeout: 5s
    send_batch_size: 500
    send_batch_max_size: 500
exporters:
  otlp_http/observability:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
service:
  pipelines:
    logs:
      receivers: [filelog/systemd]
      processors: [transform/demo_timestamps, batch]
      exporters: [otlp_http/observability]

The text sample has no journal priority field, so the demo does not assign severity from the message text.

Run the demo collector

Set the connection variables, then run from the directory containing both files:

docker run --rm --name systemd-demo-collector \
  --add-host=host.docker.internal:host-gateway \
  --memory=512m \
  -e OTEL_EXPORTER_OTLP_ENDPOINT \
  -e OTEL_EXPORTER_OTLP_AUTHORIZATION \
  -v "$(pwd)/systemd-demo.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  -v "$(pwd)/systemd-demo.log:/var/log/demo/systemd-demo.log:ro" \
  otel/opentelemetry-collector-contrib:0.161.0 \
  --config=/etc/otelcol-contrib/config.yaml

Leave the terminal open while verifying ingestion. Stop with Ctrl+C when finished. Running the demo again re-ingests the sample with refreshed timestamps.

Verify demo logs

Open Search, select Logs, filter for ServiceName:systemd-demo, and select Last 2 days. You should see the sample entries with unit, hostname, and message attributes.

Dashboards and visualization

Import the dashboard

Open Dashboards and click Import in the upper-right corner.
Upload systemd-logs-dashboard.json.
Map the charts to your Logs source and click Finish Import.

View the dashboard

Select Last 2 days for the sample and optionally filter for ServiceName:systemd-demo. For live logs, use ServiceName:systemd-logs and a time range covering your records. The dashboard includes log volume, units, SSH events, and service failures.For live journal logs, change the SSH authentication chart’s Body LIKE '%sshd%' condition to LogAttributes['SYSLOG_IDENTIFIER'] = 'sshd'. The live receiver stores the message in Body and the process identifier separately.

Troubleshooting

journalctl not found

Run the live collector on the monitored Linux host with journalctl installed and available in its execution path. The demo uses filelog and does not require it.

No logs appearing

Check the collector output for journal access, authentication, or export errors. Confirm sudo journalctl -n 20 returns entries on the monitored host. If you configured a unit filter, confirm it matches your service names. Generate a new entry after starting collection. In ClickHouse Observability, check the Logs source, service-name filter, and time range. For the demo, select Last 2 days and confirm the sample file is mounted at the configured path.

Next steps

Set up alerts for service failures or authentication failures.
Create additional dashboards for your services.
See Sending OpenTelemetry data for collector deployment guidance.

Introduction

Features

Administration

Guides

Integration partners

Self-managed

Monitoring Systemd logs with ClickHouse Observability

Prerequisites

Connection details

Integration with existing systems

Demo dataset

Dashboards and visualization

Troubleshooting

journalctl not found

No logs appearing

Next steps

​Prerequisites

​Connection details

​Integration with existing systems

​Demo dataset

​Dashboards and visualization

​Troubleshooting

​journalctl not found

​No logs appearing

​Next steps

Prerequisites

Connection details

Integration with existing systems

Demo dataset

Dashboards and visualization

Troubleshooting

journalctl not found

No logs appearing

Next steps