> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring EC2 host logs with ClickHouse Observability

> Collect EC2 syslog files with OpenTelemetry and enrich them with instance metadata in ClickHouse Observability.

Collect EC2 syslog files with an OpenTelemetry collector and send them to ClickHouse Observability with instance metadata. Use the [demo dataset](#demo-dataset) to try the integration without an EC2 instance.

## Prerequisites

* A ClickHouse Observability service
* For your own logs, a Linux EC2 instance with syslog files, access to its instance metadata service, and permission to install and run the collector
* For the demo, Docker
* `curl` to download the collector or sample

### Connection details

> [tab] Fully managed
>
> [Beta offering](https://clickhouse.com/docs/reference/settings/beta-and-experimental-features.md#beta-features)
>
> Open ClickHouse Observability and go to **Team Settings → Data → Managed Telemetry**. Copy your OTLP HTTP endpoint and API key. You can use **Copy connection details** to copy the configuration, or **Reveal key** to view the masked key.
>
> ![Managed Telemetry connection details](https://mintcdn.com/private-7c7dfe99/Zk1feNn6fCQBpold/images/clickstack/getting-started/observability-managed-telemetry.png?fit=max\&auto=format\&n=Zk1feNn6fCQBpold\&q=85\&s=e63a2ac2e6ab996d3fc360ebe0f487a5)
>
> For the OTLP exporter examples below, set these variables in the terminal where you will run the collector:
>
> ```bash theme={null}
> export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_ENDPOINT:4318'
> export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY'
> ```
>
> Replace the placeholders with the exact endpoint, including its port, and API key from your service. Use the base HTTP endpoint without a `/v1/logs` suffix; the exporter appends the signal path.

> [tab] Self-managed
>
> Use your collector's reachable OTLP HTTP endpoint (for example, `http://YOUR_COLLECTOR_HOST:4318`) and authentication settings. If your collector does not require authentication, omit the exporter's `headers` block from the configurations below. See the [self-managed getting-started guide](https://clickhouse.com/docs/clickstack/self-configured/getting-started/managed.md) for setup guidance.
>
> ```bash theme={null}
> export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318'
> export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_INGESTION_KEY'
> ```

Your EC2 instance must be able to reach the endpoint. For the Docker demo, a destination collector on your own computer is reachable at `http://host.docker.internal:4318`; the demo command includes the host mapping.

## Integration with an existing EC2 instance

Run these steps on the EC2 instance whose logs you want to collect.

**1. Verify instance metadata and syslog**

Check that the instance metadata service is accessible using IMDSv2:

```bash theme={null}
TOKEN=$(curl -fsS --connect-timeout 5 -X PUT \
  -H 'X-aws-ec2-metadata-token-ttl-seconds: 300' \
  http://169.254.169.254/latest/api/token)
curl -fsS --connect-timeout 5 -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-id
```

The response should contain your instance ID. The collector uses this metadata service to add instance ID, type, region, availability zone, and account ID to your logs. The configuration below does not collect EC2 tags.

Locate your syslog file and inspect its format:

```bash theme={null}
# Ubuntu/Debian
tail -20 /var/log/syslog

# Amazon Linux/RHEL
tail -20 /var/log/messages
```

Use the path that exists on your instance. If it writes only to the systemd journal, use the [Systemd/Journald guide](https://clickhouse.com/docs/clickstack/observability/integration-examples/systemd.md) for collection.

**2. Install the collector**

Download OpenTelemetry Collector Contrib `0.161.0` for your instance's architecture:

```bash theme={null}
case "$(uname -m)" in
  x86_64) ARCH=amd64 ;;
  aarch64) ARCH=arm64 ;;
  *) echo "Unsupported architecture"; exit 1 ;;
esac
curl -fLO "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.161.0/otelcol-contrib_0.161.0_linux_${ARCH}.tar.gz"
tar -xzf "otelcol-contrib_0.161.0_linux_${ARCH}.tar.gz" otelcol-contrib
sudo install -m 755 otelcol-contrib /usr/local/bin/otelcol-contrib
/usr/local/bin/otelcol-contrib --version
```

**3. Configure the collector**

Save the matching configuration as `ec2-host-logs.yaml`. Choose by the timestamp format in your file and adjust `include` if your syslog path differs.

> [tab] ISO timestamps
>
> For timestamps such as `2026-10-15T12:34:56.123456+00:00`:
>
> ```yaml theme={null}
> receivers:
>   filelog/ec2:
>     include: [/var/log/syslog]
>     start_at: end
>     operators:
>       - type: regex_parser
>         regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
>         timestamp:
>           parse_from: attributes.timestamp
>           layout_type: gotime
>           layout: '2006-01-02T15:04:05.999999999Z07:00'
>           location: UTC
>       - type: add
>         field: attributes.source
>         value: ec2-host-logs
>       - type: add
>         field: resource["service.name"]
>         value: ec2-host-service
>       - type: copy
>         from: attributes.hostname
>         to: resource["host.name"]
>
> processors:
>   memory_limiter:
>     check_interval: 1s
>     limit_mib: 256
>     spike_limit_mib: 64
>   resourcedetection/ec2:
>     detectors: [ec2]
>     timeout: 5s
>     override: false
>   batch:
>     timeout: 5s
>     send_batch_size: 500
>     send_batch_max_size: 500
>
> exporters:
>   otlp_http/observability:
>     endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
>     headers:
>       authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
>
> service:
>   pipelines:
>     logs/ec2:
>       receivers: [filelog/ec2]
>       processors: [memory_limiter, resourcedetection/ec2, batch]
>       exporters: [otlp_http/observability]
> ```

> [tab] Traditional syslog
>
> For timestamps such as `Oct 15 12:34:56`:
>
> ```yaml theme={null}
> receivers:
>   filelog/ec2:
>     include: [/var/log/messages]
>     start_at: end
>     operators:
>       - type: regex_parser
>         regex: '^(?P<timestamp>\w+\s+\d+ \d{2}:\d{2}:\d{2}) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
>         timestamp:
>           parse_from: attributes.timestamp
>           layout_type: gotime
>           layout: 'Jan _2 15:04:05'
>           location: UTC
>       - type: add
>         field: attributes.source
>         value: ec2-host-logs
>       - type: add
>         field: resource["service.name"]
>         value: ec2-host-service
>       - type: copy
>         from: attributes.hostname
>         to: resource["host.name"]
>
> processors:
>   memory_limiter:
>     check_interval: 1s
>     limit_mib: 256
>     spike_limit_mib: 64
>   resourcedetection/ec2:
>     detectors: [ec2]
>     timeout: 5s
>     override: false
>   batch:
>     timeout: 5s
>     send_batch_size: 500
>     send_batch_max_size: 500
>
> exporters:
>   otlp_http/observability:
>     endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
>     headers:
>       authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
>
> service:
>   pipelines:
>     logs/ec2:
>       receivers: [filelog/ec2]
>       processors: [memory_limiter, resourcedetection/ec2, batch]
>       exporters: [otlp_http/observability]
> ```
>
> Traditional timestamps omit the year and time zone. The parser infers the year; set `location` to your host's IANA time zone if it does not log in UTC.

The receiver retains the original log line and extracts hostname, unit, PID (when present), and message. The `resourcedetection/ec2` processor adds EC2 resource attributes. To collect instance tags as well, see the [EC2 detector configuration](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.161.0/processor/resourcedetectionprocessor#aws-ec2) and its permission requirements.

**4. Run the collector**

Set the [connection variables](#connection-details) in this terminal, then run from the directory containing the configuration. The collector needs permission to read the syslog file:

```bash theme={null}
sudo --preserve-env=OTEL_EXPORTER_OTLP_ENDPOINT,OTEL_EXPORTER_OTLP_AUTHORIZATION \
  /usr/local/bin/otelcol-contrib --config="$(pwd)/ec2-host-logs.yaml"
```

Keep the terminal open. The receiver uses `start_at: end`, so generate new activity that writes to the syslog file after startup. For ongoing collection, configure a service using your deployment process and supply the connection variables to that service.

**5. Verify logs**

Open **Search**, select your **Logs** source, filter for `ServiceName:ec2-host-service`, and select **Last 15 minutes**. Expand a record and check `cloud.provider`, `cloud.region`, `cloud.availability_zone`, `host.id`, and `host.type` in its resource attributes.

## Demo dataset

The sample contains 11,658 syslog lines covering 24 hours. The collector adds simulated EC2 metadata and refreshes the timestamps; no AWS account is needed.

**1. Download the sample**

```bash theme={null}
mkdir -p ec2-demo
cd ec2-demo
curl -fLO https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/host-logs/journal.log
```

**2. Configure the demo collector**

Save this as `ec2-host-logs-demo.yaml` in the download directory:

```yaml theme={null}
receivers:
  filelog/ec2:
    include: [/var/log/host/journal.log]
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S%z'
          location: UTC
      - type: add
        field: attributes.source
        value: ec2-demo
      - type: add
        field: resource["service.name"]
        value: ec2-demo
      - type: copy
        from: attributes.hostname
        to: resource["host.name"]

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64
  transform/demo_timestamps:
    error_mode: propagate
    log_statements:
      - context: log
        statements:
          - set(time, Now() - Duration("2m") - (Time("2025-11-11T23:59:01Z", "%Y-%m-%dT%H:%M:%SZ") - time))
  resource/ec2_demo:
    attributes:
      - key: cloud.provider
        value: "aws"
        action: insert
      - key: cloud.platform
        value: "aws_ec2"
        action: insert
      - key: cloud.region
        value: "us-east-1"
        action: insert
      - key: cloud.availability_zone
        value: "us-east-1a"
        action: insert
      - key: host.id
        value: "i-0abc123def456789"
        action: insert
      - key: host.type
        value: "t3.medium"
        action: insert
  batch:
    timeout: 5s
    send_batch_size: 500
    send_batch_max_size: 500

exporters:
  otlp_http/observability:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}

service:
  pipelines:
    logs/ec2:
      receivers: [filelog/ec2]
      processors: [memory_limiter, resource/ec2_demo, transform/demo_timestamps, batch]
      exporters: [otlp_http/observability]
```

> [!NOTE]
>
> The demo-only timestamp transform shifts the sample into roughly the past 24 hours, ending two minutes before ingestion. Use the [EC2 instance configuration](#existing-ec2) for your own logs to preserve their original timestamps.

**3. Run the demo collector**

Set the [connection variables](#connection-details), then run from the download directory:

```bash theme={null}
docker run --rm --name ec2-demo-collector \
  --add-host=host.docker.internal:host-gateway \
  --memory=512m \
  -e OTEL_EXPORTER_OTLP_ENDPOINT \
  -e OTEL_EXPORTER_OTLP_AUTHORIZATION \
  -v "$(pwd)/ec2-host-logs-demo.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  -v "$(pwd):/var/log/host:ro" \
  otel/opentelemetry-collector-contrib:0.161.0 \
  --config=/etc/otelcol-contrib/config.yaml
```

Keep the terminal open while the collector exports the sample, and check its output for errors. Starting a fresh container again can duplicate the sample logs.

**4. Verify the demo logs**

Open **Search**, select your **Logs** source, filter for `ServiceName:ec2-demo`, and select **Last 2 days**. Expand a log and check the simulated resource attributes: `cloud.region` is `us-east-1`, `host.id` is `i-0abc123def456789`, and `host.type` is `t3.medium`. Stop the collector with **Ctrl+C** after verifying the data.

![EC2 demo logs in ClickHouse Observability Search](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-ec2-demo-logs.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=db20aac2ee4ef9737e4d34a432100469)

## Dashboards and visualization

**1. [Download the dashboard configuration](https://clickhouse-docs-assets.s3.us-east-1.amazonaws.com/examples/host-logs-dashboard.json)**

**2. Import the dashboard**

1. Open **Dashboards** and click **Import** in the upper-right corner.
2. Upload `host-logs-dashboard.json`.
3. Map the charts to your **Logs** source and click **Finish Import**.

![Import dashboard button](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-import-dashboard.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=16bcb7e6365ca0f6decac5c693e228eb)

**3. View the dashboard**

The imported dashboard displays your logs for the selected time range. To isolate the sample from other services, optionally filter for `ServiceName:ec2-demo`. Select **Last 2 days** to see the full sample. You can also filter by EC2 resource attributes such as `ResourceAttributes.cloud.region:us-east-1` or `ResourceAttributes.host.id:i-0abc123def456789`.

The dashboard includes log volume, system units, SSH activity, firewall blocks, and service restarts. Severity is unset unless your ingestion pipeline supplies it.

![Host logs dashboard in ClickHouse Observability](https://mintcdn.com/private-7c7dfe99/dW6bcEakZdn3a5oP/images/clickstack/observability-ec2-dashboard.png?fit=max\&auto=format\&n=dW6bcEakZdn3a5oP\&q=85\&s=e58730e1639e4c411173209cc4c80eec)

> [!NOTE]
>
> If the SSH activity, Firewall blocks, or untitled security-events charts are empty after filtering for your EC2 service, remove `ServiceName = 'host-demo' AND` from their aggregation filters if present. Then use the dashboard's service-name filter to select your EC2 logs.

## Troubleshooting

### EC2 metadata is missing

Run the metadata check from the EC2 instance. Ensure IMDS is enabled and local firewall or proxy settings allow requests to `169.254.169.254`. If you use a proxy, exclude that address with `NO_PROXY`. Check collector output for metadata detection errors. The demo uses simulated metadata and does not contact IMDS.

### No logs appearing

* Confirm the path in `include` exists and the collector can read it.
* Generate new log entries after startup; `start_at: end` skips existing content.
* Check collector output for authentication or export errors and confirm the endpoint is reachable from the instance.
* Verify the **Logs** source, service-name filter, and time range. Use **Last 2 days** for the demo.

### Logs not parsing correctly

Compare your file's timestamps with the selected configuration. For traditional syslog, set the correct host time zone. Only collect files matching the configured format.

## Next steps

* Set up [alerts](https://clickhouse.com/docs/clickstack/observability/features/alerts.md) for authentication failures or service errors.
* Filter by EC2 resource attributes to investigate an instance or region.
* See [Sending OpenTelemetry data](https://clickhouse.com/docs/clickstack/observability/ingesting-data/opentelemetry.md) for deployment guidance.