> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring AWS CloudWatch logs with Managed ClickStack

> Forward AWS CloudWatch logs into Managed ClickStack via the OpenTelemetry CloudWatch receiver

This guide walks you through forwarding AWS CloudWatch logs into Managed ClickStack using the OpenTelemetry [`awscloudwatch` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver), then viewing them in the ClickStack UI.

We'll run a separate collector that polls CloudWatch via the AWS API and forwards events to your ClickStack collector via OTLP. Keep this collector in the same AWS account and region as the log groups to minimise API latency and cost.

This guide assumes you've completed [Setting up your OpenTelemetry Collector](https://clickhouse.com/docs/clickstack/managed-onboarding/setting-up-your-opentelemetry-collector) and have a ClickStack collector running.

The ClickStack collector can be deployed either as a **Docker container** (see [Setting up your OpenTelemetry Collector](https://clickhouse.com/docs/clickstack/managed-onboarding/setting-up-your-opentelemetry-collector)) or as a **Helm release** in Kubernetes via the upstream OpenTelemetry Helm chart with the ClickStack collector image (see [Deploying the collector](https://clickhouse.com/docs/clickstack/ingesting-data/collector.md#configuring-the-collector)). **Ensure you have recorded its OTLP endpoint** and the `OTLP_AUTH_TOKEN` you set when deploying it.

**1. Gather your prerequisites**

You'll need:

* An **AWS account** with one or more CloudWatch log groups and credentials with the IAM permissions below.
* A host with **Docker** installed, AWS API access, and outbound network access to your ClickStack collector. Typically this is an EC2 instance in the same AWS account and region as the log groups.
* The **OTLP endpoint** of your ClickStack collector, reachable from this host. If it's running in Docker on the same machine, use `http://host.docker.internal:4318` (see the callout in [Configure the CloudWatch receiver](#configure-receiver)). For a remote collector, use its full URL, for example `https://otel.example.com:4318`.
* The `OTLP_AUTH_TOKEN` value you set on your ClickStack collector. If you didn't secure it, you can drop the `authorization` header from the config below.

**2. Configure AWS credentials**

The receiver picks up AWS credentials from the standard environment variables. Export them on the host that will run the collector.

**For AWS SSO users:**

```shell theme={null}
aws sso login --profile YOUR_PROFILE_NAME
eval $(aws configure export-credentials --profile YOUR_PROFILE_NAME --format env)
aws sts get-caller-identity
```

**For IAM users with long-term credentials:**

```shell theme={null}
export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_REGION="us-east-1"
aws sts get-caller-identity
```

The credentials need the following IAM policy to read CloudWatch logs:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudWatchLogsRead",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:FilterLogEvents"
      ],
      "Resource": "arn:aws:logs:*:YOUR_ACCOUNT_ID:log-group:*"
    }
  ]
}
```

Replace `YOUR_ACCOUNT_ID` with your AWS account ID.

> [!NOTE]
>
> **Production credentials**
>
> For production, prefer instance-attached credentials over long-term keys: an IAM role on EC2, IRSA on EKS, or a task role on ECS. The same collector config below works without any credential env vars when the receiver can resolve credentials from the instance metadata service.

**3. Configure the CloudWatch receiver**

Export your ClickStack collector endpoint and auth token, then create an `otel-collector-config.yaml`.

> [!NOTE]
>
> **Same-host setup**
>
> The example below assumes the ClickStack collector and this CloudWatch collector run on the same host, so the receiver connects to it via `host.docker.internal` (the Docker host's address from inside a container). If your ClickStack collector lives elsewhere (an in-cluster service, a public URL, a private IP), substitute its address in `OTEL_COLLECTOR_ENDPOINT` below.

```shell theme={null}
export OTEL_COLLECTOR_ENDPOINT="http://host.docker.internal:4318"
export OTLP_AUTH_TOKEN="a-strong-shared-secret"
```

> [accordion] Discover the log groups available in your account
>
> Before editing the config, list the log groups that exist in your region so you can pick real names (and confirm the region is correct):
>
> ```shell theme={null}
> aws logs describe-log-groups --region eu-central-1 \
>   --query 'logGroups[].logGroupName' --output table
> ```
>
> Example output:
>
> ```text theme={null}
> -------------------------------
> |      DescribeLogGroups      |
> +-----------------------------+
> |  /aws-glue/jobs/error       |
> |  /aws-glue/jobs/logs-v2     |
> |  /aws-glue/jobs/output      |
> |  /aws-glue/sessions/error   |
> |  /aws-glue/sessions/output  |
> +-----------------------------+
> ```
>
> Use the names from this list directly in the `groups.named` block of Example 1 below. For the account above, the named-groups section would become:
>
> ```yaml theme={null}
> groups:
>   named:
>     /aws-glue/jobs/error:
>     /aws-glue/jobs/logs-v2:
>     /aws-glue/jobs/output:
>     /aws-glue/sessions/error:
>     /aws-glue/sessions/output:
> ```
>
> Alternatively, if the groups you want share a common prefix (here `/aws-glue/`), use Example 2 with `prefix: /aws-glue/` instead of listing them individually.

**Example 1: Named log groups (recommended)**

```shell theme={null}
cat > otel-collector-config.yaml <<'EOF'
receivers:
  awscloudwatch:
    region: eu-central-1
    logs:
      poll_interval: 1m
      max_events_per_request: 100
      groups:
        named:
          /aws-glue/jobs/error:
          /aws-glue/jobs/output:
          /aws-glue/sessions/error:

processors:
  batch:
    timeout: 10s

exporters:
  otlphttp:
    endpoint: ${OTEL_COLLECTOR_ENDPOINT}
    headers:
      authorization: ${OTLP_AUTH_TOKEN}

service:
  pipelines:
    logs:
      receivers: [awscloudwatch]
      processors: [batch]
      exporters: [otlphttp]
EOF
```

**Example 2: Autodiscover log groups by prefix**

```shell theme={null}
cat > otel-collector-config.yaml <<'EOF'
receivers:
  awscloudwatch:
    region: eu-central-1
    logs:
      poll_interval: 1m
      max_events_per_request: 100
      groups:
        autodiscover:
          limit: 100
          prefix: /aws-glue/

processors:
  batch:
    timeout: 10s

exporters:
  otlphttp:
    endpoint: ${OTEL_COLLECTOR_ENDPOINT}
    headers:
      authorization: ${OTLP_AUTH_TOKEN}

service:
  pipelines:
    logs:
      receivers: [awscloudwatch]
      processors: [batch]
      exporters: [otlphttp]
EOF
```

Key settings to adjust:

* `region` to match where your log groups live.
* `poll_interval` (`1m` default). Lower values give near-real-time logs at the cost of more AWS API calls.
* `groups.named` for an explicit list, or `groups.autodiscover.prefix` to pick up every group matching a prefix.

For the full set of options, see the [CloudWatch receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver).

> [!NOTE]
>
> **Recent logs only**
>
> On first run, the receiver checkpoints at the current time and only fetches logs from that point forward. Historical logs aren't backfilled.

**4. Start the receiver collector**

Create a `docker-compose.yaml` alongside `otel-collector-config.yaml`. The `extra_hosts` entry lets the container reach a ClickStack collector running on the same host via `host.docker.internal`; the long-form bind mount errors explicitly if the config file is missing, rather than silently creating an empty directory:

```shell theme={null}
cat > docker-compose.yaml <<'EOF'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-config.yaml"]
    volumes:
      - type: bind
        source: ./otel-collector-config.yaml
        target: /etc/otel-config.yaml
        read_only: true
    environment:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_SESSION_TOKEN
      - AWS_REGION
      - OTEL_COLLECTOR_ENDPOINT
      - OTLP_AUTH_TOKEN
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped
EOF
```

Start the collector:

```shell theme={null}
docker compose up -d
```

Tail its logs to confirm it's polling CloudWatch and exporting to your ClickStack collector:

```shell theme={null}
docker compose logs -f otel-collector
```

**5. Confirm in the ClickStack UI**

Open your service in the [ClickHouse Cloud console](https://console.clickhouse.cloud) and select **ClickStack** from the left menu.

![Launch ClickStack](https://mintcdn.com/private-7c7dfe99/uAzQoYEGwuyhrO3W/images/use-cases/observability/clickstack-cloud-v2.webp?fit=max\&auto=format\&n=uAzQoYEGwuyhrO3W\&q=85\&s=93a41f575b6192117a1421c4aeff65f8)

In the **Search** view, switch the source to `Logs` and set the time range to **Last 15 minutes**. CloudWatch events should appear within a couple of poll intervals.

![ClickStack Search view with CloudWatch logs](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudwatch/log-search-view-clickstack.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=84a1806b138c052a291028facacf2293)

Each event carries the source group and stream as resource attributes:

* `ResourceAttributes['aws.region']`: the AWS region (for example `eu-central-1`)
* `ResourceAttributes['cloudwatch.log.group.name']`: the source log group
* `ResourceAttributes['cloudwatch.log.stream']`: the source log stream
* `Body`: the original log line

Modify the search to `Timestamp, SeverityText as level, ResourceAttributes['aws.region'], ResourceAttributes['cloudwatch.log.group.name'], ResourceAttributes['cloudwatch.log.stream'], Body` to include these attributes:

![ClickStack Search view with CloudWatch logs and attributes](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudwatch/log-search-attributes-clickstack.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=745552ca836fe48490d914a3af4438b9)

Select a log entry to inspect is metadata:

![CloudWatch attributes in the log detail view](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudwatch/error-log-column-values-clickstack.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=b711f9c18174c7966659d4e3204122ff)

If nothing shows up:

* Run `aws sts get-caller-identity` on the collector host to confirm credentials are valid.
* Tail the collector with `docker compose logs -f otel-collector` and look for `AccessDeniedException` (IAM), `security token` errors (expired SSO credentials), `ResourceNotFoundException` (log group name typo or wrong region), or `connection refused` (your ClickStack collector endpoint is unreachable from inside the container, see the `host.docker.internal` note in [Configure the CloudWatch receiver](#configure-receiver)).
* Verify `OTEL_COLLECTOR_ENDPOINT` is reachable from inside the container: `docker compose exec otel-collector wget -qO- ${OTEL_COLLECTOR_ENDPOINT}/v1/logs -S 2>&1 | head -5`.
* Confirm `OTLP_AUTH_TOKEN` matches the value set on your ClickStack collector.

**6. Import the CloudWatch dashboard (optional)**

A pre-built dashboard with log volume, severity breakdown, and error distribution is available for download.

[Download `cloudwatch-logs-dashboard.json`](https://clickhouse-docs-assets.s3.us-east-1.amazonaws.com/examples/cloudwatch-logs-dashboard.json), then in the ClickStack UI navigate to **Dashboards**, click **Import**.

![Import dashboard button](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/clickstack-import-dashboard.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=87e690546f16dd611c31d3d0bf32b453)

Upload the JSON file and click **Finish Import**.

![Finish import dialog](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudwatch/finish-clickstack-import.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=4efb1cd07ee936536a3f2313e10b679e)

**7. Further reading**

* [AWS CloudWatch logs integration reference](https://clickhouse.com/docs/clickstack/integration-examples/cloudwatch.md) for the demo dataset, full troubleshooting, and tuning options.
* [Securing the collector](https://clickhouse.com/docs/clickstack/ingesting-data/collector.md#securing-the-collector) with TLS on the OTLP endpoint and least-privilege ingestion users.
* [Processing, filtering, and enriching](https://clickhouse.com/docs/clickstack/ingesting-data/collector.md#processing-filtering-transforming-enriching) events at the collector.
* [Going to production](https://clickhouse.com/docs/clickstack/managing/production.md) for recommendations when going to production.