Skip to main content
Skip to main content

Monitoring AWS CloudWatch logs

This guide walks you through forwarding AWS CloudWatch logs into Managed ClickStack using the OpenTelemetry awscloudwatch receiver, then viewing them in the ClickStack UI.

We'll run a separate collector that polls CloudWatch via the AWS API and forwards events to your ClickStack collector via OTLP. Keep this collector in the same AWS account and region as the log groups to minimise API latency and cost.

This guide assumes you've completed Setting up your OpenTelemetry Collector and have a ClickStack collector running.

The ClickStack collector can be deployed either as a Docker container (see Setting up your OpenTelemetry Collector) or as a Helm release in Kubernetes via the upstream OpenTelemetry Helm chart with the ClickStack collector image (see Deploying the collector). Ensure you have recorded its OTLP endpoint and the OTLP_AUTH_TOKEN you set when deploying it.

Gather your prerequisites​

You'll need:

  • An AWS account with one or more CloudWatch log groups and credentials with the IAM permissions below.
  • A host with Docker installed, AWS API access, and outbound network access to your ClickStack collector. Typically this is an EC2 instance in the same AWS account and region as the log groups.
  • The OTLP endpoint of your ClickStack collector, reachable from this host. If it's running in Docker on the same machine, use http://host.docker.internal:4318 (see the callout in Configure the CloudWatch receiver). For a remote collector, use its full URL, for example https://otel.example.com:4318.
  • The OTLP_AUTH_TOKEN value you set on your ClickStack collector. If you didn't secure it, you can drop the authorization header from the config below.

Configure AWS credentials​

The receiver picks up AWS credentials from the standard environment variables. Export them on the host that will run the collector.

For AWS SSO users:

aws sso login --profile YOUR_PROFILE_NAME
eval $(aws configure export-credentials --profile YOUR_PROFILE_NAME --format env)
aws sts get-caller-identity

For IAM users with long-term credentials:

export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_REGION="us-east-1"
aws sts get-caller-identity

The credentials need the following IAM policy to read CloudWatch logs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudWatchLogsRead",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:FilterLogEvents"
      ],
      "Resource": "arn:aws:logs:*:YOUR_ACCOUNT_ID:log-group:*"
    }
  ]
}

Replace YOUR_ACCOUNT_ID with your AWS account ID.

Production credentials

For production, prefer instance-attached credentials over long-term keys: an IAM role on EC2, IRSA on EKS, or a task role on ECS. The same collector config below works without any credential env vars when the receiver can resolve credentials from the instance metadata service.

Configure the CloudWatch receiver​

Export your ClickStack collector endpoint and auth token, then create an otel-collector-config.yaml.

Same-host setup

The example below assumes the ClickStack collector and this CloudWatch collector run on the same host, so the receiver connects to it via host.docker.internal (the Docker host's address from inside a container). If your ClickStack collector lives elsewhere (an in-cluster service, a public URL, a private IP), substitute its address in OTEL_COLLECTOR_ENDPOINT below.

export OTEL_COLLECTOR_ENDPOINT="http://host.docker.internal:4318"
export OTLP_AUTH_TOKEN="a-strong-shared-secret"
Discover the log groups available in your account

Before editing the config, list the log groups that exist in your region so you can pick real names (and confirm the region is correct):

aws logs describe-log-groups --region eu-central-1 \
  --query 'logGroups[].logGroupName' --output table

Example output:

-------------------------------
|      DescribeLogGroups      |
+-----------------------------+
|  /aws-glue/jobs/error       |
|  /aws-glue/jobs/logs-v2     |
|  /aws-glue/jobs/output      |
|  /aws-glue/sessions/error   |
|  /aws-glue/sessions/output  |
+-----------------------------+

Use the names from this list directly in the groups.named block of Example 1 below. For the account above, the named-groups section would become:

groups:
  named:
    /aws-glue/jobs/error:
    /aws-glue/jobs/logs-v2:
    /aws-glue/jobs/output:
    /aws-glue/sessions/error:
    /aws-glue/sessions/output:

Alternatively, if the groups you want share a common prefix (here /aws-glue/), use Example 2 with prefix: /aws-glue/ instead of listing them individually.

Example 1: Named log groups (recommended)

cat > otel-collector-config.yaml <<'EOF'
receivers:
  awscloudwatch:
    region: eu-central-1
    logs:
      poll_interval: 1m
      max_events_per_request: 100
      groups:
        named:
          /aws-glue/jobs/error:
          /aws-glue/jobs/output:
          /aws-glue/sessions/error:

processors:
  batch:
    timeout: 10s

exporters:
  otlphttp:
    endpoint: ${OTEL_COLLECTOR_ENDPOINT}
    headers:
      authorization: ${OTLP_AUTH_TOKEN}

service:
  pipelines:
    logs:
      receivers: [awscloudwatch]
      processors: [batch]
      exporters: [otlphttp]
EOF

Example 2: Autodiscover log groups by prefix

cat > otel-collector-config.yaml <<'EOF'
receivers:
  awscloudwatch:
    region: eu-central-1
    logs:
      poll_interval: 1m
      max_events_per_request: 100
      groups:
        autodiscover:
          limit: 100
          prefix: /aws-glue/

processors:
  batch:
    timeout: 10s

exporters:
  otlphttp:
    endpoint: ${OTEL_COLLECTOR_ENDPOINT}
    headers:
      authorization: ${OTLP_AUTH_TOKEN}

service:
  pipelines:
    logs:
      receivers: [awscloudwatch]
      processors: [batch]
      exporters: [otlphttp]
EOF

Key settings to adjust:

  • region to match where your log groups live.
  • poll_interval (1m default). Lower values give near-real-time logs at the cost of more AWS API calls.
  • groups.named for an explicit list, or groups.autodiscover.prefix to pick up every group matching a prefix.

For the full set of options, see the CloudWatch receiver documentation.

Recent logs only

On first run, the receiver checkpoints at the current time and only fetches logs from that point forward. Historical logs aren't backfilled.

Start the receiver collector​

Create a docker-compose.yaml alongside otel-collector-config.yaml. The extra_hosts entry lets the container reach a ClickStack collector running on the same host via host.docker.internal; the long-form bind mount errors explicitly if the config file is missing, rather than silently creating an empty directory:

cat > docker-compose.yaml <<'EOF'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-config.yaml"]
    volumes:
      - type: bind
        source: ./otel-collector-config.yaml
        target: /etc/otel-config.yaml
        read_only: true
    environment:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_SESSION_TOKEN
      - AWS_REGION
      - OTEL_COLLECTOR_ENDPOINT
      - OTLP_AUTH_TOKEN
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped
EOF

Start the collector:

docker compose up -d

Tail its logs to confirm it's polling CloudWatch and exporting to your ClickStack collector:

docker compose logs -f otel-collector

Confirm in the ClickStack UI​

Open your service in the ClickHouse Cloud console and select ClickStack from the left menu.

In the Search view, switch the source to Logs and set the time range to Last 15 minutes. CloudWatch events should appear within a couple of poll intervals.

Each event carries the source group and stream as resource attributes:

  • ResourceAttributes['aws.region']: the AWS region (for example eu-central-1)
  • ResourceAttributes['cloudwatch.log.group.name']: the source log group
  • ResourceAttributes['cloudwatch.log.stream']: the source log stream
  • Body: the original log line

Modify the search to Timestamp, SeverityText as level, ResourceAttributes['aws.region'], ResourceAttributes['cloudwatch.log.group.name'], ResourceAttributes['cloudwatch.log.stream'], Body to include these attributes:

Select a log entry to inspect is metadata:

If nothing shows up:

  • Run aws sts get-caller-identity on the collector host to confirm credentials are valid.
  • Tail the collector with docker compose logs -f otel-collector and look for AccessDeniedException (IAM), security token errors (expired SSO credentials), ResourceNotFoundException (log group name typo or wrong region), or connection refused (your ClickStack collector endpoint is unreachable from inside the container, see the host.docker.internal note in Configure the CloudWatch receiver).
  • Verify OTEL_COLLECTOR_ENDPOINT is reachable from inside the container: docker compose exec otel-collector wget -qO- ${OTEL_COLLECTOR_ENDPOINT}/v1/logs -S 2>&1 | head -5.
  • Confirm OTLP_AUTH_TOKEN matches the value set on your ClickStack collector.

Import the CloudWatch dashboard (optional)​

A pre-built dashboard with log volume, severity breakdown, and error distribution is available for download.

Download cloudwatch-logs-dashboard.json, then in the ClickStack UI navigate to Dashboards, click Import.

Upload the JSON file and click Finish Import.

Further reading​