> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring AWS Lambda logs with ClickHouse Observability using Rotel

> Collect AWS Lambda logs with the community-maintained Rotel Lambda Extension and send them to ClickHouse Observability.

Community maintained

Collect function logs and extension logs with the Rotel Lambda Extension and send them directly to ClickHouse Observability over OTLP. The extension runs alongside your Lambda function, so no separate collector is needed. You can also add OpenTelemetry instrumentation to collect traces.

## Prerequisites

* A ClickHouse Observability service
* An AWS Lambda function using a ZIP deployment and a runtime supported by the [Rotel Lambda Extension](https://github.com/streamfold/rotel-lambda-extension)
* AWS CLI v2 configured for your function's region, if using the CLI examples
* Permission to update the function's layers and environment variables
* Network access from the Lambda function to your OTLP endpoint, including outbound access for functions in a VPC

### Connection details

> [tab] Fully managed
>
> [Beta offering](https://clickhouse.com/docs/reference/settings/beta-and-experimental-features.md#beta-features)
>
> Open ClickHouse Observability and go to **Team Settings → Data → Managed Telemetry**. Copy your OTLP HTTP endpoint and API key.
>
> ![Managed Telemetry connection details](https://mintcdn.com/private-7c7dfe99/Zk1feNn6fCQBpold/images/clickstack/getting-started/observability-managed-telemetry.png?fit=max\&auto=format\&n=Zk1feNn6fCQBpold\&q=85\&s=e63a2ac2e6ab996d3fc360ebe0f487a5)
>
> Use the exact endpoint, including its port, as `ROTEL_OTLP_EXPORTER_ENDPOINT` below. Use the base HTTP endpoint without a `/v1/logs` suffix and set the protocol to `http`. The managed service requires an `Authorization=Bearer YOUR_API_KEY` header.

> [tab] Self-managed
>
> Use your collector's OTLP HTTP endpoint and authentication settings. Omit `ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS` if the destination does not require authentication. The endpoint must be reachable from Lambda; `localhost` refers to the Lambda execution environment. See the [self-managed setup guide](https://clickhouse.com/docs/clickstack/self-configured/getting-started/managed.md).
>
> Use these destination values in the Lambda environment-variable and `rotel.env` examples below:
>
> ```bash theme={null}
> ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_COLLECTOR_HOST:4318
> ROTEL_OTLP_EXPORTER_PROTOCOL=http
> ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=YOUR_INGESTION_KEY"
> ```

## Integration with existing Lambda functions

**1. Choose the appropriate Rotel Lambda Extension layer**

Choose a layer matching your function's architecture (`amd64` or `arm64`), runtime, and AWS region. Copy the full, versioned layer ARN from the [Rotel releases](https://github.com/streamfold/rotel-lambda-extension/releases). Use that ARN in the examples below; layer versions and availability vary by region.

**2. Add the Rotel layer to your Lambda function**

Replace `YOUR_ROTEL_LAYER_ARN` with the ARN you selected, and `my-function` with your function name.

##### Option 1: AWS Console

1. Open the AWS Lambda console
2. Navigate to your Lambda function
3. Scroll to the **Layers** section and click **Add a layer**
4. Select **Specify an ARN**
5. Enter the Rotel layer ARN:
   ```text theme={null}
   YOUR_ROTEL_LAYER_ARN
   ```
6. Click **Add**

##### Option 2: AWS CLI

Export the current layer list:

```bash theme={null}
aws lambda get-function-configuration \
  --function-name my-function \
  --query 'Layers[].Arn || `[]`' \
  --output json > layers.json
```

Append the Rotel layer ARN as a string in the `layers.json` array, preserving the existing entries. Then apply the complete list:

```bash theme={null}
aws lambda update-function-configuration \
  --function-name my-function \
  --layers file://layers.json
aws lambda wait function-updated --function-name my-function
```

##### Option 3: AWS SAM

```yaml theme={null}
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      # ... other configuration ...
      Layers:
        # Keep any existing layer entries here.
        - YOUR_ROTEL_LAYER_ARN
```

**3. Configure the extension to export to ClickHouse Observability**

In the Lambda console, open **Configuration → Environment variables** and add the settings below, preserving any existing variables. These are Lambda configuration values, not commands to run in your local terminal. When entering values in the console, omit the surrounding quotes shown in the examples.

##### Basic configuration (environment variables)

Add these environment variables to your Lambda function:

```bash theme={null}
# OTLP HTTP endpoint from Connection details
ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_OTEL_ENDPOINT:4318
ROTEL_OTLP_EXPORTER_PROTOCOL=http

# Required for managed ingestion
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=Bearer YOUR_API_KEY"

# Optional: Service name (defaults to Lambda function name)
ROTEL_OTEL_RESOURCE_ATTRIBUTES="service.name=my-lambda-api,service.version=1.0.0"
```

##### Advanced configuration (using .env file)

For more complex configurations, create a `rotel.env` file in your Lambda function bundle:

**rotel.env:**

```bash theme={null}
ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_OTEL_ENDPOINT:4318
ROTEL_OTLP_EXPORTER_PROTOCOL=http
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=Bearer YOUR_API_KEY"
ROTEL_OTEL_RESOURCE_ATTRIBUTES="service.name=my-lambda-api,deployment.environment=production"
```

Then set the environment variable to point to this file:

```bash theme={null}
ROTEL_ENV_FILE=/var/task/rotel.env
```

##### Using AWS Secrets Manager or Parameter Store

You can store the API key in AWS Secrets Manager or Parameter Store and reference its ARN in `rotel.env`. The examples below use fully managed authentication and assume the secret contains the API key only; Rotel adds the `Bearer` prefix from the configuration. For a self-managed collector using an ingestion key, omit the `Bearer ` prefix. Replace the example region, account, and secret names with your own:

**AWS Secrets Manager Example:**

```bash theme={null}
ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_OTEL_ENDPOINT:4318
ROTEL_OTLP_EXPORTER_PROTOCOL=http
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=Bearer ${arn:aws:secretsmanager:us-east-1:123456789012:secret:observability-api-key-abc123}"
```

**AWS Parameter Store Example:**

```bash theme={null}
ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_OTEL_ENDPOINT:4318
ROTEL_OTLP_EXPORTER_PROTOCOL=http
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=Bearer ${arn:aws:ssm:us-east-1:123456789012:parameter/observability-api-key}"
```

**Required IAM Permissions:**

Add these permissions to your Lambda execution role:

For Secrets Manager:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:BatchGetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:observability-api-key-*"
    }
  ]
}
```

For Parameter Store:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters"
      ],
      "Resource": "arn:aws:ssm:us-east-1:123456789012:parameter/observability-api-key"
    }
  ]
}
```

> [!NOTE]
>
> Secret retrieval happens during initialization and can add cold-start latency. See the [Rotel secret configuration](https://github.com/streamfold/rotel-lambda-extension#secrets) for supported formats and permissions. Customer-managed encryption keys may also require `kms:Decrypt` permission.

**4. Test the integration**

Invoke your Lambda function with an event it accepts and ensure the handler writes a log. Replace this example payload if your function expects a different event:

```bash theme={null}
aws lambda invoke \
  --function-name my-function \
  --cli-binary-format raw-in-base64-out \
  --payload '{"test": "data"}' \
  response.json
```

Check the Lambda logs for any errors:

```bash theme={null}
aws logs tail /aws/lambda/my-function --follow
```

**5. Verify logs**

Open **Search** in ClickHouse Observability, select your **Logs** source, and filter for `ServiceName:my-lambda-api` if you used the example service name. Otherwise, use your Lambda function name. Select **Last 15 minutes**.

![Lambda Log View](https://mintcdn.com/private-7c7dfe99/y4_YksnkJMz9sseb/images/clickstack/lambda/lambda-log-view.webp?fit=max\&auto=format\&n=y4_YksnkJMz9sseb\&q=85\&s=f8a7f5350b50a3e018b98c56d6e48e63)

![Lambda Log Detail](https://mintcdn.com/private-7c7dfe99/y4_YksnkJMz9sseb/images/clickstack/lambda/lambda-log.webp?fit=max\&auto=format\&n=y4_YksnkJMz9sseb\&q=85\&s=818c0110feecfd9aa1cc0799d8a1390b)

Look for these key attributes in the logs:

* `service.name`: Your configured service name, or the Lambda function name by default
* `faas.name`: AWS Lambda function name
* `faas.invocation_id`: Unique invocation ID
* `cloud.provider`: "aws"
* `cloud.platform`: "aws\_lambda"

## Disabling CloudWatch Logs (cost optimization)

By default, AWS Lambda sends all logs to CloudWatch Logs, which can be expensive at scale. Once you've verified that logs are flowing to ClickHouse Observability, you can disable CloudWatch logging to reduce costs.

**1. Remove CloudWatch permissions from the execution role**

1. Open the AWS Console and navigate to **AWS Lambda**
2. Navigate to your Lambda function
3. Select **Configuration** → **Permissions**
4. Click the execution role name to open the IAM console
5. Remove the CloudWatch Logs write permissions used by this function. If the role is shared, use a dedicated role before making this change:
   * If using a custom policy, edit to remove `logs:CreateLogGroup`, `logs:CreateLogStream`, and `logs:PutLogEvents`
   * If using the AWS managed policy `AWSLambdaBasicExecutionRole`, remove it from the role
6. Save the role

**2. Verify CloudWatch logging is disabled**

Invoke your function again and verify that:

1. No new events appear in the function's CloudWatch log group
2. Logs continue to appear in ClickHouse Observability

```bash theme={null}
aws logs tail /aws/lambda/my-function --since 5m --follow
```

## Adding OpenTelemetry auto-instrumentation

Rotel can receive telemetry from a language instrumentation layer installed alongside it. Follow the [Rotel auto-instrumentation instructions](https://github.com/streamfold/rotel-lambda-extension#auto-instrumentation) and the linked language-layer documentation to choose a compatible layer and set `AWS_LAMBDA_EXEC_WRAPPER`.

Preserve the Rotel layer and any existing layers when adding instrumentation. After invoking the function, select your **Traces** source in ClickHouse Observability and filter by the configured service name. Log/trace correlation requires the application instrumentation to include trace and span IDs in its logs.

## Example applications

See the example Python app demonstrating the Rotel Lambda Extension:

* **[Python + ClickHouse](https://github.com/streamfold/python-aws-lambda-clickhouse-example)**: Python application with manual OpenTelemetry instrumentation, sending traces and logs directly to ClickHouse

## Join the Rotel community

If you have questions about Rotel, please join the [Rotel Discord server](https://rotel.dev) and share your feedback or questions. Check out the [Rotel Lambda Extension](https://github.com/streamfold/rotel-lambda-extension) to contribute any improvements.

## Additional resources

* **[Rotel Lambda Extension](https://github.com/streamfold/rotel-lambda-extension)**: Source code and detailed documentation
* **[Rotel Core](https://github.com/streamfold/rotel)**: The lightweight OTEL data plane powering the extension