Monitoring AWS Lambda logs with ClickHouse Observability using Rotel

Collect function logs and extension logs with the Rotel Lambda Extension and send them directly to ClickHouse Observability over OTLP. The extension runs alongside your Lambda function, so no separate collector is needed. You can also add OpenTelemetry instrumentation to collect traces.

Prerequisites

A ClickHouse Observability service
An AWS Lambda function using a ZIP deployment and a runtime supported by the Rotel Lambda Extension
AWS CLI v2 configured for your function’s region, if using the CLI examples
Permission to update the function’s layers and environment variables
Network access from the Lambda function to your OTLP endpoint, including outbound access for functions in a VPC

Connection details

Fully managed
Self-managed

Open ClickHouse Observability and go to Team Settings → Data → Managed Telemetry. Copy your OTLP HTTP endpoint and API key.Use the exact endpoint, including its port, as ROTEL_OTLP_EXPORTER_ENDPOINT below. Use the base HTTP endpoint without a /v1/logs suffix and set the protocol to http. The managed service requires an Authorization=Bearer YOUR_API_KEY header.

Use your collector’s OTLP HTTP endpoint and authentication settings. Omit ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS if the destination does not require authentication. The endpoint must be reachable from Lambda; localhost refers to the Lambda execution environment. See the self-managed setup guide.Use these destination values in the Lambda environment-variable and rotel.env examples below:

ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_COLLECTOR_HOST:4318
ROTEL_OTLP_EXPORTER_PROTOCOL=http
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=YOUR_INGESTION_KEY"

Integration with existing Lambda functions

Choose the appropriate Rotel Lambda Extension layer

Choose a layer matching your function’s architecture (amd64 or arm64), runtime, and AWS region. Copy the full, versioned layer ARN from the Rotel releases. Use that ARN in the examples below; layer versions and availability vary by region.

Add the Rotel layer to your Lambda function

Replace YOUR_ROTEL_LAYER_ARN with the ARN you selected, and my-function with your function name.

Open the AWS Lambda console
Navigate to your Lambda function
Scroll to the Layers section and click Add a layer
Select Specify an ARN
Enter the Rotel layer ARN:
YOUR_ROTEL_LAYER_ARN
Click Add

Export the current layer list:

aws lambda get-function-configuration \
  --function-name my-function \
  --query 'Layers[].Arn || `[]`' \
  --output json > layers.json

Append the Rotel layer ARN as a string in the layers.json array, preserving the existing entries. Then apply the complete list:

aws lambda update-function-configuration \
  --function-name my-function \
  --layers file://layers.json
aws lambda wait function-updated --function-name my-function

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      # ... other configuration ...
      Layers:
        # Keep any existing layer entries here.
        - YOUR_ROTEL_LAYER_ARN

Configure the extension to export to ClickHouse Observability

In the Lambda console, open Configuration → Environment variables and add the settings below, preserving any existing variables. These are Lambda configuration values, not commands to run in your local terminal. When entering values in the console, omit the surrounding quotes shown in the examples.Add these environment variables to your Lambda function:

# OTLP HTTP endpoint from Connection details
ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_OTEL_ENDPOINT:4318
ROTEL_OTLP_EXPORTER_PROTOCOL=http

# Required for managed ingestion
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=Bearer YOUR_API_KEY"

# Optional: Service name (defaults to Lambda function name)
ROTEL_OTEL_RESOURCE_ATTRIBUTES="service.name=my-lambda-api,service.version=1.0.0"

For more complex configurations, create a rotel.env file in your Lambda function bundle:rotel.env:

ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_OTEL_ENDPOINT:4318
ROTEL_OTLP_EXPORTER_PROTOCOL=http
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=Bearer YOUR_API_KEY"
ROTEL_OTEL_RESOURCE_ATTRIBUTES="service.name=my-lambda-api,deployment.environment=production"

Then set the environment variable to point to this file:

ROTEL_ENV_FILE=/var/task/rotel.env

You can store the API key in AWS Secrets Manager or Parameter Store and reference its ARN in rotel.env. The examples below use fully managed authentication and assume the secret contains the API key only; Rotel adds the Bearer prefix from the configuration. For a self-managed collector using an ingestion key, omit the Bearer prefix. Replace the example region, account, and secret names with your own:AWS Secrets Manager Example:

ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_OTEL_ENDPOINT:4318
ROTEL_OTLP_EXPORTER_PROTOCOL=http
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=Bearer ${arn:aws:secretsmanager:us-east-1:123456789012:secret:observability-api-key-abc123}"

AWS Parameter Store Example:

ROTEL_OTLP_EXPORTER_ENDPOINT=https://YOUR_OTEL_ENDPOINT:4318
ROTEL_OTLP_EXPORTER_PROTOCOL=http
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=Bearer ${arn:aws:ssm:us-east-1:123456789012:parameter/observability-api-key}"

Required IAM Permissions:Add these permissions to your Lambda execution role:For Secrets Manager:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:BatchGetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:observability-api-key-*"
    }
  ]
}

For Parameter Store:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters"
      ],
      "Resource": "arn:aws:ssm:us-east-1:123456789012:parameter/observability-api-key"
    }
  ]
}

Secret retrieval happens during initialization and can add cold-start latency. See the Rotel secret configuration for supported formats and permissions. Customer-managed encryption keys may also require kms:Decrypt permission.

Test the integration

Invoke your Lambda function with an event it accepts and ensure the handler writes a log. Replace this example payload if your function expects a different event:

aws lambda invoke \
  --function-name my-function \
  --cli-binary-format raw-in-base64-out \
  --payload '{"test": "data"}' \
  response.json

Check the Lambda logs for any errors:

aws logs tail /aws/lambda/my-function --follow

Verify logs

Open Search in ClickHouse Observability, select your Logs source, and filter for ServiceName:my-lambda-api if you used the example service name. Otherwise, use your Lambda function name. Select Last 15 minutes.Look for these key attributes in the logs:

service.name: Your configured service name, or the Lambda function name by default
faas.name: AWS Lambda function name
faas.invocation_id: Unique invocation ID
cloud.provider: “aws”
cloud.platform: “aws_lambda”

Disabling CloudWatch Logs (cost optimization)

By default, AWS Lambda sends all logs to CloudWatch Logs, which can be expensive at scale. Once you’ve verified that logs are flowing to ClickHouse Observability, you can disable CloudWatch logging to reduce costs.

Remove CloudWatch permissions from the execution role

Open the AWS Console and navigate to AWS Lambda
Navigate to your Lambda function
Select Configuration → Permissions
Click the execution role name to open the IAM console
Remove the CloudWatch Logs write permissions used by this function. If the role is shared, use a dedicated role before making this change:
- If using a custom policy, edit to remove logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents
- If using the AWS managed policy AWSLambdaBasicExecutionRole, remove it from the role
Save the role

Verify CloudWatch logging is disabled

Invoke your function again and verify that:

No new events appear in the function’s CloudWatch log group
Logs continue to appear in ClickHouse Observability

aws logs tail /aws/lambda/my-function --since 5m --follow

Adding OpenTelemetry auto-instrumentation

Rotel can receive telemetry from a language instrumentation layer installed alongside it. Follow the Rotel auto-instrumentation instructions and the linked language-layer documentation to choose a compatible layer and set AWS_LAMBDA_EXEC_WRAPPER. Preserve the Rotel layer and any existing layers when adding instrumentation. After invoking the function, select your Traces source in ClickHouse Observability and filter by the configured service name. Log/trace correlation requires the application instrumentation to include trace and span IDs in its logs.

Example applications

See the example Python app demonstrating the Rotel Lambda Extension:

Python + ClickHouse: Python application with manual OpenTelemetry instrumentation, sending traces and logs directly to ClickHouse

Join the Rotel community

If you have questions about Rotel, please join the Rotel Discord server and share your feedback or questions. Check out the Rotel Lambda Extension to contribute any improvements.

Additional resources

Rotel Lambda Extension: Source code and detailed documentation
Rotel Core: The lightweight OTEL data plane powering the extension

Introduction

Features

Administration

Guides

Integration partners

Self-managed

Monitoring AWS Lambda logs with ClickHouse Observability using Rotel

Prerequisites

Connection details

Integration with existing Lambda functions

Disabling CloudWatch Logs (cost optimization)

Adding OpenTelemetry auto-instrumentation

Example applications

Join the Rotel community

Additional resources

​Prerequisites

​Connection details

​Integration with existing Lambda functions

​Disabling CloudWatch Logs (cost optimization)

​Adding OpenTelemetry auto-instrumentation

​Example applications

​Join the Rotel community

​Additional resources

Prerequisites

Connection details

Integration with existing Lambda functions

Disabling CloudWatch Logs (cost optimization)

Adding OpenTelemetry auto-instrumentation

Example applications

Join the Rotel community

Additional resources