> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Migrating agents from Elastic

> Migrate Elastic agents to fully managed ClickHouse Observability, with separate connection settings for self-managed deployments.

## Migrating agents from Elastic

The Elastic Stack provides a number of Observability data collection agents. Specifically:

* The [Beats family](https://www.elastic.co/beats) - such as [Filebeat](https://www.elastic.co/beats/filebeat), [Metricbeat](https://www.elastic.co/beats/metricbeat), and [Packetbeat](https://www.elastic.co/beats/packetbeat) - all based on the `libbeat` library. These Beats support [sending data to Elasticsearch, Kafka, Redis, or Logstash](https://www.elastic.co/docs/reference/beats/filebeat/configuring-output) over the Lumberjack protocol.
* The [`Elastic Agent`](https://www.elastic.co/elastic-agent) provides a unified agent capable of collecting logs, metrics, and traces. This agent can be centrally managed via the [Elastic Fleet Server](https://www.elastic.co/docs/reference/fleet/manage-elastic-agents-in-fleet) and supports output to Elasticsearch, Logstash, Kafka, or Redis.
* Elastic also provides a distribution of the [OpenTelemetry Collector - EDOT](https://www.elastic.co/docs/reference/opentelemetry). While it currently can't be orchestrated by the Fleet Server, it offers a more flexible and open path if you're migrating to ClickHouse Observability.

The best migration path depends on the agents currently in use. In the sections that follow, we document migration options for each major agent type. Our goal is to minimize friction and, where possible, allow you to continue using your existing agents during the transition.

## Preferred migration path

Where possible we recommend migrating to the [OpenTelemetry (OTel) Collector](https://opentelemetry.io/docs/collector/) for all log, metric, and trace collection, deploying the collector at the [edge in an agent role](https://clickhouse.com/docs/clickstack/self-configured/ingesting-data/collector.md#collector-roles-self-configured). This represents the most efficient means of sending data and avoids architectural complexity and data transformation.

> [!NOTE]
>
> **Why OpenTelemetry Collector?**
>
> The OpenTelemetry Collector provides a sustainable and vendor-neutral solution for observability data ingestion. We recognize that some organizations operate fleets of thousands—or even tens of thousands—of Elastic agents. For these users, maintaining compatibility with existing agent infrastructure may be critical. This documentation is designed to support this, while also helping teams gradually transition to OpenTelemetry-based collection.

## ClickHouse OpenTelemetry endpoint

Configure the destination before following the agent migration steps. The Vector and EDOT examples below send OpenTelemetry data over OTLP HTTP.

> [tab] Fully managed
>
> [Beta offering](https://clickhouse.com/docs/reference/settings/beta-and-experimental-features.md#beta-features)
>
> Copy your OTLP HTTP endpoint and API key from **Team Settings → Data → Managed Telemetry**. Set these variables in the environment where Vector or EDOT runs:
>
> ```shell theme={null}
> export OTEL_EXPORTER_OTLP_ENDPOINT='https://YOUR_OTEL_ENDPOINT:4318'
> export OTEL_EXPORTER_OTLP_AUTHORIZATION='Bearer YOUR_API_KEY'
> ```
>
> Use the exact endpoint, including its port, without a `/v1/logs` suffix. See [Sending OpenTelemetry data](https://clickhouse.com/docs/clickstack/observability/ingesting-data/opentelemetry.md) for connection details.

> [tab] Self-managed
>
> Deploy your receiving collector using the [self-managed setup guide](https://clickhouse.com/docs/clickstack/self-configured/getting-started/managed.md). Set its reachable OTLP HTTP endpoint and authorization header in the environment where Vector or EDOT runs:
>
> ```shell theme={null}
> export OTEL_EXPORTER_OTLP_ENDPOINT='http://YOUR_COLLECTOR_HOST:4318'
> export OTEL_EXPORTER_OTLP_AUTHORIZATION='YOUR_COLLECTOR_TOKEN'
> ```
>
> If you secured the receiving collector with `OTLP_AUTH_TOKEN`, use that token as the authorization value. If authentication is disabled, omit the `authorization` header from the examples. See [Securing the collector](https://clickhouse.com/docs/clickstack/self-configured/ingesting-data/collector.md#securing-the-collector-self-configured).
>
> Use `localhost` only when the sender and receiver share the same network namespace. For a TLS-enabled receiver, use its HTTPS endpoint and configure trust for its certificate.

## Migrating from beats

Users with extensive Beat deployments may wish to retain these when migrating to ClickHouse Observability.

**Currently this option has only been tested with Filebeat, and is therefore appropriate for Logs only.**

Beats agents use the [Elastic Common Schema (ECS)](https://www.elastic.co/docs/reference/ecs), which is currently [in the process of being merged into the OpenTelemetry](https://github.com/open-telemetry/opentelemetry-specification/blob/main/oteps/0199-support-elastic-common-schema-in-opentelemetry.md) specification used by ClickHouse Observability. However, these [schemas still differ significantly](https://www.elastic.co/docs/reference/ecs/ecs-otel-alignment-overview), and users are currently responsible for transforming ECS-formatted events into OpenTelemetry format before ingestion into ClickHouse Observability.

We recommend performing this transformation using [Vector](https://vector.dev), a lightweight and high-performance observability data pipeline that supports a powerful transformation language called Vector Remap Language (VRL).

If your Filebeat agents are configured to send data to Kafka - a supported output by Beats - Vector can consume those events from Kafka, apply schema transformations using VRL, and then forward them via OTLP to your configured OTLP endpoint.

Alternatively, Vector also supports receiving events over the Lumberjack protocol used by Logstash. This enables Beats agents to send data directly to Vector, where the same transformation process can be applied before forwarding to your configured OTLP endpoint via OTLP.

We illustrate both of these architectures below.

![Beats agents sending logs through Vector to the collector, ClickHouse, and ClickStack UI](https://mintcdn.com/private-7c7dfe99/RGhcmZvoJO57c5Gc/images/use-cases/observability/clickstack-migrating-agents-ui.png?fit=max\&auto=format\&n=RGhcmZvoJO57c5Gc\&q=85\&s=8212b4b049f62156080f7c839b10ae6a)

In the following example, we provide the initial steps to configure Vector to receive log events from Filebeat via the Lumberjack protocol. We provide VRL for mapping the inbound ECS events to OTel specification, before sending these to your configured OTLP endpoint via OTLP. Users consuming events from Kafka can replace the Vector Logstash source with the [Kafka source](https://vector.dev/docs/reference/configuration/sources/kafka/) - all other steps remain the same.

**1. Install vector**

Install Vector using the [official installation guide](https://vector.dev/docs/setup/installation/).

Install Vector on a host reachable by your Filebeat agents.

You can follow best practices with regards to architecture and security when [moving Vector to production](https://vector.dev/docs/setup/going-to-prod/).

**2. Configure vector**

Vector should be configured to receive events over the Lumberjack protocol, imitating a Logstash instance. This can be achieved by configuring a [`logstash` source](https://vector.dev/docs/reference/configuration/sources/logstash/) for Vector:

```yaml theme={null}
sources:
  beats:
    type: logstash
    address: 0.0.0.0:5044
    tls:
      enabled: false  # Set to true if you're using TLS
      # The files below are generated from the steps at https://www.elastic.co/docs/reference/fleet/secure-logstash-connections#generate-logstash-certs
      # crt_file: logstash.crt
      # key_file: logstash.key
      # ca_file: ca.crt
      # verify_certificate: true
```

> [!NOTE]
>
> **TLS configuration**
>
> If Mutual TLS is required, generate certificates and keys using the Elastic guide ["Configure SSL/TLS for the Logstash output"](https://www.elastic.co/docs/reference/fleet/secure-logstash-connections#use-ls-output). These can then be specified in the configuration as shown above.

Events will be received in ECS format. These can be converted to the OpenTelemetry schema using a Vector Remap Language (VRL) transformer. Configuration of this transformer is simple - with the script file held in a separate file:

```yaml theme={null}
transforms:
  remap_filebeat:
    inputs: ["beats"]
    type: "remap"
    file: 'beat_to_otel.vrl'
```

Note it receives events from the above `beats` source. Our remap script is shown below. This script has been tested with log events only but can form the basis for other formats.

> [accordion] VRL - ECS to OTel
>
> ```javascript theme={null}
> # Define keys to ignore at root level
> ignored_keys = ["@metadata"]
>
> # Define resource key prefixes
> resource_keys = ["host", "cloud", "agent", "service"]
>
> # Create separate objects for resource and log record fields
> resource_obj = {}
> log_record_obj = {}
>
> # Copy all non-ignored root keys to appropriate objects
> root_keys = keys(.)
> for_each(root_keys) -> |_index, key| {
>     if !includes(ignored_keys, key) {
>         val, err = get(., [key])
>         if err == null {
>             # Check if this is a resource field
>             is_resource = false
>             if includes(resource_keys, key) {
>                 is_resource = true
>             }
>
>             # Add to appropriate object
>             if is_resource {
>                 resource_obj = set(resource_obj, [key], val) ?? resource_obj
>             } else {
>                 log_record_obj = set(log_record_obj, [key], val) ?? log_record_obj
>             }
>         }
>     }
> }
>
> # Flatten both objects separately
> flattened_resources = flatten(resource_obj, separator: ".")
> flattened_logs = flatten(log_record_obj, separator: ".")
>
> # Process resource attributes
> resource_attributes = []
> resource_keys_list = keys(flattened_resources)
> for_each(resource_keys_list) -> |_index, field_key| {
>     field_value, err = get(flattened_resources, [field_key])
>     if err == null && field_value != null {
>         attribute, err = {
>             "key": field_key,
>             "value": {
>                 "stringValue": to_string(field_value)
>             }
>         }
>         if (err == null) {
>             resource_attributes = push(resource_attributes, attribute)
>         }
>     }
> }
>
> # Process log record attributes
> log_attributes = []
> log_keys_list = keys(flattened_logs)
> for_each(log_keys_list) -> |_index, field_key| {
>     field_value, err = get(flattened_logs, [field_key])
>     if err == null && field_value != null {
>         attribute, err = {
>             "key": field_key,
>             "value": {
>                 "stringValue": to_string(field_value)
>             }
>         }
>         if (err == null) {
>             log_attributes = push(log_attributes, attribute)
>         }
>     }
> }
>
> # Get timestamp for timeUnixNano (convert to nanoseconds)
> timestamp_nano = if exists(.@timestamp) {
>     to_unix_timestamp!(parse_timestamp!(.@timestamp, format: "%Y-%m-%dT%H:%M:%S%.3fZ"), unit: "nanoseconds")
> } else {
>     to_unix_timestamp(now(), unit: "nanoseconds")
> }
>
> # Get message/body field
> body_value = if exists(.message) {
>     to_string!(.message)
> } else if exists(.body) {
>     to_string!(.body)
> } else {
>     ""
> }
>
> # Create the OpenTelemetry structure
> . = {
>     "resourceLogs": [
>         {
>             "resource": {
>                 "attributes": resource_attributes
>             },
>             "scopeLogs": [
>                 {
>                     "scope": {},
>                     "logRecords": [
>                         {
>                             "timeUnixNano": to_string(timestamp_nano),
>                             "severityNumber": 9,
>                             "severityText": "info",
>                             "body": {
>                                 "stringValue": body_value
>                             },
>                             "attributes": log_attributes
>                         }
>                     ]
>                 }
>             ]
>         }
>     ]
> }
> ```

Finally, send the transformed events to your configured OTLP endpoint. This requires the configuration of a OTLP sink in Vector, which takes events from the `remap_filebeat` transform as input:

```yaml theme={null}
sinks:
  otlp:
    type: opentelemetry
    inputs: [remap_filebeat] # receives events from a remap transform - see below
    protocol:
      type: http
      uri: "${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/logs"
      method: post
      encoding:
        codec: json
      framing:
        method: newline_delimited
      headers:
        content-type: application/json
        authorization: "${OTEL_EXPORTER_OTLP_AUTHORIZATION}"
```

Use the [connection variables](#clickhouse-otel-endpoint-self-configured) for your deployment. The Vector URI includes `/v1/logs` because this output sends log records.

Our final complete configuration is shown below:

```yaml theme={null}
sources:
  beats:
    type: logstash
    address: 0.0.0.0:5044
    tls:
      enabled: false  # Set to true if you're using TLS
        #crt_file: /data/elasticsearch-9.0.1/logstash/logstash.crt
        #key_file: /data/elasticsearch-9.0.1/logstash/logstash.key
        #ca_file: /data/elasticsearch-9.0.1/ca/ca.crt
        #verify_certificate: true

transforms:
  remap_filebeat:
    inputs: ["beats"]
    type: "remap"
    file: 'beat_to_otel.vrl'

sinks:
  otlp:
    type: opentelemetry
    inputs: [remap_filebeat]
    protocol:
      type: http
      uri: "${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/logs"
      method: post
      encoding:
        codec: json
      framing:
        method: newline_delimited
      headers:
        content-type: application/json
        authorization: "${OTEL_EXPORTER_OTLP_AUTHORIZATION}"
```

**3. Configure Filebeat**

Configure Filebeat to use the Logstash output for Vector, replacing any existing output configuration. Replace `YOUR_VECTOR_HOST` with the host running Vector; use `localhost` only when both run on the same host. Configure TLS to match the Vector input:

```yaml theme={null}
# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["YOUR_VECTOR_HOST:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"
```

## Migrating from Elastic Agent

The Elastic Agent consolidates the different Elastic Beats into a single package. This agent integrates with [Elastic Fleet](https://www.elastic.co/docs/reference/fleet/fleet-server), allowing it to be centrally orchestrated and configured.

Users with Elastic Agents deployed have several migration paths:

* Configure the agent to send to a Vector endpoint over the Lumberjack protocol. **This has currently been tested for users collecting log data with the Elastic Agent only.** This can be centrally configured via the Fleet UI in Kibana.
* [Run the agent as Elastic OpenTelemetry Collector (EDOT)](https://www.elastic.co/docs/reference/fleet/otel-agent). The Elastic Agent includes an embedded EDOT Collector that allows you to instrument your applications and infrastructure once and send data to multiple vendors and backends. In this configuration, you can simply configure the EDOT collector to forward events to your configured OTLP endpoint over OTLP. **This approach supports all event types.**

We demonstrate both of these options below.

### Sending data via Vector

**1. Install and configure Vector**

Install and configure Vector using the [same steps](#install-vector-self-configured) as those documented for migrating from Filebeat.

**2. Configure Elastic Agent**

Elastic Agent needs to be configured to send data via the Logstash protocol Lumberjack. This is a [supported deployment pattern](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures/ls-networkbridge) and can either be configured centrally or [via the agent configuration file `elastic-agent.yaml`](https://www.elastic.co/docs/reference/fleet/logstash-output) if deploying without Fleet.

Central configuration through Kibana can be achieved by adding [an Output to Fleet](https://www.elastic.co/docs/reference/fleet/fleet-settings#output-settings).

![Add Logstash output](https://mintcdn.com/private-7c7dfe99/uAzQoYEGwuyhrO3W/images/use-cases/observability/add-logstash-output.webp?fit=max\&auto=format\&n=uAzQoYEGwuyhrO3W\&q=85\&s=be535c92abf9f740bb90fde2e86b8ac9)

This output can then be used in an [agent policy](https://www.elastic.co/docs/reference/fleet/agent-policy). This will automatically mean any agents using the policy will send their data to Vector.

![Agent settings](https://mintcdn.com/private-7c7dfe99/uAzQoYEGwuyhrO3W/images/use-cases/observability/agent-output-settings.webp?fit=max\&auto=format\&n=uAzQoYEGwuyhrO3W\&q=85\&s=187b3aca721ffb3e6a592b0fe4ff452e)

Since this requires secure communication over TLS to be configured, we recommend the guide ["Configure SSL/TLS for the Logstash output"](https://www.elastic.co/docs/reference/fleet/secure-logstash-connections#use-ls-output), which can be followed with the user assuming their Vector instance assumes the role of Logstash.

Note that this requires users to configure the Logstash source in Vector to also mutual TLS. Use the keys and certificates [generated in the guide](https://www.elastic.co/docs/reference/fleet/secure-logstash-connections#generate-logstash-certs) to configure the input appropriately.

```yaml theme={null}
sources:
  beats:
    type: logstash
    address: 0.0.0.0:5044
    tls:
      enabled: true  # Set to true if you're using TLS.
      # The files below are generated from the steps at https://www.elastic.co/docs/reference/fleet/secure-logstash-connections#generate-logstash-certs
      crt_file: logstash.crt
      key_file: logstash.key
      ca_file: ca.crt
      verify_certificate: true
```

### Run Elastic Agent as OpenTelemetry collector

The Elastic Agent includes an embedded EDOT Collector that allows you to instrument your applications and infrastructure once and send data to multiple vendors and backends.

> [!NOTE]
>
> **Agent integrations and orchestration**
>
> Users running the EDOT collector distributed with Elastic Agent won't be able to exploit the [existing integrations offered by the agent](https://www.elastic.co/docs/reference/fleet/manage-integrations). Additionally, the collector can't be centrally managed by Fleet - forcing the user to run the [agent in standalone mode](https://www.elastic.co/docs/reference/fleet/configure-standalone-elastic-agents), managing configuration themselves.

To run the Elastic Agent with the EDOT collector, follow the [official Elastic guide](https://www.elastic.co/docs/reference/fleet/otel-agent-transform). Configure an OTLP HTTP exporter using your [connection variables](#clickhouse-otel-endpoint-self-configured):

```yaml theme={null}
exporters:
  otlphttp/clickhouse:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
```

Add `otlphttp/clickhouse` to the `exporters` list in each logs, metrics, or traces pipeline you want to forward. Preserve the existing receivers and processors. For parallel ingestion, retain the Elastic exporter in those pipelines; remove it when you are ready to stop sending to Elastic.

The exporter uses the base endpoint and appends the signal path automatically. For an HTTPS receiver using a private certificate authority, configure `tls.ca_file`; if that receiver requires mutual TLS, also configure `tls.cert_file` and `tls.key_file`. These settings secure the EDOT-to-OTLP connection independently of the Filebeat-to-Vector TLS configuration.

## Migrating from the Elastic OpenTelemetry collector

Users already running the [Elastic OpenTelemetry Collector (EDOT)](https://www.elastic.co/docs/reference/opentelemetry) can simply reconfigure their agents to send to your configured OTLP endpoint via OTLP. The steps involved are identical to those outlined above for running the [Elastic Agent as an OpenTelemetry collector](#run-agent-as-otel-self-configured). This approach can be used for all data types.