Migrating data to ClickHouse Observability from Elastic

This guide focuses on migrating from Elastic to fully managed ClickHouse Observability, the default experience. ClickHouse manages the underlying schemas and storage. Your ingestion pipeline must send telemetry in OpenTelemetry Protocol (OTLP) format; Elasticsearch documents cannot be imported directly into the fully managed service. For self-managed deployments, review the equivalent Elasticsearch and ClickHouse concepts and type mappings. Self-managed data migration instructions are labeled separately below.

Parallel operation strategy

When migrating from Elastic to ClickHouse Observability for observability use cases, we recommend a parallel operation approach rather than attempting to migrate historical data. This strategy offers several advantages:

Minimal risk: by running both systems concurrently, you maintain access to existing data and dashboards while validating ClickHouse Observability and familiarizing your users with the new system.
Natural data expiration: most observability data has a limited retention period (typically 30 days or less), allowing for a natural transition as data expires from Elastic.
Simplified migration: no need for complex data transfer tools or processes to move historical data between systems.

Migrating dataSelf-managed deployments can use the Elasticsearch table engine to read historical documents and copy them into ClickHouse tables. See Migrating data. This is separate from the OTLP ingestion path used by fully managed ClickHouse Observability.

Implementation steps

Configure dual ingestion

Set up your data collection pipeline to send data to both Elastic and ClickHouse Observability simultaneously. For fully managed ClickHouse Observability, the pipeline must export OTLP to the managed endpoint using your ingestion API key. Forwarding Elasticsearch JSON documents to that endpoint is not sufficient.How this is achieved depends on your current agents for collection - see “Migrating Agents”.

Adjust retention periods

Review the retention periods in both systems so historical data remains available during the transition. See Retention and TTL for deployment-specific guidance.

Validate and compare

Run queries against both systems to ensure data consistency
Compare query performance and results
Migrate dashboards and alerts to ClickHouse Observability. This is currently a manual process.
Verify that all critical dashboards and alerts work as expected in ClickHouse Observability

Gradual transition

As data naturally expires from Elastic, you will increasingly rely on ClickHouse Observability
Once confidence in ClickHouse Observability is established, you can begin redirecting queries and dashboards

Long-term retention

For organizations requiring longer retention periods:

Continue running both systems in parallel until all data has expired from Elastic
For self-managed deployments, consider using materialized views to maintain aggregated or filtered historical data while allowing raw data to expire.

Migration timeline

The migration timeline will depend on your data retention requirements:

30-day retention: Migration can be completed within a month.
Longer retention: Continue parallel operation until data expires from Elastic.
Historical data: Review the deployment-specific guidance in Migrating data before planning an import.

Retention and TTL

Fully managed
Self-managed

Fully managed ClickHouse Observability retains data for 30 days during beta. Retention will be updated at general availability (GA), with further instructions to follow.

Plan the transition around this 30-day retention period and your retention in Elastic. Keep Elastic available for historical searches until that data is no longer required.

ClickHouse uses TTL clauses on MergeTree tables to automatically delete expired data.Align your table retention with your migration requirements. For examples, see self-managed production TTL setup.

Migrating data

Fully managed
Self-managed

There is currently no direct historical-data migration path from Elasticsearch to fully managed ClickHouse Observability. The managed endpoint accepts OTLP telemetry, not Elasticsearch documents or JSON exports. The Elasticsearch table engine workflow in the self-managed tab does not apply to fully managed services.

Use parallel ingestion to send new telemetry in OTLP format to your managed service while retaining historical data in Elastic. See Ingesting with OpenTelemetry for the managed endpoint and authentication settings, and Migrating agents for collection options.If historical data must be imported, you need a custom pipeline that converts Elasticsearch documents into appropriate OpenTelemetry logs, traces, or metrics and exports them over OTLP. This guide does not provide a ready-made converter. Account for the managed service’s retention period before planning a backfill.

Use the Elasticsearch table engine to read documents directly from Elasticsearch, then copy them into a local ClickHouse table with INSERT INTO ... SELECT. No export file or elasticdump installation is required.The engine is read-only: creating an Elasticsearch table exposes remote documents for querying but does not copy them into ClickHouse storage.

The Elasticsearch integration is experimental and requires enable_elasticsearch = 1 on a ClickHouse version that supports it. It requires Elasticsearch 7.12 or later and does not support OpenSearch. Your ClickHouse service must be able to reach the Elasticsearch endpoint, and the index must have _source enabled to read document contents.

Connect to the Elasticsearch index

Run the following SQL in your self-managed ClickHouse Cloud service, replacing the endpoint, index, and credentials with your own values:

SET enable_elasticsearch = 1;

CREATE TABLE elasticsearch_source
ENGINE = Elasticsearch(
    'https://<elasticsearch-host>:9200',
    '<index-name>',
    '<username>',
    '<password>'
);

SELECT _index, _id, document
FROM elasticsearch_source
LIMIT 10;

The table exposes each document’s _source as the document column of type JSON. The virtual columns _id and _index identify the source document and index. Inspect a sample before choosing the destination schema.The engine also supports API keys and named collections. See the authentication reference.

Copy documents into ClickHouse

This example stores documents as JSON in a staging table so you can inspect and transform them before configuring an observability source. Run the statements in the same database as elasticsearch_source:

CREATE TABLE migrated_elasticsearch_documents
(
    source_index String,
    document_id String,
    document JSON
)
ENGINE = MergeTree
ORDER BY (source_index, document_id);

INSERT INTO migrated_elasticsearch_documents
SELECT _index, _id, document
FROM elasticsearch_source;

Each read uses an Elasticsearch point in time to provide a consistent snapshot for that query. This is a one-time copy, not continuous replication. Repeating the insert appends rows again; the ordering key does not deduplicate documents.For larger imports, split the work into non-overlapping batches with the engine’s query argument. Use absolute timestamp boundaries in the Elasticsearch Query DSL. A ClickHouse WHERE clause does not push filters down to Elasticsearch. See the engine examples and limitations.

Map and validate the imported data

Compare document counts and representative records with the source snapshot. Check timestamps, missing fields, and nested values before using the data for searches or dashboards. The JSON type can infer types from values, including timestamps, so their text representation may change. Use explicit JSON type hints when you need to control field types.Design the destination observability schema for your queries and retention requirements, then use INSERT INTO ... SELECT to extract and convert fields from the staging table. Review the Elasticsearch and ClickHouse concepts, type mappings, and schema design guidance.Copying documents does not automatically convert them to the OpenTelemetry schema or configure a source in the UI. Configure the table and its timestamp, message, and other field mappings using Configuration options.

Introduction

Features

Administration

Guides

Integration partners

Self-managed

Migrating data to ClickHouse Observability from Elastic

Parallel operation strategy

Implementation steps

Long-term retention

Migration timeline

Retention and TTL

Migrating data

​Parallel operation strategy

​Implementation steps

​Long-term retention

​Migration timeline

​Retention and TTL

​Migrating data

Parallel operation strategy

Implementation steps

Long-term retention

Migration timeline

Retention and TTL

Migrating data