> ## Documentation Index > Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Migrating data to ClickHouse Observability from Elastic > Plan parallel ingestion from Elastic to fully managed ClickHouse Observability, with separate guidance for self-managed historical-data migration. This guide focuses on migrating from Elastic to **fully managed ClickHouse Observability**, the default experience. ClickHouse manages the underlying schemas and storage. Your ingestion pipeline must send telemetry in **OpenTelemetry Protocol (OTLP)** format; Elasticsearch documents cannot be imported directly into the fully managed service. For self-managed deployments, review the [equivalent Elasticsearch and ClickHouse concepts](https://clickhouse.com/docs/clickstack/self-configured/migration/elastic/concepts.md) and [type mappings](https://clickhouse.com/docs/clickstack/self-configured/migration/elastic/types.md). Self-managed data migration instructions are labeled separately below. ## Parallel operation strategy When migrating from Elastic to ClickHouse Observability for observability use cases, we recommend a **parallel operation** approach rather than attempting to migrate historical data. This strategy offers several advantages: 1. **Minimal risk**: by running both systems concurrently, you maintain access to existing data and dashboards while validating ClickHouse Observability and familiarizing your users with the new system. 2. **Natural data expiration**: most observability data has a limited retention period (typically 30 days or less), allowing for a natural transition as data expires from Elastic. 3. **Simplified migration**: no need for complex data transfer tools or processes to move historical data between systems. > [!NOTE] > > **Migrating data** > > Self-managed deployments can use the `Elasticsearch` table engine to read historical documents and copy them into ClickHouse tables. See [Migrating data](#migrating-data-self-configured). This is separate from the OTLP ingestion path used by fully managed ClickHouse Observability. ### Implementation steps **1. Configure dual ingestion** Set up your data collection pipeline to send data to both Elastic and ClickHouse Observability simultaneously. For fully managed ClickHouse Observability, the pipeline must export OTLP to the managed endpoint using your ingestion API key. Forwarding Elasticsearch JSON documents to that endpoint is not sufficient. How this is achieved depends on your current agents for collection - see ["Migrating Agents"](https://clickhouse.com/docs/clickstack/observability/migration/elastic/migrating-agents.md). **2. Adjust retention periods** Review the retention periods in both systems so historical data remains available during the transition. See [Retention and TTL](#retention-and-ttl-self-configured) for deployment-specific guidance. **3. Validate and compare** * Run queries against both systems to ensure data consistency * Compare query performance and results * Migrate dashboards and alerts to ClickHouse Observability. This is currently a manual process. * Verify that all critical dashboards and alerts work as expected in ClickHouse Observability **4. Gradual transition** * As data naturally expires from Elastic, you will increasingly rely on ClickHouse Observability * Once confidence in ClickHouse Observability is established, you can begin redirecting queries and dashboards ### Long-term retention For organizations requiring longer retention periods: * Continue running both systems in parallel until all data has expired from Elastic * For self-managed deployments, consider using [materialized views](https://clickhouse.com/docs/concepts/features/materialized-views/incremental-materialized-view.md) to maintain aggregated or filtered historical data while allowing raw data to expire. ### Migration timeline The migration timeline will depend on your data retention requirements: * **30-day retention**: Migration can be completed within a month. * **Longer retention**: Continue parallel operation until data expires from Elastic. * **Historical data**: Review the deployment-specific guidance in [Migrating data](#migrating-data-self-configured) before planning an import. ## Retention and TTL > [tab] Fully managed > > [Beta offering](https://clickhouse.com/docs/reference/settings/beta-and-experimental-features.md#beta-features) > > > [!NOTE] > > > > Fully managed ClickHouse Observability retains data for 30 days during beta. Retention will be updated at general availability (GA), with further instructions to follow. > > Plan the transition around this 30-day retention period and your retention in Elastic. Keep Elastic available for historical searches until that data is no longer required. > [tab] Self-managed > > ClickHouse uses [TTL clauses](https://clickhouse.com/docs/clickstack/self-configured/managing/production.md#configure-ttl-managed) on `MergeTree` tables to automatically delete expired data. > > Align your table retention with your migration requirements. For examples, see [self-managed production TTL setup](https://clickhouse.com/docs/clickstack/self-configured/managing/production.md#configure-ttl-managed). ## Migrating data > [tab] Fully managed > > [Beta offering](https://clickhouse.com/docs/reference/settings/beta-and-experimental-features.md#beta-features) > > > [!NOTE] > > > > There is currently no direct historical-data migration path from Elasticsearch to fully managed ClickHouse Observability. The managed endpoint accepts OTLP telemetry, not Elasticsearch documents or JSON exports. The `Elasticsearch` table engine workflow in the self-managed tab does not apply to fully managed services. > > Use parallel ingestion to send new telemetry in **OTLP format** to your managed service while retaining historical data in Elastic. See [Ingesting with OpenTelemetry](https://clickhouse.com/docs/clickstack/observability/ingesting-data/opentelemetry.md) for the managed endpoint and authentication settings, and [Migrating agents](https://clickhouse.com/docs/clickstack/observability/migration/elastic/migrating-agents.md) for collection options. > > If historical data must be imported, you need a custom pipeline that converts Elasticsearch documents into appropriate OpenTelemetry logs, traces, or metrics and exports them over OTLP. This guide does not provide a ready-made converter. Account for the managed service's retention period before planning a backfill. > [tab] Self-managed > > Use the [`Elasticsearch` table engine](https://clickhouse.com/docs/reference/engines/table-engines/integrations/elasticsearch) to read documents directly from Elasticsearch, then copy them into a local ClickHouse table with `INSERT INTO ... SELECT`. No export file or `elasticdump` installation is required. > > The engine is read-only: creating an `Elasticsearch` table exposes remote documents for querying but does not copy them into ClickHouse storage. > > > [!NOTE] > > > > The Elasticsearch integration is experimental and requires `enable_elasticsearch = 1` on a ClickHouse version that supports it. It requires Elasticsearch 7.12 or later and does not support OpenSearch. Your ClickHouse service must be able to reach the Elasticsearch endpoint, and the index must have `_source` enabled to read document contents. > > **1. Connect to the Elasticsearch index** > > Run the following SQL in your self-managed ClickHouse Cloud service, replacing the endpoint, index, and credentials with your own values: > > ```sql theme={null} > SET enable_elasticsearch = 1; > > CREATE TABLE elasticsearch_source > ENGINE = Elasticsearch( > 'https://:9200', > '', > '', > '' > ); > > SELECT _index, _id, document > FROM elasticsearch_source > LIMIT 10; > ``` > > The table exposes each document's `_source` as the `document` column of type `JSON`. The virtual columns `_id` and `_index` identify the source document and index. Inspect a sample before choosing the destination schema. > > The engine also supports API keys and named collections. See the [authentication reference](https://clickhouse.com/docs/reference/functions/table-functions/elasticsearch.md#authentication). > > **2. Copy documents into ClickHouse** > > This example stores documents as `JSON` in a staging table so you can inspect and transform them before configuring an observability source. Run the statements in the same database as `elasticsearch_source`: > > ```sql theme={null} > CREATE TABLE migrated_elasticsearch_documents > ( > source_index String, > document_id String, > document JSON > ) > ENGINE = MergeTree > ORDER BY (source_index, document_id); > > INSERT INTO migrated_elasticsearch_documents > SELECT _index, _id, document > FROM elasticsearch_source; > ``` > > Each read uses an Elasticsearch point in time to provide a consistent snapshot for that query. This is a one-time copy, not continuous replication. Repeating the insert appends rows again; the ordering key does not deduplicate documents. > > For larger imports, split the work into non-overlapping batches with the engine's `query` argument. Use absolute timestamp boundaries in the Elasticsearch Query DSL. A ClickHouse `WHERE` clause does not push filters down to Elasticsearch. See the [engine examples](https://clickhouse.com/docs/reference/engines/table-engines/integrations/elasticsearch.md#usage-example) and [limitations](https://clickhouse.com/docs/reference/functions/table-functions/elasticsearch.md#limitations). > > **3. Map and validate the imported data** > > Compare document counts and representative records with the source snapshot. Check timestamps, missing fields, and nested values before using the data for searches or dashboards. The `JSON` type can infer types from values, including timestamps, so their text representation may change. Use explicit JSON type hints when you need to control field types. > > Design the destination observability schema for your queries and retention requirements, then use `INSERT INTO ... SELECT` to extract and convert fields from the staging table. Review the [Elasticsearch and ClickHouse concepts](https://clickhouse.com/docs/clickstack/self-configured/migration/elastic/concepts.md), [type mappings](https://clickhouse.com/docs/clickstack/self-configured/migration/elastic/types.md), and [schema design guidance](https://clickhouse.com/docs/guides/clickhouse/data-formats/json/schema.md). > > Copying documents does not automatically convert them to the OpenTelemetry schema or configure a source in the UI. Configure the table and its timestamp, message, and other field mappings using [Configuration options](https://clickhouse.com/docs/clickstack/self-configured/managing/config.md).