The [Postgres CDC connector in ClickPipes](https://clickhouse.com/cloud/clickpipes/postgres-cdc-connector) enables users to continuously replicate data from [Postgres](https://www.postgresql.org/) to [ClickHouse Cloud](https://clickhouse.com/cloud) for blazing-fast, real-time analytics. This connector supports both one-time migrations and continuous replication and is powered by [PeerDB](https://github.com/PeerDB-io/peerdb), an open-source Postgres CDC company [acquired](https://techcrunch.com/2024/07/30/real-time-database-startup-clickhouse-acquires-peerdb-to-expand-its-postgres-support/) by ClickHouse. Since its GA launch a few months ago, we’ve seen a surge in customer requests to extend these CDC capabilities to [TimescaleDB](https://github.com/timescale/timescaledb), the time-series Postgres extension maintained by TigerData.

## Use-cases

We see use cases that involve both one-time migrations and continuous replication. They primarily fall into three buckets:

1. **Online data migrations from TimescaleDB to ClickHouse:** In this use case, customers already run analytics on TimescaleDB. As their data or workloads (e.g., advanced analytics) scale, TimescaleDB no longer performs well without significant tuning and optimizations that require deep database/DBA expertise. These customers prefer a more turnkey solution that provides [significantly faster performance](https://benchmark.clickhouse.com/#system=+lik|saB&type=-&machine=-ca2|6t|gle|6ax|ae-|6ale|3al|gel&cluster_size=-&opensource=-&tuned=+n&metric=hot&queries=-). This is where ClickHouse has been compelling for them, as it is a purpose-built analytical database. To ensure a smooth migration, they seek a one-click online migration option from TimescaleDB to ClickHouse Cloud.

[![clickbench-cdc-blog-img.png](/uploads/clickbench_timescale_ch_cdc_f00a9e26bc.png)](https://benchmark.clickhouse.com/#system=+lik|saB&type=-&machine=-ca2|6t|gle|6ax|ae-|6ale|3al|gel&cluster_size=-&opensource=-&tuned=+n&metric=hot&queries=-)

2. **Iterative migration from TimescaleDB to ClickHouse**: In this use case, similar to the one above, a customer wants to migrate from TimescaleDB to ClickHouse but prefers to do so iteratively. Because the application is complex, they choose to migrate in stages \- moving a few workloads at a time, offloading reads first, and then migrating the write pipeline. The Postgres CDC connector is particularly useful in such scenarios, as it keeps both TimescaleDB and ClickHouse in sync with lag as low as a few seconds, while gradually migrating all workloads to ClickHouse. Here is a testimonial from one of our customers, Kindly, who falls under this category of iterative migration from TimescaleDB to ClickHouse.

> “The Postgres CDC connector made it easy for us to transition to ClickHouse Cloud without redesigning our entire pipeline. It has significantly improved dashboard performance and enabled faster data exploration, making our analytics more efficient and scalable.” - Team @ Kindly.ai

3. **Co-existence use case** – In this scenario, customers use both TimescaleDB and ClickHouse to power their real-time applications. For example, they run transactional or time-series workloads on TimescaleDB while using ClickHouse for fast, advanced analytics. Postgres CDC makes sure to reliably replicate operational data from TimescaleDB to ClickHouse for lightning-fast real-time analytics.

## Features and demo

The Postgres CDC connector in ClickPipes (and PeerDB) supports both initial load/backfill (initial snapshot of data) and ongoing sync/CDC from TimescaleDB Hypertables to ClickHouse. This is possible thanks to the native logical replication support in TimescaleDB,  a benefit of it being a Postgres extension. The connector supports both compressed and uncompressed Hypertables. Here are its main features:

1. **Blazing fast initial loads** – One of the flagship features of ClickPipes is [**parallel snapshotting**](https://clickhouse.com/docs/integrations/clickpipes/postgres/parallel_initial_load), which migrates a single large table using parallel threads. This enables moving terabytes of data in just a few hours instead of days. It works for **TimescaleDB hypertables,** as well, although It does not extend to compressed hypertables, which don’t support CTID columns. In such cases, ClickPipes falls back to single-threaded execution — which remains fast thanks to a range of micro-optimizations, including cursors and efficient data movement via Avro \+ Zstd, among others.  
2. **Support for both uncompressed and compressed hypertables** – The ClickPipes Postgres CDC connector works with both uncompressed and compressed TimescaleDB hypertables, ensuring seamless replication into ClickHouse.  
3. **Support for schema changes** \- The ClickPipes Postgres CDC connector also supports automatic replication of [schema changes](https://clickhouse.com/docs/integrations/clickpipes/postgres/schema-changes) from TimescaleDB including adding and dropping columns.  
4. **Comprehensive alerts and metrics** – ClickPipes provides extensive metrics including throughput, inserts/updates/deletes per table, latency, and Postgres-native metrics such as replication slot size over time and wait events during replication. It also offers advanced [alerts](https://clickhouse.com/blog/postgres-cdc-connector-clickpipes-public-beta#user-facing-alerts) through Slack or email for issues like replication slot growth or replication failures etc. The goal is to deliver deep visibility into the replication process and ensure an enterprise-ready experience.

<iframe width="768" height="432" src="https://www.youtube.com/embed/FebWvSnW9Hk?si=-FxKbzH9BySESI-H" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## How did we build it?

### Handling hypertables and chunks in logical replication

The TimescaleDB extension introduces **hypertables**—automatically partitioned tables designed to optimize query performance for time-series data. Like Postgres partitioned tables, hypertables don’t store data directly but do it in child tables called **chunks**. This design complicates logical replication, since changes must be tracked at the chunk level rather than the parent hypertable. It also requires handling scenarios where TimescaleDB automatically creates new chunks as data arrives.

For standard [Postgres partitioned tables](https://blog.peerdb.io/real-time-change-data-capture-for-postgres-partitioned-tables), logical replication relies on the [`publish_via_partition_root`](https://amitlan.com/writing/pg/partition-logical-replication/) option, which rewrites changes to appear as if they originated from the parent table. Hypertables, however, don’t support this option. ClickPipes (via PeerDB) therefore [performs the parent lookup explicitly](https://github.com/PeerDB-io/peerdb/blob/da537889fe9b9bbe84b85799385f3b576b5b34dd/flow/connectors/postgres/cdc.go#L1152) while processing changes. As long as the publication includes all child tables, this ensures that changes are routed correctly to the target ClickHouse table.

![timescalediagramfinal.png](/uploads/timescalediagramfinal_cb752d0018.png)

To manage newly created chunks, TimescaleDB stores them under the `_timescaledb_internal` schema. By adding this schema to the Postgres publication, we capture all future chunks. It requires a bit of setup upfront but, once configured, results in a **hands-off replication experience** with ClickPipes.

### Supporting Compression

Another TimescaleDB feature beyond vanilla Postgres is **hypertable compression**, enabled via transparent compression or the newer Hypercore hybrid row-columnar engine. These configurations bring compression benefits similar to columnar and time-series databases like ClickHouse.

However, compressed storage introduces challenges. Our parallel initial-load strategy using `ctid` partitions doesn’t work here and fails with errors such as:

`ERROR: transparent decompression only supports tableoid system column`

We recently [shipped an improvement](https://github.com/search?q=repo%3APeerDB-io%2Fpeerdb+timescaledb&type=pullrequests) to detect this scenario and automatically fall back to a code path that avoids relying on the `ctid` column, ensuring replication remains reliable even with compressed hypertables.

## Conclusion

To get started with replicating or migrating your TimescaleDB workloads to ClickHouse, you can follow the resources below.

1. [Steps to setup logical replication on TimescaleDB](https://clickhouse.com/docs/integrations/clickpipes/postgres/source/timescale)  
2. [ClickPipes for replicating TimescaleDB to ClickHouse Cloud](https://clickhouse.com/docs/integrations/clickpipes/postgres)  
3. [PeerDB for replication TimescaleDB to ClickHouse Open Source](https://github.com/PeerDB-io/peerdb)

*\* Third-party logos and trademarks belong to their respective owners and are used here for identification purposes only. No endorsement is implied.*

The Postgres CDC connector in ClickPipes now supports one-time migrations and continuous replication from TimescaleDB.

large_timescaledb-cdc-clickpipes-banner.png

medium_timescaledb-cdc-clickpipes-banner.png

small_timescaledb-cdc-clickpipes-banner.png

thumbnail_timescaledb-cdc-clickpipes-banner.png

timescaledb-cdc-clickpipes-banner.png

TimescaleDB to ClickHouse replication: Use cases, features, and how we built it

## Introduction

Data ingestion workloads come in all shapes and sizes, with more or less predictable patterns. When we built [ClickPipes](https://clickhouse.com/cloud/clickpipes), we wanted to enable customers to handle any throughput, data size and topologies from the most common building blocks of data infrastructure: object storage, message brokers, and databases. Today, hundreds of customers rely on ClickPipes as a cost-effective solution to manage real-time data ingestion into ClickHouse Cloud at scale, including [Property Finder](https://clickhouse.com/blog/how-property-finder-migrated-to-clickhouse), [Flock Safety](https://clickhouse.com/blog/why-flock-safety-turned-to-clickhouse), and [Seemplicity](https://clickhouse.com/blog/seemplicity-scaled-real-time-security-analytics-with-postgres-cdc-and-clickhouse).

As the product evolves, one of the most common requests has been to provide greater flexibility in configuring streaming ClickPipes to better suit the specific needs of these diverse ingestion workloads. In response, we’re introducing **new scaling options** that allow you to control both **horizontal and vertical scaling** for your streaming ClickPipes! You can now choose the number of replicas and replica sizes directly, along with improved monitoring to help you track resource usage over time.


### How does sizing work in streaming ClickPipes?

> **Note**: database and object storage ClickPipes have a different architecture, which doesn’t require direct control over compute allocation. This new feature applies exclusively to streaming ClickPipes to give customers more control over the cost-performance ratio per ClickPipe.

ClickPipes works by deploying replica(s) within ClickHouse Cloud, each acting as a consumer of your Kafka or Kinesis streaming data source. By default, ClickPipes starts with a single Extra Small replica (0.125 vCPU, 512 MiB RAM) to process your data streams. These replicas fetch data in parallel, process and transform it as needed, commit stream offsets, and write the results directly into your ClickHouse service. This architecture enables high-throughput, scalable ingestion with fault tolerance, and efficient load distribution across replicas.

![unnamed.png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_357981c8a1.png)

### What are replicas?

In the context of ClickPipes, replicas are instances of your data processing pipeline that work in parallel to handle the incoming data streams. Each replica acts as a consumer of your Kafka or Kinesis stream, allowing the system to scale efficiently and maintain performance as data volumes grow. Replicas can be scaled both vertically and horizontally to match the specific needs of your workload.


## Flexible scaling options

We've introduced two new scaling options to provide you with finer control over the topology of streaming ClickPipes: number of replicas (*horizontal scaling*) and replica size (*vertical scaling*). These scaling options can be selected in the UI (shown below) when creating a new ClickPipe or editing an existing one. Scaling is also supported via [OpenAPI](https://clickhouse.com/docs/cloud/manage/api/swagger#tag/ClickPipes/paths/~1v1~1organizations~1%7BorganizationId%7D~1services~1%7BserviceId%7D~1clickpipes~1%7BclickPipeId%7D~1scaling/patch) and [Terraform](https://github.com/ClickHouse/terraform-provider-clickhouse/blob/619ba02fc70e5d672e221f424a9aeedc43fa2d0a/examples/clickpipe/multiple_pipes_example/main.tf).

![unnamed.gif](https://cms.clickhouse-dev.com:1337/uploads/unnamed_8240b37fa9.gif)

### Vertical scaling

Vertical scaling, or *scaling up*, involves increasing the resources (CPU and memory) allocated to individual replicas within your ClickPipe. This is ideal for workloads that require more processing power per replica, such as Kafka or Kinesis streams with large payloads or complex schemas. Vertical scaling supports the following configurations:

| Replica size           | CPU          | Memory  |
|------------------------|--------------|--------|
| Extra Small (default)  | 0.125 Cores  | 512 Mb |
| Small                  | 0.25 Cores   | 1 Gb   |
| Medium                 | 0.5 Cores    | 2 Gb   |
| Large                  | 1 Core       | 4 Gb   |
| Extra Large            | 2 Cores      | 8 Gb   |

### Benchmarks for sizing

Below is a sample performance benchmark for a Large-sized ClickPipe replica (1 vCPU / 4 GB) ingesting data from a Kafka stream. You can use these benchmarks as a reference point when choosing the appropriate replica size for your workload. For more details and additional sizing guidance, refer to the documentation.

| Replica Size | Message Size | Data Format | Throughput |
|-------------|--------------|-------------|------------|
| Large       | 1.6 kb       | JSON        | 63 mb/s    |
| Large       | 1.6 kb       | AVRO        | 99 mb/s    |

### Horizontal scaling

Horizontal scaling, or *scaling out*, involves adding more replicas to your ClickPipe. This is highly effective for distributing workloads across multiple replicas, allowing your system to handle a higher volume of data concurrently. Kafka and Kinesis efficiently handle horizontal scaling by spreading data across multiple partitions and shards, respectively, which ClickPipes can handle by horizontally scaling proportionally.


## Enhanced resource monitoring

The details page for each ClickPipe now includes per-replica CPU and memory usage, showing average resource utilization across replicas. Additionally, the charts show the replicas CPU and memory limits — including *scale up* and *scale out* events — for easier tracking of utilization over time. This helps you better understand your workloads and plan resizing operations with confidence.

![unnamed (1).png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_1_301ef80db2.png)

## How does flexible scaling affect pricing?

Previously, streaming ClickPipes were priced at a flat rate of $0.05 per hour for each replica (by default, size Medium). With the introduction of configurable replica sizes, we're making Extra Small the default replica size, and updating our pricing model for streaming ClickPipes: the price now depends on both the replica size and the number of replicas you choose; starting at $0.0125. For full pricing details, refer to our [ClickPipes pricing documentation](https://clickhouse.com/docs/cloud/manage/billing/overview#clickpipes-pricing).

| Replica Size  | Compute Units | RAM      | vCPU | Price/hour (per replica) |
|---------------|----------------|---------|------|-------------------------|
| Extra Small   | 0.0625         | 512 MiB | 0.125| $0.0125                 |
| Small         | 0.125          | 1 GiB   | 0.25 | $0.025                  |
| Medium        | 0.25           | 2 GiB   | 0.5  | $0.05                   |
| Large         | 0.5            | 4 GiB   | 1.0  | $0.10                   |
| Extra Large   | 1.0            | 8 GiB   | 2.0  | $0.20                   |

> **Note**: In addition to compute charges, ClickPipes incurs a data ingestion cost of $0.04 per GB. For full pricing details, refer to our [ClickPipes pricing documentation](https://clickhouse.com/docs/cloud/manage/billing/overview#clickpipes-pricing).


## Next Steps

With flexible scaling and enhanced resource monitoring, you now have full control over the cost-performance ratio of your streaming ClickPipes, and can better prepare for changes in your data ingestion workloads. Head over to the [documentation](https://clickhouse.com/docs/integrations/clickpipes) to learn more about how to manage the deployment lifecycle of streaming ClickPipes.





Streaming ClickPipes now support flexible scaling and enhanced monitoring,  giving you full control over cost and performance for your evolving ingestion workloads.

large_vertical_scaling_clickpipes_3.jpg

medium_vertical_scaling_clickpipes_3.jpg

small_vertical_scaling_clickpipes_3.jpg

thumbnail_vertical_scaling_clickpipes_3.jpg

vertical_scaling_clickpipes_3.jpg

New: Flexible scaling and enhanced monitoring for streaming ClickPipes

Since its inception, ClickPipes has simplified the process of setting up robust continuous data pipelines to ClickHouse Cloud, allowing users to connect sources like Apache Kafka, Amazon S3, Google Cloud Storage, Amazon Kinesis, PostgreSQL, and now MySQL with just a few clicks.

This has enabled organizations to focus more on deriving insights from their data rather than managing complex ingestion processes. As of today, ClickPipes has successfully moved more than **20 trillion rows** to ClickHouse Cloud.

![0_clickpipes.png](/uploads/0_clickpipes_8ccd16e87f.png)

Today, we are introducing a new set of features designed to make ClickPipes a first-class tool in every developer's toolkit. These capabilities are built to support scale, automation, security, and observability in modern data architectures. 

But first, let’s start with usability improvements:

## Revamped UI

The ClickPipes user interface has been redesigned for greater clarity and efficiency. From configuring sources to monitoring pipeline health, every task is now more intuitive and discoverable, with polished UI animations and transitions.

![1_clickpipes.gif](/uploads/1_clickpipes_3969a2a564.gif)

## ClickPipes Editing

With support for editing, users can now modify running ClickPipes pipelines without deleting and recreating them. This includes updating ClickPipes names, schema mappings, and connection settings, allowing teams to iterate quickly while maintaining continuity of service.

![2_clickpipes.gif](/uploads/2_clickpipes_9f72aebfd0.gif)


## ClickPipes API and Terraform Provider

ClickPipes now includes dedicated API endpoints, part of the [ClickHouse Cloud OpenAPI](https://clickhouse.com/docs/cloud/manage/api/api-overview), and a Terraform provider. 
These tools support automated management of pipeline configurations and simplify integration into CI/CD pipelines.


<pre><code type='click-ui' language='json'>
{
  "name": "???? I was created using API!",
  "source": {
    "kafka": {
      "type": "confluent",
      "format": "JSONEachRow",
      "authentication": "PLAIN",
      "credentials": {
        "username": "xxx",
        "password": "xxx"
      },
      "brokers": "{{kafka_broker_url_and_port}}",
      "topics": "cell_towers"
    }
  },
  "destination": {
    "database": "default",
    "table": "my_table",
    "managedTable": true,
    "tableDefinition": {
      "engine": {
        "type": "MergeTree"
      }
    },
    "columns": [
      {
        "name": "area",
        "type": "Int64"
      },
      {
        "name": "averageSignal",
        "type": "Int64"
      }
    ]
  }
}
</code></pre>

With infrastructure as code, teams can consistently deploy and manage ingestion workflows across environments and version pipelines, improving collaboration and automation. 
The ClickPipes Terraform provider is available in our official Terraform [registry entry](https://registry.terraform.io/providers/ClickHouse/clickhouse/3.2.0-alpha1/docs/resources/clickpipe) (part of the 3.2.0-alpha1 version at the time of writing).

<pre><code type='click-ui'>
resource "clickhouse_clickpipe" "kafka_clickpipe" {
  name           = "My Kafka ClickPipe"
  description    = "Data pipeline from Kafka to ClickHouse"

  service_id     = "e9465b4b-f7e5-4937-8e21-8d508b02843d"

  scaling {
    replicas = 1
  }

  state = "Running"

  source {
    kafka {
      type = "confluent"
      format = "JSONEachRow"
      brokers = "my-kafka-broker:9092"
      topics = "my_topic"

      consumer_group = "clickpipe-test"

      credentials {
        username = "user"
        password = "***"
      }
    }
  }

  destination {
    table    = "my_table"
    managed_table = true

    tableDefinition {
      engine {
        type = "MergeTree"
      }
    }

    columns {
      name = "my_field1"
      type = "String"
    }

    columns {
      name = "my_field2"
      type = "UInt64"
    }
  }

  field_mappings = [
    {
      source_field = "my_field"
      destination_field = "my_field1"
    }
  ]
}
</code></pre>

**NOTE:** The Terraform Provider is currently not available for Postgres and MySQL CDC ClickPipes. 
However, you can manage these ClickPipes via [OpenAPI](https://clickhouse.com/docs/integrations/clickpipes/postgres/faq#can-clickpipe-creation-be-automated-or-done-via-api-or-cli), which is available in Beta. 
Terraform support for CDC ClickPipes is planned for later in 2025.

## AWS PrivateLink Self-Service

We introduced support for AWS PrivateLink within ClickPipes, enabling customers to securely ingest data into ClickHouse Cloud using private network paths. 
With this set of features, users can create private endpoints that facilitate direct communication between their AWS Virtual Private Clouds (VPCs) and ClickHouse Cloud. 
Additionally, ClickPipes now supports reverse private endpoints, which allow ClickHouse Cloud to securely initiate connections to private customer resources.

![3_clickpipes.png](/uploads/3_clickpipes_810c9f5005.png)

![4_clickpipes.png](/uploads/4_clickpipes_7d6644e2b3.png)


The integration supports popular AWS services, including Amazon RDS, Amazon MSK, and other VPC-based resources. Configuration is designed to be straightforward, with documentation available to guide users through the setup process. To learn more and get started, refer to the full guide on [AWS PrivateLink for ClickPipes.](https://clickhouse.com/docs/integrations/clickpipes/aws-privatelink)

## Notifications

A built-in notification system provides real-time alerts about pipeline activity. 
Users can stay updated on ingestion health, failure events, and operational changes.

![5_clickpipes.png](/uploads/5_clickpipes_d4bd32b519.png)

This feature reduces operational blind spots and enables proactive issue resolution. We support receiving notifications through email, ClickHouse Cloud UI, and Slack. More details can be found on the notifications [documentation page](https://clickhouse.com/docs/cloud/notifications).

![6_clickpipes.png](/uploads/6_clickpipes_d207817c7b.png)

## Prometheus Exporter for Observability

ClickPipes now produces Prometheus metrics, part of the [ClickHouse Cloud Prometheus integration](https://clickhouse.com/docs/integrations/prometheus), making ingestion metrics available for dashboards in Grafana, Datadog, and other observability tools.

![7_clickpipes.png](/uploads/7_clickpipes_7acf55e924.png)


Metrics include ingestion volume (bytes or number of rows) and error counts, giving engineering teams the insight needed to maintain performance. The documentation includes integration guides for [Grafana](https://clickhouse.com/docs/integrations/prometheus#integrating-with-grafana) and [Datadog](https://clickhouse.com/docs/integrations/prometheus#integrating-with-datadog).

## System Tables Centralization

Finally, a centralized system table aggregates ClickPipes-related logs, making it possible to monitor Kafka/Kinesis/S3 pipelines from a single SQL interface, the same way users monitor all the ClickHouse services activity, like merges and queries. 

![8_clickpipes.png](/uploads/8_clickpipes_7ef2bbaec6.png)

This design enables users to query pipeline status and health using familiar SQL syntax, build dashboards, and create custom alerts using third-party tooling. The feature is currently in private preview and will soon be generalized to Kafka/Kinesis/S3 ClickPipes users.

## Why This Matters?

These new capabilities help complete the feature landscape of ClickPipes, taking numerous steps towards a complete ingestion platform for ClickHouse Cloud. Together, they provide:

* A streamlined experience with the revamped UI  
* Flexibility with editable pipelines  
* Automation through the ClickPipes API and Terraform  
* Secure, private connectivity via AWS PrivateLink  
* Visibility and alerts through notifications  
* Observability with Prometheus and the upcoming centralized system tables

We look forward to hearing your feedback and continuing to evolve ClickPipes to meet your needs. 
This release is a major step toward a more automated, observable, and developer-friendly ingestion experience.


ClickPipes evolves with a revamped UI/UX, new API and Terraform Provider, AWS PrivateLink integration, notification system, Prometheus metrics for observability, and centralized system tables for enhanced data pipeline management.

The ClickPipes Team

Blog posts by The ClickPipes Team

TimescaleDB to ClickHouse replication: Use cases, features, and how we built it

New: Flexible scaling and enhanced monitoring for streaming ClickPipes

The Evolution of ClickPipes: Revamped UI/UX, ClickPipes API and Terraform Provider, Prometheus metrics, and more!