Introducing WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

Today, we’re announcing WalShadow, an open-source engine that replicates Postgres data to ClickHouse directly from physical WAL.

In our benchmarks, transactions committed in Postgres became visible in ClickHouse in around 200 ms, while WalShadow sustained 289K rows/sec, effectively keeping pace with the source Postgres instance.

Unlike traditional CDC based systems, WalShadow doesn’t use Postgres logical replication. It consumes the same physical WAL stream used by Postgres replicas, decodes it outside the source database, and writes ClickHouse-native blocks directly into ClickHouse. The result is a replication architecture that gets close to the latency and throughput of a Postgres physical standby, while making the data immediately available for analytics in ClickHouse.

WalShadow supports the complete replication lifecycle, including initial load, continuous replication, schema evolution, restart recovery, and planned source switchovers.

By consuming physical WAL directly, WalShadow eliminates the need for logical replication slots, removes much of the operational overhead associated with logical replication, and significantly reduces resource consumption on the source Postgres instance. It also supports complex schema changes such as ADD COLUMN, RENAME COLUMN, DROP COLUMN, and CREATE TABLE.

WalShadow is fully open source and available today on GitHub.

Bringing WalShadow to ClickHouse Managed Postgres #

For a fully managed experience, we’re also launching WalShadow in private preview for ClickHouse Managed Postgres.

Physical WAL is key to WalShadow’s architecture, but most managed Postgres services don’t expose it to customers, making it impossible to use WalShadow. ClickHouse Managed Postgres manages both sides of the stack, allowing us to integrate WalShadow directly into the Postgres replication layer and provide a native path from Postgres WAL to ClickHouse.

Sign up for the private preview of WalShadow on ClickHouse Managed Postgres.

Architecture: From physical WAL to ClickHouse-native blocks #

WalShadow takes a new approach to Postgres-to-ClickHouse replication, effectively turning ClickHouse into an analytical physical standby.

WalShadow consumes the same WAL stream Postgres generates for physical replication and recovery. Instead of asking the source database to decode changes into logical events, WalShadow processes the WAL outside the source through four stages:

1. Track the live schema. WalShadow filters catalog WAL records and replays them into a schema-only shadow Postgres instance. This maintains an up-to-date catalog of tables, columns, and Postgres types as the source schema evolves.
2. Decode data changes in parallel. Heap records are distributed across a pool of Rust-based decoders without passing through the shadow Postgres instance.
3. Build ClickHouse-native blocks. A batcher groups decoded rows by table into complete ClickHouse-native blocks, preserving data fidelity without intermediate format conversions.
4. Insert in parallel. A separate pool of inserters writes multiple blocks to ClickHouse concurrently, allowing decoding and insertion to scale independently.

This creates a direct path from Postgres to ClickHouse: no logical decoding output plugin, no Kafka, and no JSON serialization or separate normalization step.

Because WalShadow processes blocks in parallel, they may arrive in ClickHouse out of order. WalShadow preserves correctness by attaching the source WAL position (_lsn) to every row, allowing ClickHouse to retain the latest version for each key. Operations that require strict ordering, such as schema changes and truncations, introduce barriers that wait for all preceding data to become durable before they are applied.

The source only needs to ship physical WAL, resulting in a load profile similar to a physical standby while allowing changes to reach ClickHouse within a second.

For a detailed overview of architecture, see the architecture documentation. To understand various available tuning settings affecting performance and functionality, see the configuration guide.

Performance benchmarks #

We benchmarked WalShadow against PeerDB, our state-of-the-art Postgres-to-ClickHouse CDC tool powered by logical replication which powers ClickPipes.

For me, this comparison is bittersweet. We’re proud that PeerDB, which we created, serves thousands of customers. WalShadow carries that journey forward by reimagining replication directly from physical WAL and moving us closer to a unified Postgres and ClickHouse stack.

The benchmark replicated a continuous stream of changes from a single table, with Postgres, ClickHouse, and each replication tool running in the same region. We used modest c8i.2xlarge instances with 8 vCPUs each. Performance will vary by workload, but these results offer a useful indication of what WalShadow can offer.

Around 200 ms commit-to-visible latency #

Commit-to-visible latency measures the time from a transaction committing in Postgres to its rows becoming visible in ClickHouse. A native Postgres physical replica established a practical baseline of around 50 ms. WalShadow achieved approximately 200 ms, compared with around 10 seconds for PeerDB, about 50x lower latency in this benchmark.

289,000 rows per second sustained throughput #

The source Postgres instance sustained approximately 290,000 inserted rows per second, establishing the maximum rate the replication pipeline could process. WalShadow replicated 289,000 rows per second, effectively matching the source without becoming the bottleneck. PeerDB sustained approximately 120,000 rows per second, or around 40% of the source rate.

These results show that low latency does not have to come at the cost of throughput: WalShadow keeps data real-time while operating at nearly the full speed of the source.

Conclusion and vision #

At ClickHouse, we have taken several major steps to bring Postgres and ClickHouse closer together: acquisition of PeerDB, launching Postgres CDC in ClickPipes, and introducing ClickHouse Managed Postgres, an enterprise-grade Postgres service built on local NVMe storage for fast OLTP and natively integrated with ClickHouse for fast OLAP.

These efforts share one goal: to give developers a unified data stack that combines Postgres for transactions and ClickHouse for analytics, without added complexity.

WalShadow represents a major milestone toward that vision. By replicating directly from physical WAL into ClickHouse-native blocks, it delivers sub-second analytics, removes much of the operational overhead of logical replication, and supports advanced schema changes that are difficult to handle through conventional logical-decoding pipelines.

Over the coming months, we will work closely with customers to harden WalShadow across real-world workloads. We are already collaborating with an initial group of design partners and are now ready to expand access.

Sign up for the private preview, and we’ll get you access within a day or two.

WalShadow is one of several initiatives underway to make Postgres and ClickHouse work seamlessly together. Stay tuned for more.