Replicate data from MongoDB into ClickHouse Cloud in just a few clicks for blazing-fast analytics on document-based data. Now with more connectivity options, improved reliability, and support for DocumentDB.

Summary

The <a href="https://clickhouse.com/cloud/clickpipes">ClickPipes MongoDB CDC connector</a> is now in Public Beta! &#128640; 

After working with dozens of early access customers during Private Preview, we've added support for **sharded clusters** and **Amazon DocumentDB**, improved **reliability** for production workloads, and made it easier to **securely connect** to private MongoDB deployments.

Whether you're running a one-time migration or continuously replicating operational data for analytics, the connector delivers up to **100x faster queries** without the overhead of managing custom pipelines.

> [Sign up for ClickHouse Cloud](https://console.clickhouse.cloud/signup) today to try out the [MongoDB CDC connector for ClickPipes](https://clickhouse.com/cloud/clickpipes)!

<iframe width="768" height="432" src="https://www.youtube.com/embed/ZWM6xsaF23M?si=TlAAhBb7J5aav-pI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

Here’s what our customers are saying:

>Before using ClickPipes, we relied on multiple tools to replicate MongoDB data into ClickHouse, many of which required constant maintenance or manual intervention. After switching to the MongoDB connector in ClickPipes, those issues disappeared: the initial setup was simple, and once configured, it required no ongoing attention. Using materialized views makes data modeling flexible and easy to evolve.
> 
> 
>Overall, ClickPipes has significantly reduced operational overhead and has worked great to replicate several terabytes of business data a day for analytical workloads. - <a href="https://www.rapidata.ai">Rapidata</a>

## What’s new in Public Beta?

### Broader platform compatibility

The MongoDB CDC connector initially launched with support for single [replica sets](https://www.mongodb.com/docs/manual/core/replica-set-members/). We’ve now extended support to [**sharded clusters**](https://www.mongodb.com/docs/manual/core/sharded-cluster-components/), enabling replication from common MongoDB topologies at any scale — whether you're running a single replica set or a distributed deployment with data spread across multiple shards.

We've also added compatibility with **Amazon DocumentDB**, a managed document database service that is API-compatible with MongoDB. See our [new integration guide](https://clickhouse.com/docs/integrations/clickpipes/mongodb/source/documentdb) for step-by-step instructions on how to configure DocumentDB for change stream replication.

### Secure connectivity options

To allow secure connections to MongoDB instances in private networks, which is a table stakes requirement for production environments, you can now configure both **AWS PrivateLink** and **SSH tunneling** when creating a new MongoDB ClickPipe. We've also added support for [X.509 certificate authentication](https://www.mongodb.com/docs/manual/core/security-x.509/), enabling **mutual TLS authentication** between the connector and your MongoDB deployment.

### Production-readiness

We've made significant improvements to reliability and observability based on Private Preview feedback. The connector now handles data type edge cases more gracefully (*e.g.* very large floats, dates outside the standard range) and uses enhanced error classification to automatically retry transient failures that would previously trigger unnecessary alerts.

We've also refined the initial snapshot phase with smarter batching logic that measures uncompressed data size to prevent out-of-memory issues. This addresses the most significant operational issue we identified with large-scale deployments during early access: MongoDB's high JSON compression ratios caused batch sizes to balloon unexpectedly when decompressed during ingestion of very large collections. &#128165;

## Main features

In addition to these Public Beta improvements, the MongoDB CDC connector provides several core capabilities designed to make real-time replication from MongoDB to ClickHouse Cloud reliable and performant:

### Real-time replication

The connector leverages MongoDB's native [Change Streams](https://www.mongodb.com/docs/manual/changestreams/) interface to capture all document changes with low latency, ensuring near real-time synchronization between MongoDB and ClickHouse Cloud. This combination creates a powerful architecture for modern applications that need both operational flexibility and analytical performance, with up to 100x faster analytics — we’re calling it the *Document Data Stack*.

![image1.png](/uploads/image1_cba4cd460a.png)

Keep your MongoDB clusters focused on serving mission-critical operational workloads, including high-throughput CRUD operations and efficient document lookups, while ClickHouse Cloud handles complex analytical queries, reports, and business intelligence workloads without impacting upstream performance.

### Advanced JSON Support

To preserve MongoDB’s rich document structures and provide high-precision replication, the connector uses ClickHouse's powerful [native JSON data type](https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse). This enables **high-performance** analytical queries on semi-structured data, at the same time o**ptimizing compression and reducing storage cost**.

### Fully managed experience

The connector is fully integrated into the ClickHouse Cloud experience. It provides **built-in metrics and monitoring** for visibility into replication health, **detailed logs** for error diagnosis and debugging, **in-place pipe editing**, and more.

## What’s next?

As we work towards General Availability (GA), we're focused on making the connector faster and more flexible for production workloads.

* **Parallel snapshot ingestion:** as is, the connector snapshots collections using a single thread, which can lead to a very long initial snapshotting phase. We’re evaluating (logical) partitioning strategies to improve initial load times for large collections.

* **Flattened mode:** we’ll add an option to automatically map top-level document fields to target columns. This will make it easier to model and query replicated data, while preserving schema evolution capabilities.

* **OpenAPI and Terraform support:** for teams managing infrastructure as code, MongoDB ClickPipes will also be available via Open API and Terraform, similar to other ClickPipe types.

**A note on billing:** usage of MongoDB CDC ClickPipes continues to be **free** until GA. Customers will be notified ahead of the GA launch to review and optimize their ClickPipes usage. You can estimate future costs by referring to [billing for Postgres CDC ClickPipes](https://clickhouse.com/docs/cloud/reference/billing/clickpipes).

## Getting started with the MongoDB CDC connector

The MongoDB CDC connector is available to new and existing ClickHouse Cloud customers, in all service tiers. To get started, navigate to the *Data Sources* tab in the ClickHouse Cloud console, configure the connection details for your MongoDB database, and you’re good to go! For step-by-step instructions, frequently asked questions, and gotchas, check out the [documentation for MongoDB ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/mongodb).

Ready to accelerate analytics on MongoDB data? Try the MongoDB CDC connector today and experience a fully managed, native integration experience with ClickHouse Cloud - the world’s fastest analytics database.

Try the MongoDB CDC connector today

large_MongoDB ClickPipe Banner.jpg

medium_MongoDB ClickPipe Banner.jpg

small_MongoDB ClickPipe Banner.jpg

thumbnail_MongoDB ClickPipe Banner.jpg

MongoDB ClickPipe Banner.jpg

MongoDB CDC connector for ClickPipes is now in Public Beta

Ingest data from Amazon S3 into ClickHouse Cloud in **any** **order** for event-driven, blazing-fast analytics. Files are no longer required to follow lexicographical order.

When you’re on the receiving end of a data integration pipeline, you quickly learn that real-world data knows no rules. Sometimes it arrives, sometimes it doesn’t. Then it arrives twice, or maybe out-of-order. For object storage, in particular, this creates an interesting problem: how do you efficiently track what's new without scanning too many objects all the time?

Like most tools out there, [ClickPipes](https://clickhouse.com/cloud/clickpipes?utm_source=google.com&utm_medium=paid_search&utm_campaign=21862172336_181837693625&utm_content=764403839926&utm_term=clickpipes_g_c&gad_source=1&gad_campaignid=21862172336&gclid=CjwKCAjwisnGBhAXEiwA0zEOR3kM4ZBQO0_NpfRPpYU2YWG5onZqwF-Kf8m845Ol2IJeyN96r0k8DxoCit4QAvD_BwE) didn’t have a good solution for this problem, and simply expected you to ensure there were *some* rules to guarantee files landed in your bucket in lexicographical order. **Not anymore!**

![image6.png](/uploads/image6_b5104fce2a.png)

We’ve added a new mode to the S3 ClickPipe that allows ingesting files in any order using S3 Event Notifications via [Amazon SQS](https://aws.amazon.com/sqs/). Whether you’re dealing with backfills, retries, late-arriving data, or some other source of out-of-orderness — we handle it.

Sign up for ClickHouse Cloud

## Why is this a big deal?

By default, the S3 ClickPipe assumes files are added to a bucket in lexicographical order, and relies on this implicit order to ingest files sequentially. Besides not rolling off the tongue, lexicographical order means that any new file *must* be lexically greater than the last ingested file. For example, files named `events_2024-12-01.parquet`, `events_2024-12-02.parquet`, and `events_2024-12-03.parquet` will be ingested in order, but if a backfill named `events_2024-11-30.parquet` lands later in the bucket, it will be ignored.

This is not great.

With unordered mode, this limitation no longer applies: instead of polling your bucket every 30 seconds looking for the next file to process, the ClickPipe waits to be notified that a new file has landed. When a notification arrives, ClickPipes just goes and processes that file regardless of its relative order to previously processed files. Although this requires you to configure event notifications, it's a more robust and performant alternative to the otherwise common approach of listing all objects in scope on every sync — which scales poorly and gets expensive fast when you have millions of objects.

## How does it work?

To configure an S3 ClickPipe to ingest files that don’t have an implicit order, you need to set up an [Amazon Simple Queue System (SQS)](https://aws.amazon.com/sqs/) queue connected to the bucket. ClickPipes can then listen for `ObjectCreated` events in the queue that match the specified path. 

![ClickPipes S3 Blog Banner.jpg](/uploads/Click_Pipes_S3_Blog_Banner_08474993cb.jpg)

**Unordered mode:** Files land in S3 in **any order** (A) and trigger SQS notifications (B-C). ClickPipes polls and processes files in the specified path, using a metadata store to track state (1-5). Data is inserted into the target tables with exactly-once guarantees (6).

***“What about failures?”*** Since the steps above span multiple systems and don’t happen in one big transaction™️, failures can occur at any step — reading from S3, marking files as processed, inserting data into ClickHouse, and so on. If a failure occurs, ClickPipes automatically reprocesses the batch. *“**But what about duplicates?”*** Even if files are reprocessed multiple times, the S3 ClickPipe guarantees exactly-once semantics, so no duplicates make it into your target table.

Let’s see it working!

### Create an Amazon SQS queue

*The following instructions assume you already have an S3 bucket with some data in it, as well as enough permissions to manage IAM roles and create new resources in your AWS account.*

**1.** In the AWS Console, navigate to **Simple Queue Service > Create queue**. Use the defaults to create a new queue.

**2.** Edit the queue trust policy to allow your bucket (`<bucket-arn>`) in your AWS account (`<aws-account-id>`) to send messages to the SQS queue (`<queue-arn>`).

```
{
 "Version":"2012-10-17", 
 "Id": "example-ID",
 "Statement": [
 {
 "Sid": "AllowS3ToSendMessage",
 "Effect": "Allow",
 "Principal": {
 "Service": "s3.amazonaws.com"
 },
 "Action": [
 "SQS:SendMessage"
 ],
 "Resource": "<sqs-queue-arn>",
 "Condition": {
 "ArnLike": {
 "aws:SourceArn": "<bucket-arn>"
 },
 "StringEquals": {
 "aws:SourceAccount": "<aws-account-id>"
 }
 }
 }
 ]
}
```

**3.** In the ClickHouse Cloud console, navigate to **Settings > Network security information** and copy the IAM role ARN for your service.

![image2.png](/uploads/image2_2accfed654.png)

**4.** Back in the AWS Console, go to **IAM > Roles > Create role**. Choose **Custom trust policy** and paste the IAM role ARN for your ClickHouse Cloud service.

```json
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "AllowAssumeRole",
 "Effect": "Allow",
 "Principal": {
 "AWS": "<ch-cloud-arn>"
 },
 "Action": "sts:AssumeRole"
 }
 ]
}
```

**5.** Create an inline policy for the IAM role with the required permissions to scan and fetch objects from the S3 bucket and manage messages in the SQS queue.

```json
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "S3BucketMetadataAccess",
 "Effect": "Allow",
 "Action": [
 "s3:GetBucketLocation",
 "s3:ListBucket"
 ],
 "Resource": "<bucket-arn>"
 },
 {
 "Sid": "AllowGetListObjects",
 "Effect": "Allow",
 "Action": [
 "s3:Get*",
 "s3:List*"
 ],
 "Resource": "<bucket-arn>/*"
 },
 {
 "Sid": "SQSNotificationsAccess",
 "Effect": "Allow",
 "Action": [
 "sqs:DeleteMessage",
 "sqs:ListQueues",
 "sqs:ReceiveMessage",
 "sqs:GetQueueAttributes"
 ],
 "Resource": "<sqs-queue-arn>"
 }
 ]
}
```

That was a lot of IAM config, but you’re ready to create a ClickPipe to continuously ingest data from your bucket whenever a new file lands!

### Create a ClickPipe

**1.** In the ClickHouse Cloud console, navigate to **Data Sources > Create ClickPipe**, then choose **Amazon S3**. Enter the details to connect to your S3 bucket, using **IAM role** as the authentication method with the ARN role you created in the previous section.

![image5.png](/uploads/image5_19483129ab.png)

**2.** Click **Incoming data**. Toggle on **Continuous ingestion**, where you’ll see the new **Any order** ingestion option.

![image3.png](/uploads/image3_40f8b8de3b.png)

**3.** Click **Parse information**. Define a **Sorting key** for the target table (very important) and make any necessary adjustments to the mapped schema. Finally, configure a role for the ClickPipes database user.

![image1.gif](/uploads/image1_f958d131c1.gif)


4. Sit back and relax. ClickPipes will now perform an initial scan of your bucket, then start processing files as new notification events arrive. &#128640;


If that seems like a lot of clicking, the good news is that ClickPipes is fully supported in the [ClickHouse Terraform provider](https://registry.terraform.io/providers/ClickHouse/clickhouse/latest/docs), so you can configure this setup as code from start to finish.

## What’s next?

We’re excited to enable more complex ingestion patterns from object storage into ClickHouse Cloud with the new unordered mode in S3 ClickPipes — you get the same fully-managed, full-speed experience with a little extra flexibility\! For now, this feature is only available for Amazon S3 as a data source. We plan to extend support to Google Cloud Storage (GCS) and Azure Blob Storage (ABS) in the near future, to ensure feature parity across all Object Storage ClickPipes.

If you have any feedback or run into any snags while setting up ClickPipes, reach out to our team\! For step-by-step instructions, frequently asked questions, and gotchas, check out the [documentation for S3 ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/object-storage/s3/overview#continuous-ingestion-any-order).

Ready to eliminate your ETL complexity and reduce your data movement costs? Try the S3 ClickPipe connector today and experience a fully managed, native integration experience with ClickHouse Cloud - the world’s fastest analytics database.

Get started today

Ingest data from Amazon S3 into ClickHouse Cloud in any order for event-driven, blazing-fast analytics.

large_ClickPipes S3 Blog Banner.png

medium_ClickPipes S3 Blog Banner.png

small_ClickPipes S3 Blog Banner.png

thumbnail_ClickPipes S3 Blog Banner.png

ClickPipes S3 Blog Banner.png

Unordered mode for S3 ClickPipes is now available

![azure-blog-banner.png](/uploads/azure_blog_banner_5fda937064.png)

We're excited to announce that the **Azure Blob Storage connector for [ClickPipes](https://clickhouse.com/cloud/clickpipes?utm_source=google.com&utm_medium=paid_search&utm_campaign=21862172336_181837693625&utm_content=764403839926&utm_term=clickpipes_g_c&gad_source=1&gad_campaignid=21862172336&gclid=CjwKCAjwisnGBhAXEiwA0zEOR3kM4ZBQO0_NpfRPpYU2YWG5onZqwF-Kf8m845Ol2IJeyN96r0k8DxoCit4QAvD_BwE)**  is now Generally Available! You can seamlessly load files from Azure Blob Storage into ClickHouse Cloud with even **better performance**, and leverage the new support for **programmatic access** (via OpenAPI and Terraform) to manage your ClickPipes with confidence.

Over the past few months, the connector has gone through an extensive beta phase, with customers moving anywhere between a few GiB to **as much as 60+ PiB of data** into ClickHouse Cloud — validating its reliability and scalability for both small- and enterprise-scale data lake analytics workloads.

> [Sign up for ClickHouse Cloud](https://console.clickhouse.cloud/signup) today to try out the [Azure Blob Storage connector for ClickPipes](https://clickhouse.com/cloud/clickpipes)!

![azure-blog.gif](/uploads/azure_blog_d52244e13f.gif)

## What's new?

Based on customer feedback, and to align the connector with the product experience for other object storage ClickPipes, the Azure Blob Storage connector has been upgraded with:

### Custom HTTP client for better performance

From the rollout of ClickHouse v25.8 to ClickHouse Cloud, the connector will leverage a new, custom HTTP client that delivers more consistent and better performance than the original Azure SDK native client. This brings the connector to parity with object storage ClickPipes for other cloud providers, when it comes to performance.

### Terraform and OpenAPI support

For mature ClickPipes deployments using workflow automation and Infrastructure as Code (IaC), a new `azureblobstorage`  source type is available in [the ClickPipes OpenAPI specification](https://clickhouse.com/docs/cloud/manage/api/swagger#tag/ClickPipes), as well as the ClickHouse Terraform Provider ([3.6.0-alpha1](https://github.com/ClickHouse/terraform-provider-clickhouse/releases/tag/v3.6.0-alpha1)+). Here's an example of how you'd configure an Azure Blob Storage ClickPipe resource in Terraform:

> **Note**: ClickPipes resources are only supported in [alpha releases](https://registry.terraform.io/providers/ClickHouse/clickhouse/3.7.0-alpha1/docs/resources/clickpipe) of the ClickHouse Terraform provider. Regardless, support for object storage ClickPipes is considered **stable**.

```
resource "clickhouse_clickpipe" "azure_blob" {
  name        = "Azure Blob Storage ClickPipe"
  service_id = var.service_id

  source = {
    object_storage = {
      type    = "azureblobstorage"
      format  = "JSONEachRow"

      path                  = var.azure_path              
      azure_container_name  = var.azure_container_name
      connection_string     = var.azure_connection_string

      authentication = "CONNECTION_STRING"
    }
  }

  destination = {
    table         = "my_azure_data_table"
    managed_table = true

    table_definition = {
      engine = {
        type = "MergeTree"
      }
    }

    columns = [
      {
        name = "id"
        type = "UInt64"
      }, {
        name = "name"
        type = "String"
      }, {
        name = "timestamp"
        type = "DateTime"
      }
    ]
  }

  field_mappings = [
    {
      source_field      = "user_id"
      destination_field = "id"
    }, {
      source_field      = "user_name"
      destination_field = "name"
    }, {
      source_field      = "created_at"
      destination_field = "timestamp"
    }
  ]
}
```

In addition to these improvements, it's worth recapping the base capabilities of the Azure Blob Storage connector:

### Built for reliability

Built on top of the [azureBlobStorage](https://clickhouse.com/docs/sql-reference/table-functions/azureBlobStorage) table function, with additional layers of reliability to handle real-world scale. The connector gracefully handles **automatic retries** on failures and guarantees **exactly-once delivery semantics**, so you don't have to worry about data consistency or partial loads.

### Secure by design

Supports ingestion from **private buckets** using connection strings with account credentials or storage account URLs. You can also use **Shared Access Signatures (SAS)** within the connection string for secure, time-limited access.

### Optimized for performance

The connector dynamically adjusts **ingestion parallelism** and **ClickHouse tuning** based on your cloud instance size and workload. It's built to move terabytes of data quickly and efficiently with no need for manual tuning.

### Fully managed experience

The connector is fully integrated into the ClickHouse Cloud experience. It provides **built-in metrics and monitoring, detailed logs** for error diagnosis and debugging, **in-place pipe editing** (e.g., adding columns), and more.

## Pricing

Object storage ClickPipes have a predictable pricing model based on data volume, with minimal added compute cost. The Azure Blob Storage connector for ClickPipes is **10-25x more cost-effective** than using third-party ETL tools, reflecting our commitment to offering ClickPipes as a seamless but affordable option to connect ClickHouse Cloud and your various data sources. For more details, see the [ClickPipes billing documentation](https://clickhouse.com/docs/cloud/reference/billing/clickpipes/streaming-and-object-storage).

## Getting started with the Azure Blob Storage connector

The Azure Blob Storage connector is available to new and existing ClickHouse Cloud customers, in all service tiers. To get started, navigate to the Data Sources tab in the ClickHouse Cloud console, configure the connection details for your storage container, and you're good to go! For step-by-step instructions, frequently asked questions, and gotchas, check out the [documentation for object storage ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/object-storage).

> Ready to eliminate your ETL complexity and reduce your data movement costs? [Try the Azure Blob Storage connector today](https://clickhouse.com/cloud) and experience a fully managed, native integration experience with ClickHouse Cloud — the world's fastest analytics database.



We’re excited to announce the general availability of the Azure Blob Storage connector for ClickPipes. Natively integrated with ClickHouse Cloud, this connector enables blazing-fast load from Azure Blob Storage with just a few clicks.

Marta Paes

Blog posts by Marta Paes

MongoDB CDC connector for ClickPipes is now in Public Beta

Unordered mode for S3 ClickPipes is now available

Azure Blob Storage connector for ClickPipes is now Generally Available