Stream data from Pub/Sub in a few clicks for blazing-fast analytics over GCP infrastructure logs, application events, and operational signals. No routing through GCS or Dataflow required.

Summary

Pub/Sub is the primary messaging layer for a wide range of GCP services: Cloud Logging, Cloud Monitoring, GKE workloads, and custom event producers across the GCP stack. As more teams route their GCP infrastructure logs, application events, and operational signals to ClickHouse Cloud, we wanted to lower the barrier to get this data in.

Today, we're announcing a new **GCP Pub/Sub connector for [ClickPipes](https://clickhouse.com/cloud/clickpipes)**, now available in Private Preview! You can seamlessly stream data from GCP Pub/Sub into ClickHouse Cloud in **real time** with a few clicks and **no additional infrastructure**. The connector supports all **common formats** (JSON, Avro, Protobuf) and **schema registry integration**, with attribute-based message filtering, flexible seek options, and per-key ordered delivery. Like all ClickPipes connectors, it can also be managed programmatically via OpenAPI and the ClickHouse Terraform provider.


Join the waitlist today to try out the GCP Pub/Sub connector for ClickPipes!


GCP Pub/Sub connector for ClickPipes

pubsub_compressed.mp4

## Why build a ClickPipe?

**Simpler than building your own pipeline.** Previously, getting data from Pub/Sub into ClickHouse meant routing through GCS or standing up a Dataflow pipeline. The Pub/Sub ClickPipe handles all of this automatically: point it at your topic, configure your target table, and the connector manages subscription lifecycle, schema inference, and message delivery for you.

**Built for continuous ingestion.** ClickPipes consumes messages in real-time, optimizing ingestion performance out-of-the-box and eliminating trade-offs between freshness and cost. The connector scales horizontally and vertically for high-throughput topics, is optimized for ClickHouse Cloud's architecture, and handles failures gracefully with configurable replicas and automatic retries.

**Fully managed and native to ClickHouse Cloud.** Built-in metrics, monitoring, and detailed logs give you visibility into your pipeline without standing up external tooling. You can edit ClickPipes in place (*e.g.* add columns, adjust ingestion settings) without recreating them from scratch.

## Main features

### Integration with schema registry

All common serialization formats are supported: **JSON**, **Avro**, and **Protobuf**; including native integration with the **Pub/Sub Schema Registry**. Schemas and field types are inferred and mapped automatically to a target table with ClickHouse native data types. Compressed payloads are **detected and decompressed automatically**, with no additional configuration needed.

### Configurable start offsets

The starting offset supports Pub/Sub's native [seek feature](https://docs.cloud.google.com/pubsub/docs/replay-overview), giving you precise **control over where ingestion begins**: latest, earliest, or a specific timestamp for backfills and point-in-time replay for failure scenarios.

### Message filtering

During ClickPipe creation, you can configure a Pub/Sub [subscription filter](https://docs.cloud.google.com/pubsub/docs/subscription-message-filter) on message attributes to drop messages before they are pulled from the subscription, reducing ingestion volume and associated costs.

### Terraform & Open API support

For mature ClickPipes deployments using workflow automation and Infrastructure as Code (IaC), a new `pubsub` source type is available in [the ClickPipes OpenAPI specification](https://clickhouse.com/docs/cloud/manage/api/swagger#tag/ClickPipes), as well as the ClickHouse Terraform Provider ([3.16.0+](https://github.com/ClickHouse/terraform-provider-clickhouse/releases/tag/v3.16.0)). Here's an example of how you'd configure a GCP Pub/Sub ClickPipe resource in Terraform:

```
resource "clickhouse_clickpipe" "pubsub" {
  service_id = var.service_id
  name       = var.pipe_name

  source = {
    pubsub = {
      format         = "JSONEachRow"
      project_id     = var.gcp_project_id
      topic          = var.pubsub_topic
      authentication = "SERVICE_ACCOUNT"
      seek_type      = "latest"

      service_account_key = {
        service_account_file = var.gcp_service_account_b64
      }
    }
  }

  destination = {
    table         = var.table
    managed_table = true

    table_definition = {
      engine = {
        type = "MergeTree"
      }
    }

    columns = [
      {
        name = "my_field1"
        type = "String"
      },
      {
        name = "my_field2"
        type = "UInt64"
      },
    ]
  }

  field_mappings = [
    {
      source_field      = "my_field"
      destination_field = "my_field1"
    }
  ]
}

output "clickpipe_id" {
  value = clickhouse_clickpipe.pubsub.id
}
```

## How to sign up for Private Preview?

The GCP Pub/Sub connector is available upon request, and **free** to use during Private Preview. Fill out [this form](https://clickhouse.com/cloud/clickpipes#pubsub-private-preview) to join the waitlist, or reach out to your ClickHouse Cloud account manager. To get started with step-by-step instructions, check out the [documentation for GCP Pub/Sub ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/pubsub).


Ready to test ClickHouse with your Pub/Sub data?

In our benchmarks, ClickHouse delivers up to 4x faster query performance and significant cost savings for interactive analytics. Join the waitlist today to try it out for yourself with the new GCP Pub/Sub connector for ClickPipes.


Ready to test ClickHouse with your Pub/Sub data?

large_Sub connector for ClickPipes is now in Private Preview.jpg

medium_Sub connector for ClickPipes is now in Private Preview.jpg

small_Sub connector for ClickPipes is now in Private Preview.jpg

thumbnail_Sub connector for ClickPipes is now in Private Preview.jpg

Sub connector for ClickPipes is now in Private Preview.jpg

GCP Pub/Sub connector for ClickPipes is now in Private Preview

Sign up for ClickHouse Cloud today to try out seamless data ingestion with ClickPipes!

Get started today

Today, we’re announcing the general availability of ClickPipes resources in Terraform and endpoints in OpenAPI! Although we [introduced beta support for these interfaces last year](https://clickhouse.com/blog/evolution-of-clickpipes), there were some feature and usability gaps that prevented *all* users from managing their ClickPipes as code. Most notably, we were missing full connector coverage: well, now you can also create CDC ClickPipes (Postgres, MySQL, MongoDB) as code, too!

## **What is supported?** {#what_is_supported}

In addition to support for CDC ClickPipes, we’ve worked on adding new ClickPipes connectors and features to both interfaces on launch (*e.g.* BigQuery and Azure Blob Storage connectors, S3 and GCS unordered mode), as well as backfilling important gaps like support for creating and managing reverse private endpoints. This means that you’re unlikely to find something in the ClickPipes UI that isn’t exposed via OpenAPI or Terraform. Unless that something is SSH tunneling, but we’re [working on it](#bookmark=id.uz6whn8nwd37)!

Every operation now cleanly maps to a resource in the [ClickHouse Terraform provider](https://registry.terraform.io/providers/ClickHouse/clickhouse/latest/docs) or an [OpenAPI endpoint](https://clickhouse.com/docs/cloud/manage/api/swagger):

| Action | Terraform | OpenAPI |
| :---- | :---- | :---- |
| List | `terraform state list` | `GET /clickpipes` |
| Create | Add resource + `terraform apply` | `POST /clickpipes` |
| Read | `terraform show / refresh` | `GET /clickpipes/{id}` |
| Update | Change config + `terraform apply` | `PATCH /clickpipes/{id}` |
| Stop | `stopped = true` + `terraform apply` | `PATCH /clickpipes/{id}/state {"action": "stop"}` |
| Resume | `stopped = false` + terraform apply | `PATCH /clickpipes/{id}/state {"action": "start"}` |
| Update settings | Update `settings` block + `terraform apply` | `PUT /clickpipes/{id}/settings` |
| Scale | Update `scaling` block + `terraform apply` | `PATCH /clickpipes/{id}/scaling` |
| Trigger resync | `trigger_resync = true` + `terraform apply` | `PATCH /clickpipes/{id}` |
| Delete | `terraform destroy` | `DELETE /clickpipes/{id}` |

If you're provisioning a ClickPipe from scratch, start with [*How to: configure a new ClickPipe*](#bookmark=id.qa1u83ivk9g1). If you already have pipes running and want to bring them under version control, check [*How to: import existing ClickPipes*](#bookmark=id.1fs3jt8zn01g).

## How to: configure a new ClickPipe

#### Create a Cloud API key

**1\.** The ClickHouse Terraform provider uses the [Cloud API](https://clickhouse.com/docs/cloud/manage/cloud-api) to interact with your ClickHouse Cloud service, so you’ll need a valid API key for authentication. If you’re new to the provider, the first step is to create a new API key. Navigate to **Organization > API keys > New API key**, and note (or download) the provided **Key ID** and a **Key Secret** pair.

![](/uploads/terraform_clickpipes_apr2026_image2_c51c42d023.png)

**2\.** Next, configure the provider. To keep credentials out of source control, we recommend using [Terraform variables](https://developer.hashicorp.com/terraform/language/values/variables) and a `terraform.tfvars` file that you can add to `.gitignore`:

**variables.tf**

<pre><code type='click-ui' language='bash'>
variable "organization_id" {
  description = "ClickHouse Cloud organization ID"
  type        = string
}

variable "service_id" {
  description = "ClickHouse Cloud service ID"
  type        = string
}

variable "token_key" {
  description = "ClickHouse Cloud API key ID"
  type        = string
  sensitive   = true
}

variable "token_secret" {
  description = "ClickHouse Cloud API key secret"
  type        = string
  sensitive   = true
}
...
</code></pre>

**terraform.tfvars**

<pre><code type='click-ui' language='bash'>
organization_id   = "" # your org ID from cloud.clickhouse.com
service_id        = "" # target ClickHouse service ID
token_key         = "" # API key ID
token_secret      = "" # API key secret
...
</code></pre>

#### Configure a ClickPipes resource

In this example, we’ll configure a resource for a [Postgres CDC ClickPipe](https://clickhouse.com/docs/integrations/clickpipes/postgres) to continuously ingest changes from a Postgres database into ClickHouse in near real time. Remember: before creating a CDC ClickPipe, make sure to follow the [configuration guides](https://clickhouse.com/docs/integrations/clickpipes/postgres#prerequisites) for your data source, which guide you through enabling replication upstream.

**1\.** Once you’re ready to create a ClickPipe, add a new [`clickhouse_clickpipe` resource](https://registry.terraform.io/providers/ClickHouse/clickhouse/latest/docs/resources/clickpipe) to your Terraform configuration using `postgres` as the source attribute. Below is a basic configuration example that creates a single ClickPipe to sync a single Postgres table (`public.firenibble`) and propagate any new changes in the `source_table` (*i.e.* inserts, updates, deletes) to the `target_table` in ClickHouse Cloud using `cdc`.

**main.tf**

<pre><code type='click-ui' language='bash'>
terraform {
 required_providers {
   clickhouse = {
     source  = "ClickHouse/clickhouse"
     version = ">= 3.14.0"
   }
 }
}

provider "clickhouse" {
 organization_id = var.organization_id
 token_key       = var.token_key
 token_secret    = var.token_secret
}

resource "clickhouse_clickpipe" "pg_pipe" {
 name       = "tf-postgres-clickpipe"
 service_id = var.service_id

 source = {
   postgres = {
     host     = var.postgres_host
     port     = 5432
     database = var.postgres_database

     credentials = {
       username = var.postgres_user
       password = var.postgres_password
     }

     settings = {
       replication_mode = "cdc"
     }

     table_mappings = [
       {
         source_schema_name = "public"
         source_table       = "firenibble"
         target_table       = "public_firenibble"
       }
     ]
   }
 }

 destination = {
   database = "default"
 }
}
</code></pre>

**2\.** Now that you’ve passed your Cloud API credentials and resource configuration to Terraform, you’re ready to deploy. Run `terraform init` to install the ClickHouse provider, then `terraform apply` to provision the pipe. Terraform will output a plan and prompt you to confirm the planned changes before deploying:

<pre><code type='click-ui' language='bash'>
terraform apply

...
Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

clickhouse_clickpipe.pg_pipe: Creating...
clickhouse_clickpipe.pg_pipe: Still creating... [00m10s elapsed]
clickhouse_clickpipe.pg_pipe: Creation complete after 11s [id=10128f88-100c-4830-b480-e242fa89570f]

</code></pre>

All done! Your first CDC ClickPipe created programmatically is now up and running. Head over to the ClickHouse Cloud console and double-check it is indeed there (under **Data sources > ClickPipes**), churning through data ingestion.

![](/uploads/terraform_clickpipes_apr2026_image1_1ac36961ee.png)

You can find examples on how to configure ClickPipes in Terraform for different pipe types, ingestion modes and network setups in the [provider repo](#bookmark=id.1fs3jt8zn01g).

## **How to: import existing ClickPipes**

*The following steps were tested with HashiCorp Terraform v1.5+. The import workflow should be broadly compatible with alternative implementations (e.g. OpenTofu), but minimum versions and base methods might differ.*

If you already have a project that uses the ClickHouse Terraform provider, you can import existing ClickPipes into your Terraform state and start managing them as code, instead of manually configuring them.

**1\.** Use the Cloud API to retrieve existing ClickPipes for the organization and service in scope:

```shell
curl -s \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes" \
  -u "$KEY_ID:$KEY_SECRET" | jq -r '.result[] | "\(.id)\t\(.name)"'

1667855d-1646-4693-8cac-60e1c386ccb1    existing_pg_pipe
2e4c3974-025e-47a0-861b-4c200b6c0249    existing_mysql_pipe
76a42195-ef25-40fc-ae3d-31733d377f77    existing_mongo_pipe
```

**2\.** Edit your configuration file to include import blocks for any ClickPipes you want to import into Terraform:

<pre><code type='click-ui' language='bash'>
...
import {
  to = clickhouse_clickpipe.existing_pg_pipe
  id = "&lt;service_id&gt;:1667855d-1646-4693-8cac-60e1c386ccb1"
}
...
</code></pre>

**3\.** Generate the Terraform resource configuration from the existing resources:

```shell
terraform plan -generate-config-out=generated.tf
```

Review `generated.tf` and fill in any sensitive fields (like credentials) that Terraform can't retrieve from state.

<pre><code type='click-ui' language='bash'>
# __generated__ by Terraform
# Please review these resources and move them into your main configuration files.

# __generated__ by Terraform
resource "clickhouse_clickpipe" "pg_pipe" {
…
credentials    = {
username = var.postgres_user
password = var.postgres_password
}
…}
</code></pre>

**4\.** Run `terraform plan` to verify there are no unexpected diffs, then `terraform apply` to finalize.

*Note: For Postgres CDC ClickPipes, you’ll need to set `stopped = true` in the resource config before running `terraform apply`. Terraform will pause the pipe and write credentials to state in one pass. Once complete, set `stopped = false` and apply again to resume. This is a [known limitation](https://github.com/ClickHouse/terraform-provider-clickhouse/issues/497).*

## What’s next?

ClickPipes resources are available in stable releases of the ClickHouse Terraform provider from [v3.14.0](https://github.com/ClickHouse/terraform-provider-clickhouse/releases/tag/v3.14.0), and OpenAPI endpoints have also graduated out of beta ([spec](https://clickhouse.com/docs/cloud/manage/api/swagger)). As ClickPipes evolves, we’ll continue treating Terraform and OpenAPI as first-class interfaces that are part of our “definition of done”. We’re actively working on some complex usability limitations like the one mentioned above, as well as the much-requested support for **SSH tunneling** configuration!

If you have feedback or run into any snags while configuring and managing ClickPipes using Terraform and OpenAPI, reach out to our team. Soon, we’ll also make ClickPipes available in the [recently released ClickHouse CLI](https://clickhouse.com/blog/introducing-clickhousectl-official-cli-for-clickhouse-local-and-cloud); more on this in a few weeks!

Ready to eliminate your ETL complexity and reduce your data movement costs? Try ClickPipes today and experience a fully managed, native integration experience with ClickHouse Cloud — the world’s fastest analytics database.

Try ClickPipes today

Provision and manage ClickPipes resources “as code” using Terraform and OpenAPI, now with full connector coverage and improved usability

large_Terraform Open API ClickPipes GA.jpg

medium_Terraform Open API ClickPipes GA.jpg

small_Terraform Open API ClickPipes GA.jpg

thumbnail_Terraform Open API ClickPipes GA.jpg

Terraform Open API ClickPipes GA.jpg

Terraform & OpenAPI for ClickPipes is now Generally Available

> *Ingest data from Google Cloud Storage into ClickHouse Cloud in **any** **order** for event-driven, blazing-fast analytics. Files are no longer required to follow lexicographical order.*

A few months ago, we removed one of the biggest usability wrinkles in S3 ClickPipes by [supporting ingesting files in any order](https://clickhouse.com/blog/clickpipes-s3-unordered-mode) (*aka* unordered mode). We’re now extending that functionality to the Google Cloud Storage (GCS) connector, with a little help from [Google Cloud Pub/Sub notifications for Cloud Storage](https://docs.cloud.google.com/storage/docs/pubsub-notifications).

![gcs-unordered-mode.png](https://cms.clickhouse-dev.com:1337/uploads/gcs_unordered_mode_fd4559c5df.png)

This means that you no longer need to worry about ensuring files land in your bucket in lexicographical order: with unordered mode, we’ll simply listen to notifications for new files and ingest files as they land in the GCS bucket. Whether you’re dealing with backfills, retries, late-arriving data, or some other source of out-of-orderness — this is now covered.

Sign up for ClickHouse Cloud today to try out the GCS connector for ClickPipes!

## Why is this a big deal? {#why-is-this-a-big-deal}

By default, the GCS ClickPipe assumes files are added to a bucket in lexicographical order, and relies on this implicit order to ingest files sequentially. This means that any new file *must* be lexically greater than the last ingested file, which isn’t always true in the real world. For example, files named `events_2024-12-01.parquet`, `events_2024-12-02.parquet`, and `events_2024-12-03.parquet` will be ingested in order, but if a backfill named `events_2024-11-30.parquet` lands later in the bucket, it will be ignored.

Not cool.

With unordered mode, this limitation no longer applies: instead of polling the bucket every 30 seconds looking for the next file to process, the ClickPipe waits for new file notifications. When a notification arrives, ClickPipes just goes and processes that file regardless of its relative order to previously processed files. Although this requires a little extra configuration, it's a more robust and scalable approach when you’re dealing with millions of objects.

## How does it work? {#how-does-it-work}

To configure a GCS ClickPipe to ingest files that don’t have an implicit order, you need to configure notifications from the bucket to a Pub/Sub topic. ClickPipes can then listen for `OBJECT_FINALIZE` events and ingest any new files regardless of the file naming convention.  

![564927906-277a62f5-799d-4d30-951d-35ebc8b346f2.jpg](https://cms.clickhouse-dev.com:1337/uploads/564927906_277a62f5_799d_4d30_951d_35ebc8b346f2_56abffa4ed.jpg)

_**Unordered mode:** Files land in GCS in **any order** (A) and trigger Pub/Sub notifications (B-C). ClickPipes polls and processes files in the specified path, using a metadata store to track state (1-5). Data is inserted into the target tables with exactly-once guarantees (6)._

***“What about failures?”*** Since the steps above span multiple systems and don’t happen in a single transaction, failures can occur at any step — reading from GCS, marking files as processed, inserting data into ClickHouse, and so on. If a failure occurs, ClickPipes automatically reprocesses the batch. ***"But what about duplicates?”*** Even if files are reprocessed multiple times, the GCS ClickPipe guarantees exactly-once semantics, so no duplicates make it into your target table.

Let’s see it in action!

### Create a Pub/Sub topic {#create-a-pubsub-topic}

*The following instructions assume you already have a GCS bucket with some data in it, as well as enough permissions to manage IAM roles and create new resources in your Google Cloud account.*

**1\.** In the Google Cloud Console, navigate to **Pub/Sub > Topics > Create topic**. Create a new topic with a default subscription and note the **Topic Name**.

**2\.** Configure a [service account](http://docs.cloud.google.com/iam/docs/keys-create-delete) with the minimum required set of permissions to allow ClickPipes to list and fetch objects in the specified bucket, as well as consume and monitor notifications from the Pub/Sub subscription.

![](/uploads/gcs_unordered_mar2026_image1_5969bf8711.png)

**3\.** Configure your GCS bucket to send notifications to Pub/Sub when a new object lands in the bucket. This step cannot be performed in the Google Cloud Console, so you must use `gcloud` or your preferred programmatic interface.

**3.1.** Using `gcloud`, add a notification configuration to your GCS bucket that triggers notifications for the `OBJECT_FINALIZE` [event type](https://docs.cloud.google.com/storage/docs/pubsub-notifications#events):

<pre><code language='bash'>
# Create a Pub/Sub notification for new objects in the bucket
gcloud storage buckets notifications create "gs://${YOUR_BUCKET_NAME}" \
--topic="projects/${YOUR_PROJECT_ID}/topics/${YOUR_TOPIC_NAME}" \
--event-types="OBJECT_FINALIZE" \
--payload-format="json"

# List the Pub/Sub notifications in the bucket
gcloud storage buckets notifications describe
</code></pre>

We strongly recommend configuring a [**Dead-Letter topic**](https://docs.cloud.google.com/pubsub/docs/dead-letter-topics), too, so it's easier to debug and retry failed notifications. But that’s it — you’re ready to create a ClickPipe to continuously ingest data from your bucket whenever a new file lands!

### Create a ClickPipe {#create-a-clickpipe}

**1\.** In the ClickHouse Cloud console, navigate to **Data Sources > Create ClickPipe**, then choose **Google Cloud Storage**. Enter the details to connect to your GCS bucket, using **Service account** as the authentication method and providing the `.json` service account key.

**2\.** Toggle on **Continuous ingestion**, then select the new **Any order** ingestion mode to enable unordered mode. Enter the path to your Pub/Sub subscription.

![](/uploads/gcs_unordered_mar2026_image5_01fc577707.png)

**3\.** Click **Incoming data**. Define a **Sorting key** for the target table (very important) and make any necessary adjustments to the mapped schema. Finally, configure a role for the ClickPipes database user.

![cp-gcs-unordered.gif](https://cms.clickhouse-dev.com:1337/uploads/cp_gcs_unordered_26a91e8d0f.gif)

**4\.** **Sit back and relax.** ClickPipes will now perform an initial scan of your bucket, then start processing files as new notification events arrive. 🚀

If that seems like a lot of clicking, the good news is that ClickPipes is fully supported in the [ClickHouse Terraform provider](https://registry.terraform.io/providers/ClickHouse/clickhouse/latest/docs), so you can configure this setup as code from start to finish.

## What’s next? {#whats-next}

We’re excited to enable more complex ingestion patterns from object storage into ClickHouse Cloud with the new unordered mode in GCS ClickPipes — you get the same fully-managed, full-speed experience with a little extra flexibility! This feature is also available for Amazon S3 as a data source. We plan to extend support to Azure Blob Storage in the near future, to ensure feature parity across **all** Object Storage ClickPipes.

If you have any feedback or run into any snags while setting up ClickPipes, reach out to our team! For step-by-step instructions, frequently asked questions, and gotchas, check out the [documentation for GCS ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/object-storage/gcs/get-started).

*Ready to eliminate your ETL complexity and reduce your data movement costs? [Try the GCS ClickPipe connector today](https://clickhouse.com/cloud/clickpipes) and experience a fully managed, native integration experience with ClickHouse Cloud — the world’s fastest analytics database.*


Try the GCS ClickPipe connector today  and experience a fully managed, native integration experience with ClickHouse Cloud — the world’s fastest analytics database.

Ready to eliminate your ETL complexity and reduce your data movement costs?

You can now ingest data from Google Cloud Storage into ClickHouse Cloud in any order for event-driven, blazing-fast analytics.

large_ClickPipes GCS Blog Banner.jpg

medium_ClickPipes GCS Blog Banner.jpg

small_ClickPipes GCS Blog Banner.jpg

thumbnail_ClickPipes GCS Blog Banner.jpg

ClickPipes GCS Blog Banner.jpg

Unordered mode for GCS ClickPipes is now available

Load data from BigQuery into ClickHouse Cloud in a few clicks for fast exploration and prototyping. The connector simplifies data migration for testing ClickHouse's real-time query performance on your BigQuery datasets.


Running ClickHouse Cloud as a _speed-layer_ on top of BigQuery is an increasingly common pattern: you let BigQuery handle batch processing and data warehousing, while ClickHouse powers **interactive queries that need sub-second latency**. As we onboard more customers with this architecture, it became clear that moving data over is a significant first hurdle in the onboarding experience. Teams would spend days iterating on export scripts, writing custom type conversion logic, and trial-and-error — before they could even start prototyping and testing their queries.

> [*Join the waitlist*](https://clickhouse.com/cloud/clickpipes/bigquery-connector) *today to try out the BigQuery connector for ClickPipes!*

Today, we’re announcing the first step towards making this integration a no-brainer: **a new BigQuery connector for ClickPipes**, in Private Preview! Whether you're evaluating ClickHouse Cloud for specific use cases where sub-second query latency matters, or exploring how BigQuery and ClickHouse can work together in your data architecture, the connector provides a straightforward path to get your data into ClickHouse Cloud for rapid prototyping and performance testing.

<iframe width="768" height="432" src="https://www.youtube.com/embed/9skyBoIQbTM?si=y3LwPrrj7s5Vsi6N" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## Why build a ClickPipe?

**Simpler than building your own pipeline.** Previously, loading data from BigQuery into ClickHouse meant manually exporting tables to GCS, managing service account permissions across both platforms, writing custom SQL, and handling schema mapping yourself. The BigQuery ClickPipe handles all of this automatically: point it at your BigQuery dataset(s), configure your target tables, and the connector manages all those steps for you.

**Built for continuous ingestion.** The connector is built on [PeerDB](http://github.com/PeerDB-io/peerdb), an ✨open-source✨ data replication platform maintained by ClickHouse. Although BigQuery doesn't have a replication log for log-based CDC, PeerDB has the primitives to support query-based CDC for incremental syncs. We'll use this to add continuous ingestion capabilities in future releases as we evaluate the right cost-performance tradeoffs for BigQuery’s architecture.

**Data movement is just the start.** Loading your BigQuery tables into ClickHouse gets you far: you can prototype queries, validate performance, and build proof-of-concepts for the interactive workloads you want to migrate. But a successful, end-to-end workload migration also means translating SQL queries, optimizing schemas for ClickHouse's architecture, and validating results across (potentially thousands of) queries. As our teams make progress on AI-assisted migrations (see [clickhouse.build](https://clickhouse.com/blog/clickhouse-build-agentic-cli-accelerate-postgres-clickhouse-apps) for a recent example), we're exploring how to package this up for BigQuery and other connectors, too!

## Main features

#### Efficient bulk loads

Private Preview supports **bulk data loads** from BigQuery tables. This is what most teams need as that first step: load a representative dataset, test some queries, and validate that ClickHouse performs the way you expect. The connector exports each table as compressed Avro files (auto-sharded at 1GB per file) to an intermediate GCS bucket, then loads them into ClickHouse Cloud in parallel. For very large datasets, it’s possible to adjust ingestion settings and sizing to speed up the initial load.

#### Table- and column-level filtering

The connector automatically discovers all datasets and tables within your BigQuery project (that the associated service account is scoped to). From the ClickPipes UI, you have fine-grained control over what to load, whether that's a single table, an entire dataset, or even tables across different datasets.

![image2.png](/uploads/image2_4fd922a530.png)

For each table, you can exclude specific columns, customize column mappings, and configure granular settings like how to handle `NULL` values. This eliminates the manual work of exporting tables one-by-one and gives you control over what data lands in ClickHouse without writing any code.

#### Implicit type mapping

The connector handles BigQuery's type system automatically, including complex conversions that don't map directly to ClickHouse (*e.g.* `DATE` values stored as days-since-epoch become proper ClickHouse dates, `BYTES` columns get base64 decoded). With type mapping out of the way, you avoid writing error-prone conversion logic and investing time in debugging edge cases when data doesn't load correctly or queries return unexpected results.

#### Terraform and Open API support

Although you can only sync full tables in Private Preview, for smaller tables it can be useful to do full periodic reloads. To support workflow automation and managing the ClickPipe as code, a new `bigquery` source type is available in [the ClickPipes OpenAPI specification](https://clickhouse.com/docs/cloud/manage/api/swagger#tag/ClickPipes), as well as the ClickHouse Terraform Provider ([3.8.3-alpha1](https://github.com/ClickHouse/terraform-provider-clickhouse/releases/tag/v3.8.3-alpha1)+).

## How does it work?

Building a connector for a cloud data warehouse presents a unique challenge: moving (potentially very) large datasets efficiently while minimizing upstream costs. The new ClickPipes connector uses BigQuery's native export jobs (**free!**) to avoid incurring query processing charges.

![image1.jpg](/uploads/image1_a424ad43f3.jpg)

***Bigquery ClickPipe:** the user configures a ClickPipe to point to the desired tables in BigQuery, providing a service account key for authentication (1). A BigQuery job exports the data into an intermediate GCS bucket (2-3). ClickPipes reads from the staging bucket (4) and inserts data into ClickHouse Cloud tables (5). Authentication between services is managed via IAM.*

**Setup requirements:**

* A BigQuery project with the dataset(s) you want to export tables from, and a [ClickHouse Cloud account](https://clickhouse.com/cloud).
* A Google Cloud service account with BigQuery `dataViewer` and `jobUser` permissions, as well as GCS `objectAdmin` and bucketViewer permissions.  
* A dedicated GCS bucket for intermediate staging. In Private Preview, this bucket must be provisioned in your GCP account. In the future, ClickPipes will provision and manage the staging bucket.

## Next up: incremental syncs

<p>
After a successful onboarding, the most common request we get from customers is performing incremental loads to keep BigQuery and ClickHouse in sync. Unlike databases with transaction logs, BigQuery requires querying tables to detect changes, or using advanced ($$) features like time travel — all of which drive up costs. &#128200; We're evaluating strategies for query-based CDC like timestamp-based incremental loads for append-only tables, partition-aware sync for time-partitioned data, and metadata-driven triggers to minimize query costs. <strong>If incremental syncs are critical for your use case, we’d love to work with you as a design partner</strong>.
</p>

In the meantime, we recommend [using the Google Cloud Storage ClickPipe](https://clickhouse.com/docs/integrations/clickpipes/object-storage/gcs/overview) to continuously sync BigQuery data exports to ClickHouse Cloud, once the initial load via the BigQuery ClickPipes is completed. We’ll publish guides on how to implement this pattern in the next few weeks!

## How to sign up for Private Preview?

The BigQuery connector is in Private Preview and available upon request. Fill out [this form](https://clickhouse.com/cloud/clickpipes/bigquery-connector) to join the waitlist, or reach out to your ClickHouse Cloud account manager! We'll work through requests as soon as possible, and expect to reach out to you, worst case, within a few days of signing up.

We're actively looking for design partners to help shape this connector, in particular teams who need incremental syncs, or are planning large-scale BigQuery migrations. Your requirements and feedback will directly influence what we build next!


In our benchmarks, ClickHouse delivers up to 4x faster queries with significant cost savings for interactive analytics. 

Ready to test ClickHouse with your BigQuery data?

Load data from BigQuery into ClickHouse Cloud in a few clicks for fast exploration and prototyping. The connector simplifies data migration for testing ClickHouse's real-time query performance on your BigQuery datasets.

large_image3.jpg

medium_image3.jpg

small_image3.jpg

thumbnail_image3.jpg

image3.jpg

BigQuery connector for ClickPipes is now in Private Preview

Replicate data from MongoDB into ClickHouse Cloud in just a few clicks for blazing-fast analytics on document-based data. Now with more connectivity options, improved reliability, and support for DocumentDB.

<p>The <a href="https://clickhouse.com/cloud/clickpipes">ClickPipes MongoDB CDC connector</a> is now in Public Beta! &#128640; </p>

After working with dozens of early access customers during Private Preview, we've added support for **sharded clusters** and **Amazon DocumentDB**, improved **reliability** for production workloads, and made it easier to **securely connect** to private MongoDB deployments.

Whether you're running a one-time migration or continuously replicating operational data for analytics, the connector delivers up to **100x faster queries** without the overhead of managing custom pipelines.

> [Sign up for ClickHouse Cloud](https://console.clickhouse.cloud/signup) today to try out the [MongoDB CDC connector for ClickPipes](https://clickhouse.com/cloud/clickpipes)!

<iframe width="768" height="432" src="https://www.youtube.com/embed/ZWM6xsaF23M?si=TlAAhBb7J5aav-pI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

Here’s what our customers are saying:

>Before using ClickPipes, we relied on multiple tools to replicate MongoDB data into ClickHouse, many of which required constant maintenance or manual intervention. After switching to the MongoDB connector in ClickPipes, those issues disappeared: the initial setup was simple, and once configured, it required no ongoing attention. Using materialized views makes data modeling flexible and easy to evolve.
> 
> <br />
>Overall, ClickPipes has significantly reduced operational overhead and has worked great to replicate several terabytes of business data a day for analytical workloads. - <a href="https://www.rapidata.ai">Rapidata</a>

## What’s new in Public Beta?

### Broader platform compatibility

The MongoDB CDC connector initially launched with support for single [replica sets](https://www.mongodb.com/docs/manual/core/replica-set-members/). We’ve now extended support to [**sharded clusters**](https://www.mongodb.com/docs/manual/core/sharded-cluster-components/), enabling replication from common MongoDB topologies at any scale — whether you're running a single replica set or a distributed deployment with data spread across multiple shards.

We've also added compatibility with **Amazon DocumentDB**, a managed document database service that is API-compatible with MongoDB. See our [new integration guide](https://clickhouse.com/docs/integrations/clickpipes/mongodb/source/documentdb) for step-by-step instructions on how to configure DocumentDB for change stream replication.

### Secure connectivity options

To allow secure connections to MongoDB instances in private networks, which is a table stakes requirement for production environments, you can now configure both **AWS PrivateLink** and **SSH tunneling** when creating a new MongoDB ClickPipe. We've also added support for [X.509 certificate authentication](https://www.mongodb.com/docs/manual/core/security-x.509/), enabling **mutual TLS authentication** between the connector and your MongoDB deployment.

### Production-readiness

We've made significant improvements to reliability and observability based on Private Preview feedback. The connector now handles <strong>data type edge cases</strong> more gracefully (*e.g.* very large floats, dates outside the standard range) and uses enhanced error classification to <strong>automatically retry transient failures</strong> that would previously trigger unnecessary alerts.

<p>We've also refined the initial snapshot phase with smarter batching logic that measures uncompressed data size to <strong>prevent out-of-memory issues</strong>. This addresses the most significant operational issue we identified with large-scale deployments during early access: MongoDB's high JSON compression ratios caused batch sizes to balloon unexpectedly when decompressed during ingestion of very large collections. &#128165;</p>

## Main features

In addition to these Public Beta improvements, the MongoDB CDC connector provides several core capabilities designed to make real-time replication from MongoDB to ClickHouse Cloud reliable and performant:

### Real-time replication

The connector leverages MongoDB's native [Change Streams](https://www.mongodb.com/docs/manual/changestreams/) interface to capture all document changes with low latency, ensuring near real-time synchronization between MongoDB and ClickHouse Cloud. This combination creates a powerful architecture for modern applications that need both operational flexibility and analytical performance, with up to 100x faster analytics — we’re calling it the *Document Data Stack*.

![image1.png](/uploads/image1_cba4cd460a.png)

Keep your MongoDB clusters focused on serving mission-critical operational workloads, including high-throughput CRUD operations and efficient document lookups, while ClickHouse Cloud handles complex analytical queries, reports, and business intelligence workloads without impacting upstream performance.

### Advanced JSON Support

To preserve MongoDB’s rich document structures and provide high-precision replication, the connector uses ClickHouse's powerful [native JSON data type](https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse). This enables **high-performance** analytical queries on semi-structured data, at the same time o**ptimizing compression and reducing storage cost**.

### Fully managed experience

The connector is fully integrated into the ClickHouse Cloud experience. It provides **built-in metrics and monitoring** for visibility into replication health, **detailed logs** for error diagnosis and debugging, **in-place pipe editing**, and more.

## What’s next?

As we work towards General Availability (GA), we're focused on making the connector faster and more flexible for production workloads.

* **Parallel snapshot ingestion:** as is, the connector snapshots collections using a single thread, which can lead to a very long initial snapshotting phase. We’re evaluating (logical) partitioning strategies to improve initial load times for large collections.

* **Flattened mode:** we’ll add an option to automatically map top-level document fields to target columns. This will make it easier to model and query replicated data, while preserving schema evolution capabilities.

* **OpenAPI and Terraform support:** for teams managing infrastructure as code, MongoDB ClickPipes will also be available via Open API and Terraform, similar to other ClickPipe types.

**A note on billing:** usage of MongoDB CDC ClickPipes continues to be **free** until GA. Customers will be notified ahead of the GA launch to review and optimize their ClickPipes usage. You can estimate future costs by referring to [billing for Postgres CDC ClickPipes](https://clickhouse.com/docs/cloud/reference/billing/clickpipes).

## Getting started with the MongoDB CDC connector

The MongoDB CDC connector is available to new and existing ClickHouse Cloud customers, in all service tiers. To get started, navigate to the *Data Sources* tab in the ClickHouse Cloud console, configure the connection details for your MongoDB database, and you’re good to go! For step-by-step instructions, frequently asked questions, and gotchas, check out the [documentation for MongoDB ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/mongodb).


Ready to accelerate analytics on MongoDB data? Try the MongoDB CDC connector today and experience a fully managed, native integration experience with ClickHouse Cloud - the world’s fastest analytics database.

Try the MongoDB CDC connector today

large_MongoDB ClickPipe Banner.jpg

medium_MongoDB ClickPipe Banner.jpg

small_MongoDB ClickPipe Banner.jpg

thumbnail_MongoDB ClickPipe Banner.jpg

MongoDB ClickPipe Banner.jpg

MongoDB CDC connector for ClickPipes is now in Public Beta

Ingest data from Amazon S3 into ClickHouse Cloud in **any** **order** for event-driven, blazing-fast analytics. Files are no longer required to follow lexicographical order.

When you’re on the receiving end of a data integration pipeline, you quickly learn that real-world data knows no rules. Sometimes it arrives, sometimes it doesn’t. Then it arrives twice, or maybe out-of-order. For object storage, in particular, this creates an interesting problem: how do you efficiently track what's new without scanning too many objects all the time?

Like most tools out there, [ClickPipes](https://clickhouse.com/cloud/clickpipes?utm_source=google.com&utm_medium=paid_search&utm_campaign=21862172336_181837693625&utm_content=764403839926&utm_term=clickpipes_g_c&gad_source=1&gad_campaignid=21862172336&gclid=CjwKCAjwisnGBhAXEiwA0zEOR3kM4ZBQO0_NpfRPpYU2YWG5onZqwF-Kf8m845Ol2IJeyN96r0k8DxoCit4QAvD_BwE) didn’t have a good solution for this problem, and simply expected you to ensure there were *some* rules to guarantee files landed in your bucket in lexicographical order. **Not anymore!**

![image6.png](/uploads/image6_b5104fce2a.png)

We’ve added a new mode to the S3 ClickPipe that allows ingesting files in any order using S3 Event Notifications via [Amazon SQS](https://aws.amazon.com/sqs/). Whether you’re dealing with backfills, retries, late-arriving data, or some other source of out-of-orderness — we handle it.

Sign up for ClickHouse Cloud

## Why is this a big deal?

By default, the S3 ClickPipe assumes files are added to a bucket in lexicographical order, and relies on this implicit order to ingest files sequentially. Besides not rolling off the tongue, lexicographical order means that any new file *must* be lexically greater than the last ingested file. For example, files named `events_2024-12-01.parquet`, `events_2024-12-02.parquet`, and `events_2024-12-03.parquet` will be ingested in order, but if a backfill named `events_2024-11-30.parquet` lands later in the bucket, it will be ignored.

This is not great.

With unordered mode, this limitation no longer applies: instead of polling your bucket every 30 seconds looking for the next file to process, the ClickPipe waits to be notified that a new file has landed. When a notification arrives, ClickPipes just goes and processes that file regardless of its relative order to previously processed files. Although this requires you to configure event notifications, it's a more robust and performant alternative to the otherwise common approach of listing all objects in scope on every sync — which scales poorly and gets expensive fast when you have millions of objects.

## How does it work?

To configure an S3 ClickPipe to ingest files that don’t have an implicit order, you need to set up an [Amazon Simple Queue System (SQS)](https://aws.amazon.com/sqs/) queue connected to the bucket. ClickPipes can then listen for `ObjectCreated` events in the queue that match the specified path.  

![ClickPipes S3 Blog Banner.jpg](/uploads/Click_Pipes_S3_Blog_Banner_08474993cb.jpg)

**Unordered mode:** Files land in S3 in **any order** (A) and trigger SQS notifications (B-C). ClickPipes polls and processes files in the specified path, using a metadata store to track state (1-5). Data is inserted into the target tables with exactly-once guarantees (6).

***“What about failures?”*** Since the steps above span multiple systems and don’t happen in one big transaction™️, failures can occur at any step — reading from S3, marking files as processed, inserting data into ClickHouse, and so on. If a failure occurs, ClickPipes automatically reprocesses the batch. *“**But what about duplicates?”*** Even if files are reprocessed multiple times, the S3 ClickPipe guarantees exactly-once semantics, so no duplicates make it into your target table.

Let’s see it working!

### Create an Amazon SQS queue

*The following instructions assume you already have an S3 bucket with some data in it, as well as enough permissions to manage IAM roles and create new resources in your AWS account.*

**1.** In the AWS Console, navigate to **Simple Queue Service > Create queue**. Use the defaults to create a new queue.

**2.** Edit the queue trust policy to allow your bucket (`<bucket-arn>`) in your AWS account (`<aws-account-id>`) to send messages to the SQS queue (`<queue-arn>`).

```
{
    "Version":"2012-10-17",                   
    "Id": "example-ID",
    "Statement": [
        {
            "Sid": "AllowS3ToSendMessage",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "SQS:SendMessage"
            ],
            "Resource": "<sqs-queue-arn>",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "<bucket-arn>"
                },
                "StringEquals": {
                    "aws:SourceAccount": "<aws-account-id>"
                }
            }
        }
    ]
}
```

**3.** In the ClickHouse Cloud console, navigate to **Settings > Network security information** and copy the IAM role ARN for your service.

![image2.png](/uploads/image2_2accfed654.png)

**4.** Back in the AWS Console, go to **IAM > Roles > Create role**. Choose **Custom trust policy** and paste the IAM role ARN for your ClickHouse Cloud service.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAssumeRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<ch-cloud-arn>"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

**5.** Create an inline policy for the IAM role with the required permissions to scan and fetch objects from the S3 bucket and manage messages in the SQS queue.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3BucketMetadataAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": "<bucket-arn>"
    },
    {
      "Sid": "AllowGetListObjects",
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": "<bucket-arn>/*"
    },
    {
      "Sid": "SQSNotificationsAccess",
      "Effect": "Allow",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:ListQueues",
        "sqs:ReceiveMessage",
        "sqs:GetQueueAttributes"
      ],
    "Resource": "<sqs-queue-arn>"
    }
  ]
}
```

That was a lot of IAM config, but you’re ready to create a ClickPipe to continuously ingest data from your bucket whenever a new file lands!

### Create a ClickPipe

**1.** In the ClickHouse Cloud console, navigate to **Data Sources > Create ClickPipe**, then choose **Amazon S3**. Enter the details to connect to your S3 bucket, using **IAM role** as the authentication method with the ARN role you created in the previous section.

![image5.png](/uploads/image5_19483129ab.png)

**2.** Click **Incoming data**. Toggle on **Continuous ingestion**, where you’ll see the new **Any order** ingestion option.

![image3.png](/uploads/image3_40f8b8de3b.png)

**3.** Click **Parse information**. Define a **Sorting key** for the target table (very important) and make any necessary adjustments to the mapped schema. Finally, configure a role for the ClickPipes database user.

![image1.gif](/uploads/image1_f958d131c1.gif)

<p>
<strong>4. Sit back and relax.</strong> ClickPipes will now perform an initial scan of your bucket, then start processing files as new notification events arrive. &#128640;
</p>

If that seems like a lot of clicking, the good news is that ClickPipes is fully supported in the [ClickHouse Terraform provider](https://registry.terraform.io/providers/ClickHouse/clickhouse/latest/docs), so you can configure this setup as code from start to finish.

## What’s next?

We’re excited to enable more complex ingestion patterns from object storage into ClickHouse Cloud with the new unordered mode in S3 ClickPipes — you get the same fully-managed, full-speed experience with a little extra flexibility\! For now, this feature is only available for Amazon S3 as a data source. We plan to extend support to Google Cloud Storage (GCS) and Azure Blob Storage (ABS) in the near future, to ensure feature parity across all Object Storage ClickPipes.

If you have any feedback or run into any snags while setting up ClickPipes, reach out to our team\! For step-by-step instructions, frequently asked questions, and gotchas, check out the [documentation for S3 ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/object-storage/s3/overview#continuous-ingestion-any-order).

Ready to eliminate your ETL complexity and reduce your data movement costs? Try the S3 ClickPipe connector today and experience a fully managed, native integration experience with ClickHouse Cloud - the world’s fastest analytics database.

Ingest data from Amazon S3 into ClickHouse Cloud in any order for event-driven, blazing-fast analytics.

Marta Paes

Blog posts by Marta Paes

GCP Pub/Sub connector for ClickPipes is now in Private Preview

Terraform & OpenAPI for ClickPipes is now Generally Available

Unordered mode for GCS ClickPipes is now available

BigQuery connector for ClickPipes is now in Private Preview

MongoDB CDC connector for ClickPipes is now in Public Beta

Unordered mode for S3 ClickPipes is now available