Kafka ClickPipes FAQ

General

How does ClickPipes for Kafka work?

ClickPipes uses a dedicated architecture running the Kafka Consumer API to read data from a specified topic and then inserts the data into a ClickHouse table on a specific ClickHouse Cloud service.

What's the difference between ClickPipes and the ClickHouse Kafka Table Engine?

The Kafka Table engine is a ClickHouse core capability that implements a "pull model" where the ClickHouse server itself connects to Kafka, pulls events then writes them locally.

ClickPipes is a separate cloud service that runs independently of the ClickHouse Service, it connects to Kafka (or other data sources) and pushes events to an associated ClickHouse Cloud service. This decoupled architecture allows for superior operational flexibility, clear separation of concerns, scalable ingestion, graceful failure management, extensibility and more.

What are the requirements for using ClickPipes for Kafka?

In order to use ClickPipes for Kafka, you will need a running Kafka broker and a ClickHouse Cloud service with ClickPipes enabled. You will also need to ensure that ClickHouse Cloud can access your Kafka broker. This can be achieved by allowing remote connection on the Kafka side, whitelisting ClickHouse Cloud Egress IP addresses in your Kafka setup. Alternatively, you can use AWS PrivateLink to connect ClickPipes for Kafka to your Kafka brokers.

Does ClickPipes for Kafka support AWS PrivateLink?

AWS PrivateLink is supported. See the documentation for more information on how to set it up.

Can I use ClickPipes for Kafka to write data to a Kafka topic?

No, the ClickPipes for Kafka is designed for reading data from Kafka topics, not writing data to them. To write data to a Kafka topic, you will need to use a dedicated Kafka producer.

Does ClickPipes support multiple brokers?

Yes, if the brokers are part of the same quorum they can be configured together delimited with ,.

Can ClickPipes replicas be scaled?

Yes, ClickPipes for streaming can be scaled both horizontally and vertically. Horizontal scaling adds more replicas to increase throughput, while vertical scaling increases the resources (CPU and RAM) allocated to each replica to handle more intensive workloads. This can be configured during ClickPipe creation, or at any other point under Settings -> Advanced Settings -> Scaling.

Upstash

Does ClickPipes support Upstash?

Yes. The Upstash Kafka product entered into a deprecation period on 11th September 2024 for 6 months. Existing customers can continue to use ClickPipes with their existing Upstash Kafka brokers using the generic Kafka tile on the ClickPipes user interface. Existing Upstash Kafka ClickPipes are unaffected before the deprecation notice. When the the deprecation period is up the ClickPipe will stop functioning.

Does ClickPipes support Upstash schema registry?

No. ClickPipes is not Upstash Kafka schema registry compatible.

Does ClickPipes support the Upstash QStash Workflow?

No. Unless a Kafka compatible surface is introduced in QStash Workflow it will not work with Kafka ClickPipes.

Azure EventHubs

Does the Azure Event Hubs ClickPipe work without the Kafka surface?

No. ClickPipes requires the Azure Event Hubs to have the Kafka surface enabled. The Kafka protocol is supported for their Standard, Premium and Dedicated SKU only pricing tiers.

Does Azure schema registry work with ClickPipes

No. ClickPipes is not currently Event Hubs Schema Registry compatible.

What permissions does my policy need to consume from Azure Event Hubs?

To list topics and consume event, the shared access policy that is given to ClickPipes will at minimum require a 'Listen' claim.

Why is my Event Hubs not returning any data?

If your ClickHouse instance is in a different region or continent from your Event Hubs deployment, you may experience timeouts when onboarding your ClickPipes, and higher-latency when consuming data from the Event Hub. It is considered a best practice to locate your ClickHouse Cloud deployment and Azure Event Hubs deployment in cloud regions located close to each other to avoid adverse performance.

Should I include the port number for Azure Event Hubs?

Yes. ClickPipes expects you to include your port number for the Kafka surface, which should be :9093.

Are the ClickPipes IPs still relevant for Azure Event Hubs?

Yes. If you restrict traffic to your Event Hubs instance please add the [documented static NAT IPs](../ /index.md#list-of-static-ips).

Is the connection string for the Event Hub, or is it for the Event Hub namespace?

Both will work, however, we recommend using a shared access policy at the namespace level to retrieve samples from multiple Event Hubs.