Creating your first Kafka ClickPipe

In this guide, we will walk you through the process of creating your first Kafka ClickPipe.

Navigate to data sources

Select the Data Sources button on the left-side menu and click on "Set up a ClickPipe".

Select a data source

Select your Kafka data source from the list.

Configure the data source

Fill out the form by providing your ClickPipe with a name, a description (optional), your credentials, and other connection details.

Configure a schema registry (optional)

A valid schema is required for Avro streams. See Schema registries for more details on how to configure a schema registry.

Configure a reverse private endpoint (optional)

Configure a Reverse Private Endpoint to allow ClickPipes to connect to your Kafka cluster using AWS PrivateLink. See our AWS PrivateLink documentation for more information.

Select your topic

Select your topic and the UI will display a sample document from the topic.

Configure your destination table

In the next step, you can select whether you want to ingest data into a new ClickHouse table or reuse an existing one. Follow the instructions in the screen to modify your table name, schema, and settings. You can see a real-time preview of your changes in the sample table at the top.

You can also customize the advanced settings using the controls provided

Configure permissions

ClickPipes will create a dedicated user for writing data into a destination table. You can select a role for this internal user using a custom role or one of the predefined role:

• Full access: with the full access to the cluster. It might be useful if you use Materialized View or Dictionary with the destination table.
• Only destination table: with the INSERT permissions to the destination table only.

Complete setup

Clicking on "Create ClickPipe" will create and run your ClickPipe. It will now be listed in the Data Sources section.