Integrating Confluent platform with ClickHouse

​

Prerequisites

• ClickHouse Connector Sink
• Confluent Platform and Custom Connectors.

​

The official Kafka connector from ClickHouse with Confluent Platform

​

Installing on Confluent platform

​

Create a Topic

​

Important notes

• The Kafka topic name must be the same as the ClickHouse table name. The way to tweak this is by using a transformer (for example ExtractTopic).
• More partitions doesn’t always mean more performance - see our upcoming guide for more details and performance tips.

​

Install connector

'Connector Class' - 'com.clickhouse.kafka.connect.ClickHouseSinkConnector'
'Connector type' - Sink
'Sensitive properties' - 'password'. This will ensure entries of the ClickHouse password are masked during configuration.

​

Gather your connection details

​

Configure the connector

{
  "database": "<DATABASE_NAME>",
  "errors.retry.timeout": "30",
  "exactlyOnce": "false",
  "schemas.enable": "false",
  "hostname": "<CLICKHOUSE_HOSTNAME>",
  "password": "<SAMPLE_PASSWORD>",
  "port": "8443",
  "ssl": "true",
  "topics": "<TOPIC_NAME>",
  "username": "<SAMPLE_USERNAME>",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "value.converter.schemas.enable": "false"
}

​

Specify the connection endpoints

​

Known limitations

• Custom Connectors must use public internet endpoints. Static IP addresses aren’t supported.
• You can override some Custom Connector properties. See the fill list in the official documentation.
• Custom Connectors are available only in some AWS regions
• See the list of Custom Connectors limitations in the official docs