Schema registries for Kafka ClickPipe

​

Supported registries for Kafka ClickPipes

• Confluent-compatible registries: any registry that is API-compatible with the Confluent Schema Registry, such as the Confluent Schema Registry itself and the Redpanda Schema Registry. Supports Avro and Protobuf.
• AWS Glue Schema Registry: for Avro data serialized with the AWS Glue SerDe, typically from Amazon MSK.

​

Confluent-compatible registries

​

Configuration

1. Provide a complete path to the schema subject (e.g. https://registry.example.com/subjects/events)
   • Optionally, a specific version can be referenced by appending /versions/[version] to the URL (otherwise ClickPipes will retrieve the latest version).
2. Provide a complete path to the schema ID (e.g. https://registry.example.com/schemas/ids/1000)
3. Provide the root schema registry URL (e.g. https://registry.example.com)

​

Network connectivity

• The schema registry URL must use https://.
• If the registry hostname resolves to a private address, it must be reachable through a reverse private endpoint selected for the ClickPipe; otherwise the connectivity check during setup will fail.

​

How it works

• If there’s a schema ID embedded in the message, it will use that to retrieve the schema.
• If there’s no schema ID embedded in the message, it will use the schema ID or subject name specified in the ClickPipe configuration to retrieve the schema.
• If the message is written without an embedded schema ID, and no schema ID or subject name is specified in the ClickPipe configuration, then the schema will not be retrieved and the message will be skipped with a SOURCE_SCHEMA_ERROR logged in the ClickPipes error table.
• If the message does not conform to the schema, then the message will be skipped with a DATA_PARSING_ERROR logged in the ClickPipes error table.
• For Protobuf schemas only: ClickPipes will load any imported schemas defined as dependencies. Avro schemas with external references are not yet supported.

​

AWS Glue Schema Registry

​

Configuration

​

IAM setup

​

Option A: reuse the broker’s IAM identity

• IAM role: Add the statement to the permissions policy for the role configured for MSK.
• IAM credentials: Add the statement to the permissions policy for the IAM principal associated with the access key.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ClickPipesGlueSchemaRegistryRead",
      "Effect": "Allow",
      "Action": ["glue:GetSchemaVersion"],
      "Resource": "*"
    }
  ]
}

​

Option B: use a dedicated registry role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "{ClickHouse_IAM_ARN}"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ClickPipesGlueSchemaRegistryRead",
      "Effect": "Allow",
      "Action": ["glue:GetSchemaVersion"],
      "Resource": "*"
    }
  ]
}

​

Troubleshooting

​

Limitations

• Avro only. JSON Schema and Protobuf via the Glue SerDe are not supported.
• Kafka sources only. Kinesis ClickPipes cannot use a Glue registry.

​

Schema mapping

• If the schema contains a field that is not included in the ClickHouse destination mapping, that field is ignored.
• If the schema is missing a field defined in the ClickHouse destination mapping, the ClickHouse column will be populated with a “zero” value, such as 0 or an empty string. Note that DEFAULT expressions are not supported.
• If the schema field and the ClickHouse column are incompatible, inserts of that row/message will fail, and the failure will be recorded in the ClickPipes error table. Note that several implicit conversions are supported (e.g., between numeric types), but not all (e.g., an Avro record field cannot be inserted into an Int32 ClickHouse column).