ClickPipes OpenAPI reference

All ClickPipes types can be created, updated, and deleted programmatically using the ClickHouse Cloud OpenAPI. This page covers authentication and the available ClickPipes endpoints, with example curl requests for each ClickPipe type.

Authentication

The ClickHouse Cloud API uses HTTP Basic authentication. You need an API key (key ID + key secret) with permissions on the target service. See Managing API keys for instructions on creating one. Set up the following environment variables before running any of the examples below:

export KEY_ID=<your_key_id>
export KEY_SECRET=<your_key_secret>
export ORG_ID=<your_organization_id>
export SERVICE_ID=<your_service_id>

Base URL

https://api.clickhouse.cloud/v1

Endpoints

All ClickPipes endpoints are scoped to a ClickHouse Cloud service:

Method	Path	Description
`GET`	`/organizations/{organizationId}/services/{serviceId}/clickpipes`	List all ClickPipes
`POST`	`/organizations/{organizationId}/services/{serviceId}/clickpipes`	Create a ClickPipe
`GET`	`/organizations/{organizationId}/services/{serviceId}/clickpipes/context`	Get service capabilities and the GCP workload identity principal
`GET`	`/organizations/{organizationId}/services/{serviceId}/clickpipes/{clickPipeId}`	Get a ClickPipe
`PATCH`	`/organizations/{organizationId}/services/{serviceId}/clickpipes/{clickPipeId}`	Update a ClickPipe
`DELETE`	`/organizations/{organizationId}/services/{serviceId}/clickpipes/{clickPipeId}`	Delete a ClickPipe
`GET`	`/organizations/{organizationId}/services/{serviceId}/clickpipes/{clickPipeId}/settings`	Get ClickPipe settings
`PUT`	`/organizations/{organizationId}/services/{serviceId}/clickpipes/{clickPipeId}/settings`	Update ClickPipe settings
`PATCH`	`/organizations/{organizationId}/services/{serviceId}/clickpipes/{clickPipeId}/scaling`	Update ClickPipe scaling
`PATCH`	`/organizations/{organizationId}/services/{serviceId}/clickpipes/{clickPipeId}/state`	Update ClickPipe state (start/stop)

For CDC ClickPipes (Postgres, MySQL, MongoDB), there are additional org-level endpoints for shared CDC infrastructure scaling:

Method	Path	Description
`GET`	`/organizations/{organizationId}/clickpipes/cdcScaling`	Get CDC ClickPipes scaling
`PATCH`	`/organizations/{organizationId}/clickpipes/cdcScaling`	Update CDC ClickPipes scaling

For the full request and response schemas for each endpoint, see the API reference.

Examples

List ClickPipes

curl -u "$KEY_ID:$KEY_SECRET" \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

Get a ClickPipe

CLICKPIPE_ID=<your_clickpipe_id>

curl -u "$KEY_ID:$KEY_SECRET" \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes/$CLICKPIPE_ID"

Stop or start a ClickPipe

# Stop
curl -u "$KEY_ID:$KEY_SECRET" \
  -X PATCH \
  -H "Content-Type: application/json" \
  -d '{"action": "stop"}' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes/$CLICKPIPE_ID/state"

# Start
curl -u "$KEY_ID:$KEY_SECRET" \
  -X PATCH \
  -H "Content-Type: application/json" \
  -d '{"action": "start"}' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes/$CLICKPIPE_ID/state"

Delete a ClickPipe

curl -u "$KEY_ID:$KEY_SECRET" \
  -X DELETE \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes/$CLICKPIPE_ID"

GCP workload identity (Private Preview)

Follow the GCP workload identity guide to retrieve the ClickPipes-managed service account and grant it access to your source before using the workload identity examples below.

Creating ClickPipes

The POST /clickpipes request body varies by source type. The examples below show the structure for each supported ClickPipe type. For the authoritative JSON schemas, refer to the API reference.

Kafka

Supported Kafka-compatible sources: kafka, confluent, msk, azureeventhub, redpanda, warpstream, and Private Preview gcmk.

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Kafka ClickPipe",
    "source": {
      "kafka": {
        "type": "confluent",
        "format": "JSONEachRow",
        "brokers": "broker.example.com:9092",
        "topics": "my_topic",
        "consumerGroup": "clickpipes-consumer-group",
        "authentication": "PLAIN",
        "credentials": {
          "username": "my_user",
          "password": "my_password"
        }
      }
    },
    "destination": {
      "table": "my_table",
      "managedTable": true,
      "tableDefinition": {
        "engine": { "type": "MergeTree" }
      },
      "columns": [
        { "name": "id", "type": "UInt64" },
        { "name": "message", "type": "String" },
        { "name": "timestamp", "type": "DateTime" }
      ]
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

For Google Managed Kafka, grant the workload identity principal roles/managedkafka.client on the cluster project and configure the required Kafka ACLs. Then use this source object without a credentials field:

{
  "kafka": {
    "type": "gcmk",
    "format": "JSONEachRow",
    "brokers": "bootstrap.kafka.us-central1.managedkafka.example-project.cloud.goog:9092",
    "topics": "events",
    "authentication": "SERVICE_ACCOUNT_WORKLOAD_IDENTITY"
  }
}

Google Managed Kafka workload identity requires TLS.

Amazon Kinesis

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Kinesis ClickPipe",
    "source": {
      "kinesis": {
        "format": "AvroConfluent",
        "streamName": "my-stream",
        "region": "us-east-1",
        "iteratorType": "TRIM_HORIZON",
        "authentication": "IAM_USER",
        "schemaRegistry": {
          "type": "glue",
          "glueRegion": "us-east-1",
          "glueRegistryName": "my-registry"
        },
        "accessKey": {
          "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
          "secretKey": "<secret_key>"
        }
      }
    },
    "destination": {
      "table": "my_table",
      "managedTable": true,
      "tableDefinition": {
        "engine": { "type": "MergeTree" }
      },
      "columns": [
        { "name": "id", "type": "UInt64" },
        { "name": "message", "type": "String" }
      ]
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

This example configures Avro with AWS Glue Schema Registry. To decode Protobuf using Glue, set format to Protobuf and retain schemaRegistry. To upload a Protobuf schema instead, set format to Protobuf and provide the base64-encoded .proto file or serialized FileDescriptorSet in protobufSchema; do not include schemaRegistry in that request.

Amazon S3

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My S3 ClickPipe",
    "source": {
      "objectStorage": {
        "type": "s3",
        "url": "https://my-bucket.s3.amazonaws.com/data/*.json",
        "format": "JSONEachRow",
        "authentication": "IAM_USER",
        "accessKey": {
          "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
          "secretKey": "<secret_key>"
        }
      }
    },
    "destination": {
      "table": "my_table",
      "managedTable": true,
      "tableDefinition": {
        "engine": { "type": "MergeTree" }
      },
      "columns": [
        { "name": "id", "type": "UInt64" },
        { "name": "message", "type": "String" }
      ]
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

Google Cloud Storage

serviceAccountKey must be the base64-encoded content of a GCP service account JSON key file.

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My GCS ClickPipe",
    "source": {
      "objectStorage": {
        "type": "gcs",
        "url": "gs://my-bucket/data/*.json",
        "format": "JSONEachRow",
        "authentication": "SERVICE_ACCOUNT",
        "serviceAccountKey": "<base64_encoded_service_account_json>"
      }
    },
    "destination": {
      "table": "my_table",
      "managedTable": true,
      "tableDefinition": {
        "engine": { "type": "MergeTree" }
      },
      "columns": [
        { "name": "id", "type": "UInt64" },
        { "name": "message", "type": "String" }
      ]
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

To use workload identity instead of a service account key, grant the principal roles/storage.objectViewer on the source bucket and use this source object:

{
  "objectStorage": {
    "type": "gcs",
    "url": "gs://my-bucket/data/*.json",
    "format": "JSONEachRow",
    "authentication": "SERVICE_ACCOUNT_WORKLOAD_IDENTITY"
  }
}

For unordered ingestion, also grant the principal roles/pubsub.subscriber and roles/pubsub.viewer on the notification subscription. Set isContinuous to true and use a fully-qualified queueUrl in the form projects/<project-id>/subscriptions/<subscription-id>.

Google Cloud Pub/Sub

The following example uses workload identity. Grant the principal the required Pub/Sub permissions before creating the ClickPipe.

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Pub/Sub ClickPipe",
    "source": {
      "pubsub": {
        "format": "JSONEachRow",
        "projectId": "my-gcp-project",
        "topic": "events",
        "authentication": "SERVICE_ACCOUNT_WORKLOAD_IDENTITY",
        "seekType": "latest"
      }
    },
    "destination": {
      "table": "my_table",
      "managedTable": true,
      "tableDefinition": {
        "engine": { "type": "MergeTree" }
      },
      "columns": [
        { "name": "id", "type": "UInt64" },
        { "name": "message", "type": "String" }
      ]
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

Azure Blob Storage

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Azure Blob ClickPipe",
    "source": {
      "objectStorage": {
        "type": "azureblobstorage",
        "azureContainerName": "my-container",
        "path": "data/*.json",
        "format": "JSONEachRow",
        "authentication": "CONNECTION_STRING",
        "connectionString": "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net"
      }
    },
    "destination": {
      "table": "my_table",
      "managedTable": true,
      "tableDefinition": {
        "engine": { "type": "MergeTree" }
      },
      "columns": [
        { "name": "id", "type": "UInt64" },
        { "name": "message", "type": "String" }
      ]
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

Postgres CDC

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Postgres CDC ClickPipe",
    "source": {
      "postgres": {
        "host": "postgres.example.com",
        "port": 5432,
        "database": "mydb",
        "credentials": {
          "username": "postgres_user",
          "password": "<password>"
        },
        "settings": {
          "replicationMode": "cdc"
        },
        "tableMappings": [
          {
            "sourceSchemaName": "public",
            "sourceTable": "users",
            "targetTable": "public_users"
          }
        ]
      }
    },
    "destination": {
      "database": "default"
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

MySQL CDC

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My MySQL CDC ClickPipe",
    "source": {
      "mysql": {
        "host": "mysql.example.com",
        "port": 3306,
        "credentials": {
          "username": "mysql_user",
          "password": "<password>"
        },
        "settings": {
          "replicationMode": "cdc"
        },
        "tableMappings": [
          {
            "sourceSchemaName": "mydb",
            "sourceTable": "orders",
            "targetTable": "mydb_orders"
          }
        ]
      }
    },
    "destination": {
      "database": "default"
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

MongoDB CDC

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My MongoDB CDC ClickPipe",
    "source": {
      "mongodb": {
        "uri": "mongodb+srv://cluster0.example.mongodb.net",
        "readPreference": "secondaryPreferred",
        "credentials": {
          "username": "mongo_user",
          "password": "<password>"
        },
        "settings": {
          "replicationMode": "cdc"
        },
        "tableMappings": [
          {
            "sourceDatabaseName": "mydb",
            "sourceCollection": "users",
            "targetTable": "mydb_users"
          }
        ]
      }
    },
    "destination": {
      "database": "default"
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

BigQuery

serviceAccountFile must be the base64-encoded content of a GCP service account JSON key file.

curl -u "$KEY_ID:$KEY_SECRET" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My BigQuery ClickPipe",
    "source": {
      "bigquery": {
        "snapshotStagingPath": "gs://my-staging-bucket/staging/",
        "credentials": {
          "serviceAccountFile": "<base64_encoded_service_account_json>"
        },
        "settings": {
          "replicationMode": "snapshot"
        },
        "tableMappings": [
          {
            "sourceDatasetName": "my_dataset",
            "sourceTable": "my_table",
            "targetTable": "my_bigquery_table"
          }
        ]
      }
    },
    "destination": {
      "database": "default"
    }
  }' \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes"

To use workload identity instead of a service account key, grant the principal the required BigQuery and GCS permissions and use this source object:

{
  "bigquery": {
    "authentication": "SERVICE_ACCOUNT_WORKLOAD_IDENTITY",
    "projectId": "my-gcp-project",
    "snapshotStagingPath": "gs://my-staging-bucket/staging/",
    "settings": {
      "replicationMode": "snapshot"
    },
    "tableMappings": [
      {
        "sourceDatasetName": "my_dataset",
        "sourceTable": "my_table",
        "targetTable": "my_bigquery_table"
      }
    ]
  }
}

ClickPipes for Streaming

ClickPipes for Object Storage

ClickPipes for CDC

Programmatic access

Security and access

Networking

Authentication

Base URL

Endpoints

Examples

List ClickPipes

Get a ClickPipe

Stop or start a ClickPipe

Delete a ClickPipe

GCP workload identity (Private Preview)

Creating ClickPipes

Kafka

Amazon Kinesis

Amazon S3

Google Cloud Storage

Google Cloud Pub/Sub

Azure Blob Storage

Postgres CDC

MySQL CDC

MongoDB CDC

BigQuery

​Authentication

​Base URL

​Endpoints

​Examples

​List ClickPipes

​Get a ClickPipe

​Stop or start a ClickPipe

​Delete a ClickPipe

​GCP workload identity (Private Preview)

​Creating ClickPipes

​Kafka

​Amazon Kinesis

​Amazon S3

​Google Cloud Storage

​Google Cloud Pub/Sub

​Azure Blob Storage

​Postgres CDC

​MySQL CDC

​MongoDB CDC

​BigQuery

Authentication

Base URL

Endpoints

Examples

List ClickPipes

Get a ClickPipe

Stop or start a ClickPipe

Delete a ClickPipe

GCP workload identity (Private Preview)

Creating ClickPipes

Kafka

Amazon Kinesis

Amazon S3

Google Cloud Storage

Google Cloud Pub/Sub

Azure Blob Storage

Postgres CDC

MySQL CDC

MongoDB CDC

BigQuery