Configuring automatic TLS provisioning via ACME

Experimental feature. Learn more.

Not supported in ClickHouse Cloud

Note

This page is not applicable to ClickHouse Cloud. The procedure documented here is automated in ClickHouse Cloud services.

This guide provides describes how to configure ClickHouse to use ACME (described by RFC8555) protocol. With ACME support, ClickHouse can obtain and renew certificates from providers like Let's Encrypt or ZeroSSL automatically. TLS encryption protects data in transit between clients and ClickHouse servers, preventing eavesdropping on sensitive queries and results.

Overview

ACME protocol defines automatic certificate update process with services like Let's Encrypt or ZeroSSL. In short, ClickHouse as certificate requester needs to confirm domain ownership via predefined challenge types in order to get a certificate.

To enable ACME, configure HTTP and HTTPS ports along with the acme block:

<http_port>80</http_port>
<https_port>443</https_port>

<acme>
    <email>[email protected]</email>
    <terms_of_service_agreed>true</terms_of_service_agreed>
    <domains>
        <domain>example.com</domain>
    </domains>
</acme>

The HTTP port serves ACME HTTP-01 challenge (more on challenge types here) requests during domain validation. Once validation completes and a certificate is issued, the HTTPS port serves encrypted traffic using the obtained certificate.

The HTTP port does not need to be 80 on the server itself; it may be remapped using nftables or similar tools. Check your ACME provider's documentation for accepted ports for HTTP-01 challenges.

In the acme block, we're defining email for account creation, and accepting ACME service terms of service. After that, the only thing we need is a list of domains.

Current limitations

• Only HTTP-01 challenge type is supported.
• Only RSA 2048 keys are supported.
• Rate limiting is not handled.

Configuration parameters

Configuration options available in acme section:

Parameter	Default value	Description
zookeeper_path	/clickhouse/acme	ZooKeeper path used to store ACME account data, certificates, and coordination state between ClickHouse nodes.
directory_url	https://acme-v02.api.letsencrypt.org/directory	ACME directory endpoint used for certificate issuance. Defaults to the Let’s Encrypt production server.
email		Email address used to create and manage the ACME account. ACME providers may use it for expiration notices and important updates.
terms_of_service_agreed	false	Indicates whether the ACME provider’s Terms of Service are accepted. Must be set to true to enable ACME.
domains		List of domain names for which TLS certificates should be issued. Each domain is specified as a <domain> entry.
refresh_certificates_before	2592000 (one month, in seconds)	Time before certificate expiration when ClickHouse will attempt to renew the certificate.
refresh_certificates_task_interval	3600 (one hour, in seconds)	Interval at which ClickHouse checks whether certificates need renewal.

Note that configuration uses Let's Encrypt production directory by default. To avoid hitting request quota due to probable misconfiguration, it is recommended to test certificate issuing process with staging directory first.

Administration

Initial deployment

When enabling the ACME client on a cluster with multiple replicas, additional care is required during the initial certificate issuance.

The first replica that starts with ACME enabled will immediately attempt to create an ACME order and perform HTTP-01 challenge validation. If only a subset of replicas is serving traffic at that moment, the challenge is likely to fail, as other replicas will not be able to respond to validation requests.

If possible, it is recommended to temporarily route traffic to a single replica (for example, by adjusting DNS records) and allow it to complete the initial certificate issuance. Once the certificate is successfully issued and stored in Keeper, ACME can be enabled on the remaining replicas. They will automatically reuse the existing certificate and participate in future renewals.

If routing traffic to a single replica is not feasible, an alternative approach is to manually upload the existing certificate and private key into Keeper before enabling the ACME client. This avoids the initial validation step and allows all replicas to start with a valid certificate already present.

After the initial certificate has been issued or imported, certificate renewal does not require special handling, as all replicas will already be running the ACME client and sharing state through Keeper.

Keeper data structure

/clickhouse/acme
└── <acme-directory-host>
    ├── account_private_key          # ACME account private key (PEM)
    ├── challenges                   # Active HTTP-01 challenge state
    └── domains
        └── <domain-name>
            ├── certificate          # Issued TLS certificate (PEM)
            └── private_key          # Domain private key (PEM)

Migrating from other ACME clients

It is possible to migrate current TLS certificate and key to Keeper for easier migration. At the moment, server supports only RSA 2048 keys.

Assuming we're migrating from certbot and using /etc/letsencrypt/live directory, one may use the following set of commands:

DOMAIN=example.com
CERT_DIR=/etc/letsencrypt/live/$DOMAIN
ZK_BASE=/clickhouse/acme/acme-v02.api.letsencrypt.org/domains/$DOMAIN

clickhouse keeper-client -q "create '/clickhouse' ''"
clickhouse keeper-client -q "create '/clickhouse/acme' ''"
clickhouse keeper-client -q "create '/clickhouse/acme/acme-v02.api.letsencrypt.org' ''"
clickhouse keeper-client -q "create '/clickhouse/acme/acme-v02.api.letsencrypt.org/domains' ''"
clickhouse keeper-client -q "create '$ZK_BASE' ''"

clickhouse keeper-client -q "create '$ZK_BASE/certificate' \"$(cat $CERT_DIR/fullchain.pem)\""
clickhouse keeper-client -q "set '$ZK_BASE/certificate' \"$(cat $CERT_DIR/fullchain.pem)\""

clickhouse keeper-client -q "create '$ZK_BASE/private_key' \"$(cat $CERT_DIR/privkey.pem)\""
clickhouse keeper-client -q "set '$ZK_BASE/private_key' \"$(cat $CERT_DIR/privkey.pem)\""