Deploy ClickHouse with FIPS Compliance (Government)

This tutorial covers deploying ClickHouse Private with FIPS 140-3 compliance on AWS. It builds on the standard AWS deployment as an overlay — you must complete the AWS resource setup first, then follow these FIPS-specific steps for the Kubernetes resources.

Deploying on GKE?The FIPS-specific Kubernetes steps below — certificate generation, certificate secrets, and the openSSL Helm flags — are cloud-agnostic. For GCP, provision the infrastructure with Deploy on GCP GKE, then apply those steps here, substituting the GCS storage settings and Workload Identity from the GCP guide.

Prerequisites

Before starting this tutorial, you must have completed the following from tutorials/deploy-aws.md:

Step 1: IAM role for ECR access
Step 2: Container images copied to your ECR (using FIPS version tags — see below)
Step 3: VPC created
Step 4: EKS cluster created
Step 5: Node groups created
Step 6: S3 bucket and IAM roles created

FIPS Version Tags

Use the following FIPS-specific image tags when copying artifacts:

Artifact	Tag
clickhouse-server	`<<SERVER_TAG>>-fips`
clickhouse-keeper	`<<KEEPER_TAG>>-fips`
clickhouse-operator	`main-<<OPERATOR_TAG>>-fips`
helm/clickhouse-operator-helm	`<<OPERATOR_TAG>>`
helm/onprem-clickhouse-cluster	`<<CR_HELM_TAG>>`

S3 Bucket Naming Restriction

The S3 bucket name cannot contain any periods (.) when using FIPS. The S3 FIPS endpoints require virtual-host-style addressing, which does not work when the bucket name contains periods.

Step 1: Create FIPS Certificates

FIPS compliance requires TLS certificates generated using FIPS-approved algorithms in a FIPS-enabled environment. You need three sets of certificates:

Certificate	Private Key	Public Key	SAN Required
Server	`server.key`	`server.crt`	`*.c-${CLUSTER_NAME}-server-headless.ns-${CLUSTER_NAME}.svc.cluster.local`, `c-${CLUSTER_NAME}-server-any.ns-${CLUSTER_NAME}.svc.cluster.local`
Client	`client.key`	`client.crt`	(none)
Keeper	`keeper.key`	`keeper.crt`	`*.c-${CLUSTER_NAME}-keeper-headless.ns-${CLUSTER_NAME}.svc.cluster.local`

You also need a CA certificate (ca.crt and ca.key), either self-signed or from your organization’s certificate authority. Requirements:

Minimum RSA 3072-bit key size
SHA-256 or higher signature algorithm
Generated in a FIPS-enabled environment
A unique set of certificates (excluding CA) must be created for each ClickHouse cluster

For the full certificate generation procedure using a Red Hat UBI8 container with FIPS crypto policy, see how-to/configure-fips-certificates.md.

Step 2: Install Kubernetes Prerequisites

Install VolumeSnapshot CRDs

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

Install StorageClass (Optional)

Only if you do not have a custom or existing StorageClass:

ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws
CHART_VERSION=<<CR_HELM_TAG>>

helm install clickhouse-prerequisites \
    oci://$ECR_HOST/helm/onprem-clickhouse-cluster \
    --version=$CHART_VERSION \
    -n default \
    --set-json="storageClass.create=true" \
    --set-json="createCluster=false" \
    --set-json="serviceAccount.create=false"

Step 3: Install Operator

Note the FIPS ECR endpoint format: 0000000000.dkr-ecr-fips.us-west-2.on.aws (not the standard .amazonaws.com endpoint).

# update ECR_HOST as needed
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws

# should use the version of the operator's helm chart, not of the operator itself (eg <<OPERATOR_TAG>>, not main-<<OPERATOR_TAG>>)
OPERATOR_VERSION=<<OPERATOR_TAG>>

# image tag -- the FIPS operator build (note the main- prefix and the -fips suffix).
# Pin this: if left unset the chart falls back to its floating ":main" appVersion
# default, which is not a FIPS build.
OPERATOR_IMAGE_TAG=main-<<OPERATOR_TAG>>-fips

# set AZs as determined by VPC subnets
AZ_LIST='["us-west-2a","us-west-2b","us-west-2c"]'

helm install clickhouse-operator \
   oci://$ECR_HOST/helm/clickhouse-operator-helm \
   --version=$OPERATOR_VERSION \
   --create-namespace \
   -n clickhouse-operator-system \
   --set-json="image.repository=\"$ECR_HOST/clickhouse-operator\"" \
   --set-json="image.tag=\"$OPERATOR_IMAGE_TAG\"" \
   --set-json='cilium.enabled=false' \
   --set-json='idleScalerEnabled=false' \
   --set-json='webhooks.enabled=false' \
   --set-json='operator.debug=true' \
   --set-json='operator.metricsScraper.enabled=false' \
   --set-json="operator.availabilityZones=$AZ_LIST" \
   --set-json='operator.featureFlags.backupOnASeparatePod=true' \
   --set-json='operator.featureFlags.serverCMEKEnabled=true'

Step 4: Create Certificate Secrets

Secrets must be installed in the same Kubernetes namespace where the ClickHouse cluster will run. If your cluster name is default-xx-01, the namespace will be ns-default-xx-01. Ensure the namespace exists, then create the secrets. Do not change the secret names or key names.

CERT_DIR=.
CLUSTER_NAME=default-xx-01

# server
kubectl create secret generic -n ns-${CLUSTER_NAME} ${CLUSTER_NAME}-server-cert-secret \
  --from-file=ca.crt="$CERT_DIR/ca/ca.crt" \
  --from-file=server.crt="$CERT_DIR/server/server.crt" \
  --from-file=server.key="$CERT_DIR/server/server.key" \
  --from-file=client.crt="$CERT_DIR/client/client.crt" \
  --from-file=client.key="$CERT_DIR/client/client.key"

# keeper
kubectl create secret generic -n ns-${CLUSTER_NAME} ${CLUSTER_NAME}-keeper-cert-secret \
  --from-file=ca.crt="$CERT_DIR/ca/ca.crt" \
  --from-file=keeper.crt="$CERT_DIR/keeper/keeper.crt" \
  --from-file=keeper.key="$CERT_DIR/keeper/keeper.key"

Step 5: Deploy ClickHouse Cluster

Use the correct FIPS version of the endpoints for ECR and S3. FIPS endpoints can be found at https://aws.amazon.com/compliance/fips/.

Guaranteed QoS (recommended)ClickHouse workloads should run with matching requests and limits for both CPU and memory. The single SERVER_CPU/SERVER_MEMORY/KEEPER_CPU/ KEEPER_MEMORY values below are applied to both resources.requests and resources.limits in the helm invocation that follows, which places the pods in the Guaranteed QoS class. If you need to run with a different QoS class, review Pod QoS: Guaranteed (recommended) first for the trade-offs.

# this will be the `default` user's password
PASSWORD='My super secret p@$$w0rd'
HASHED_PASSWORD=`echo -n "$PASSWORD" | shasum -a 256 | awk '{printf $1}' | base64`

# update values below as needed,
# the ecr host should be the one in your account
# the s3 bucket should be the one created earlier
CLUSTER_NAME=default-xx-01
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws
SA_IAM_ROLE_ARN=arn:aws:iam::0000000000:role/CH-S3-${CLUSTER_NAME}-uw2-00-Role

S3_BUCKET_REGION=us-gov-west-1
S3_BUCKET_URL=https://my-clickhouse-data.us-gov-west-1.amazonaws.com
S3_BUCKET_NAME=my-clickhouse-data

# s3 key prefix can use any UUID value, but must be unique for all clusters storing data in the bucket specified above
S3_KEY_PREFIX=ch-s3-$(uuidgen | tr '[:upper:]' '[:lower:]')

# these should match the clickhouseGroup label value added to the node group
SERVER_NG_GROUP=server-arm64
KEEPER_NG_GROUP=keeper-arm64

# these values should change depending on selected instance sizes
# be sure to take daemon set requirements into account when setting CPU and MEMORY values
SERVER_CPU=64
SERVER_MEMORY=256Gi
KEEPER_CPU=4
KEEPER_MEMORY=16Gi

# Update the `verificationMode` field to control how TLS certificates are verified.
# Defaults to `none`. See Context.h in the ClickHouse source code for more details.
SERVER_VERIFICATION_MODE=none
KEEPER_VERIFICATION_MODE=none

# Currently, usage of `strict` is not recommended and you might encounter issues with the Operator.
# If you require mTLS cert auth, use `relaxed` (Teleport integration requires it)
# SERVER_VERIFICATION_MODE=relaxed
# KEEPER_VERIFICATION_MODE=relaxed

CHART_VERSION=<<CR_HELM_TAG>>

# As a general rule, set cache size to 70-80% of the allocatable SSD disk, accounting for DaemonSets and Cloud Provider reserved disk space.
# For example for type m6id.2xlarge with 1x474 NVMe it should be set to 379Gi (~80%)
CACHE_DISK_SIZE_GI=379Gi

helm install $CLUSTER_NAME \
    oci://$ECR_HOST/helm/onprem-clickhouse-cluster \
    --version=$CHART_VERSION \
    -n ns-$CLUSTER_NAME \
    --create-namespace \
    --set-json="account.hashedPassword=\"$HASHED_PASSWORD\"" \
    --set-json="server.image.repository=\"$ECR_HOST/clickhouse-server\"" \
    --set-json="server.storage.s3.endpoint=\"$S3_BUCKET_URL\"" \
    --set-json="server.storage.s3.region=\"$S3_BUCKET_REGION\"" \
    --set-json="server.storage.s3.bucketName=\"$S3_BUCKET_NAME\"" \
    --set-json="server.storage.s3.keyPrefix=\"$S3_KEY_PREFIX\"" \
    --set-json='server.arm64=true' \
    --set-json="server.image.tag=\"<<SERVER_TAG>>-fips\"" \
    --set-json="server.podPolicy.nodeSelector.clickhouseGroup=\"$SERVER_NG_GROUP\"" \
    --set-json="server.openSSL.enabled=true" \
    --set-json="server.openSSL.required=true" \
    --set-json="server.openSSL.verificationMode=$SERVER_VERIFICATION_MODE" \
    --set-json="keeper.image.repository=\"$ECR_HOST/clickhouse-keeper\"" \
    --set-json='keeper.arm64=true' \
    --set-json="keeper.image.tag=\"<<KEEPER_TAG>>-fips\"" \
    --set-json="keeper.podPolicy.nodeSelector.clickhouseGroup=\"$KEEPER_NG_GROUP\"" \
    --set-json="keeper.openSSL.enabled=true" \
    --set-json="keeper.openSSL.required=true" \
    --set-json="keeper.openSSL.verificationMode=$KEEPER_VERIFICATION_MODE" \
    --set-json="serviceAccount.annotations={\"eks.amazonaws.com/role-arn\":\"$SA_IAM_ROLE_ARN\"}" \
    --set-json='server.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]' \
    --set-json='keeper.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]' \
    --set-json="server.ssdCacheConfiguration.cacheDiskSize=\"$CACHE_DISK_SIZE_GI\"" \
    --set-json="server.podPolicy.resources.limits.cpu=\"$SERVER_CPU\"" \
    --set-json="server.podPolicy.resources.limits.memory=\"$SERVER_MEMORY\"" \
    --set-json="server.podPolicy.resources.requests.cpu=\"$SERVER_CPU\"" \
    --set-json="server.podPolicy.resources.requests.memory=\"$SERVER_MEMORY\"" \
    --set-json="keeper.podPolicy.resources.limits.cpu=\"$KEEPER_CPU\"" \
    --set-json="keeper.podPolicy.resources.limits.memory=\"$KEEPER_MEMORY\"" \
    --set-json="keeper.podPolicy.resources.requests.cpu=\"$KEEPER_CPU\"" \
    --set-json="keeper.podPolicy.resources.requests.memory=\"$KEEPER_MEMORY\""

Key differences from the standard AWS deployment:

Setting	Standard	FIPS
ECR endpoint	`0000000000.dkr.ecr.us-west-2.amazonaws.com`	`0000000000.dkr-ecr-fips.us-west-2.on.aws`
Server image tag	`<<SERVER_TAG>>`	`<<SERVER_TAG>>-fips`
Keeper image tag	`<<KEEPER_TAG>>`	`<<KEEPER_TAG>>-fips`
Operator image tag	`main-<<OPERATOR_TAG>>`	`main-<<OPERATOR_TAG>>-fips`
S3 endpoint	(default)	Explicit FIPS endpoint URL
`server.openSSL.enabled`	not set	`true`
`server.openSSL.required`	not set	`true`
`keeper.openSSL.enabled`	not set	`true`
`keeper.openSSL.required`	not set	`true`
Cache configuration	`bytesPerGiRAM`	`cacheDiskSize` (explicit size)

Step 6: Verify Installation

Port-forward on TLS Port

With FIPS mode, ClickHouse uses TLS on port 9440 instead of the standard port 9000.

kubectl port-forward svc/c-default-xx-01-server-any 9440:9440 -n ns-default-xx-01

Create Client Configuration

Create a client-config.xml file to tell the ClickHouse client where to find the CA certificate. Update the path to your ca.crt file:

<config>
    <openSSL>
        <client>
            <caConfig>/path/to/certs/ca.crt</caConfig>
        </client>
    </openSSL>
</config>

Connect and Run a Query

clickhouse client --host localhost --port 9440 --password $PASSWORD --secure --config=client-config.xml

Run a test query:

clickhouse-cloud :) select 1;

Expected output:

SELECT 1
Query id: 825591bf-a8e5-4995-ac9f-afb864854ba2
   ┌─1─┐
1. │ 1  │
   └───┘
1 row in set. Elapsed: 0.001 sec.

Important: Port 9000 (plaintext native) and port 8123 (plaintext HTTP) will refuse connections when openSSL.required=true. Use port 9440 (native TLS) or port 8443 (HTTPS) instead.

Next Steps

Teleport integration (optional): For secure database access via Teleport Enterprise, see Integrate Teleport with ClickHouse Private. This requires SERVER_VERIFICATION_MODE=relaxed and KEEPER_VERIFICATION_MODE=relaxed.

ClickHouse Cloud

Bring Your Own Cloud

ClickHouse Private

Deploy ClickHouse with FIPS Compliance (Government)

Prerequisites

FIPS Version Tags

S3 Bucket Naming Restriction

Step 1: Create FIPS Certificates

Step 2: Install Kubernetes Prerequisites

Install VolumeSnapshot CRDs

Install StorageClass (Optional)

Step 3: Install Operator

Step 4: Create Certificate Secrets

Step 5: Deploy ClickHouse Cluster

Step 6: Verify Installation

Port-forward on TLS Port

Create Client Configuration

Connect and Run a Query

Next Steps

​Prerequisites

​FIPS Version Tags

​S3 Bucket Naming Restriction

​Step 1: Create FIPS Certificates

​Step 2: Install Kubernetes Prerequisites

​Install VolumeSnapshot CRDs

​Install StorageClass (Optional)

​Step 3: Install Operator

​Step 4: Create Certificate Secrets

​Step 5: Deploy ClickHouse Cluster

​Step 6: Verify Installation

​Port-forward on TLS Port

​Create Client Configuration

​Connect and Run a Query

​Next Steps

Prerequisites

FIPS Version Tags

S3 Bucket Naming Restriction

Step 1: Create FIPS Certificates

Step 2: Install Kubernetes Prerequisites

Install VolumeSnapshot CRDs

Install StorageClass (Optional)

Step 3: Install Operator

Step 4: Create Certificate Secrets

Step 5: Deploy ClickHouse Cluster

Step 6: Verify Installation

Port-forward on TLS Port

Create Client Configuration

Connect and Run a Query

Next Steps