> ## Documentation Index > Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Infrastructure Requirements Reference This document consolidates infrastructure prerequisites, node specifications, networking, storage, and IAM requirements across all supported ClickHouse Private deployment platforms. *** ## Prerequisites by Platform ### AWS (EKS) | Requirement | Details | | -------------------- | -------------------------------------------------------------------------- | | AWS CLI | Installed and configured | | `kubectl` | Installed | | `helm` | v3.x installed | | `skopeo` | Recommended for copying container images (retains all architectures) | | EKS cluster IAM role | Created via AWS console "Create recommended role" with default permissions | | Node IAM roles | Created via AWS console "Create recommended role" with default permissions | | OIDC provider | Created for the EKS cluster | ### GCP (GKE) | Requirement | Details | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------- | | GCP Project | Billing enabled | | `gcloud` CLI | Installed and authenticated | | `kubectl` | Installed | | `helm` | v3.x installed | | `skopeo` | Installed (for copying container images) | | AWS credentials | Read access to ClickHouse ECR (for image copying) | | Bastion host | In the VPC for accessing the private cluster | | Deployer service account | With appropriate permissions (see [IAM / Service Account Requirements](#iam-service-account-requirements)) | | **(FIPS only)** Red Hat UBI8 image | Pulled via GKE for certificate generation | ### Azure (AKS) | Requirement | Details | | ------------------ | --------------------------------------------------------------------------------------------- | | Azure subscription | With permissions to create AKS clusters, ACR, Storage Accounts, VNets, and Managed Identities | | `az` CLI | Installed and authenticated | | `kubectl` | Installed | | `helm` | v3.x installed | | `skopeo` | Installed (for copying container images) | | AWS credentials | Read access to ClickHouse ECR (for image copying) | | OIDC issuer | Enabled on the AKS cluster (`--enable-oidc-issuer`) | | Workload Identity | Enabled on the AKS cluster (`--enable-workload-identity`) | ### Bare Metal | Requirement | Details | | -------------------------- | ------------------------------------------------------------------------------------- | | Kubernetes | Any conformant distribution, v1.33+. RKE2, kubeadm, and k3s are all suitable. | | `kubectl` | Installed | | `helm` | v3.x installed | | `skopeo` | For copying container images between registries | | Private container registry | OCI-compatible and reachable from the cluster, to hold the mirrored images and charts | | Object storage | MinIO AIStor recommended on-prem | | Object storage credentials | Static access key / secret key | *** ## Node Group Specifications ### AWS (EKS) #### Keeper Node Group | Property | Value | | ------------------------- | -------------------------------------------------- | | AMI (x86) | `AL2023_x86_64` | | AMI (arm64) | `AL2023_ARM_64` | | Disk size | 20 GB | | Recommended instance type | `m7g.2xlarge` | | Min nodes | 3 per ClickHouse cluster | | Kubernetes label (x86) | `clickhouseGroup: keeper` | | Kubernetes label (arm64) | `clickhouseGroup: keeper-arm64` | | Taint | `clickhouse.com/do-not-schedule: true, NoSchedule` | | Taint (arm64 only) | `clickhouse.com/arch: arm64, NoSchedule` | | Recommended resources | CPU: 4, Memory: 16Gi | #### Server Node Group | Property | Value | | ------------------------- | ----------------------------------------------------------------- | | AMI (x86) | `AL2023_x86_64` | | AMI (arm64) | `AL2023_ARM_64` | | Disk size | 20 GB | | Recommended instance type | `m7gd.16xlarge` (strongly recommended "d" series with NVMe SSD) | | Min nodes | Equal to desired number of ClickHouse replicas | | Kubernetes label (x86) | `clickhouseGroup: server` | | Kubernetes label (arm64) | `clickhouseGroup: server-arm64` | | Taint | `clickhouse.com/do-not-schedule: true, NoSchedule` | | Taint (arm64 only) | `clickhouse.com/arch: arm64, NoSchedule` | | Recommended resources | CPU: 64, Memory: 256Gi | | Launch template | Custom template required for NVMe SSD RAID setup (see source doc) | #### Operator Node Group (x86) | Property | Value | | --------------------- | --------------------------------------------------------------------------- | | AMI | `AL2023_x86_64` | | Disk size | 20 GB | | Minimum instance size | xlarge | | Instance type | Any x86-compatible type | | Notes | Can be an existing node group if already x86-compatible with minimum xlarge | **All AWS nodes require [IMDS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html) for authentication.** For keeper and server node groups, **create one node group per AZ** if you wish to support cluster autoscaler across AZs. Autoscaler tags (if using cluster-autoscaler): * `k8s.io/cluster-autoscaler/enabled: true` * `k8s.io/cluster-autoscaler/$CLUSTER_NAME: owned` ### GCP (GKE) #### Keeper Node Pool | Property | Value | | --------------------- | ------------------------------------------------ | | Machine type (dev) | `n2-standard-4` | | Machine type (prod) | `n2-standard-8` | | Disk type | `pd-ssd` | | Disk size | 20 GB | | Node label | `clickhouseGroup=keeper` | | Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` | | Autoscaling | min 1, max 3 | | Workload metadata | `GKE_METADATA` (for Workload Identity) | | Recommended resources | CPU: 2, Memory: 8Gi | #### Server Node Pool | Property | Value | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Machine type (dev) | `n2-standard-8` | | Machine type (prod) | `n2-standard-64` | | Disk type | `pd-standard` + ephemeral local SSD | | Disk size | 20 GB | | Local SSD | Attached as ephemeral storage (`--ephemeral-storage-local-ssd count=N`); 375 GiB each, striped into RAID-0 when `count > 1`. Scale `count` so the array holds the cache (`bytesPerGiRAM * server_memory`) plus headroom for images/logs | | Node label | `clickhouseGroup=server` | | Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` | | Autoscaling | min 1, max 10 | | Workload metadata | `GKE_METADATA` (for Workload Identity) | | Recommended resources | CPU: 4, Memory: 16Gi (dev); CPU: 64, Memory: 256Gi (prod) | | SSD cache | `emptyDir` backed by ephemeral local SSD; enabled by `isOnEmptyDir=true` (GCP base config default) | #### Default Node Pool | Property | Value | | ------------ | --------------- | | Machine type | `n2-standard-4` | | Disk type | `pd-standard` | | Disk size | 50 GB | | Nodes | 1 | ### Azure (AKS) #### Keeper Node Pool | Property | Value | | --------------------- | ------------------------------------------------ | | VM size | `Standard_D8ds_v5` | | OS | AzureLinux | | OS disk | 32 GB Premium SSD (Managed) | | Min nodes | 3 per ClickHouse cluster | | Kubernetes label | `clickhouseGroup=keeper` | | Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` | | Recommended resources | CPU: 4, Memory: 16Gi | #### Server Node Pool | Property | Value | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | VM size (dev) | `Standard_D16ds_v5` | | VM size (prod) | `Standard_D48ds_v5` | | OS | AzureLinux | | OS disk type | `Ephemeral` (no managed disk cost; lands on VM cache disk) | | Min nodes | Equal to desired number of ClickHouse replicas | | Kubernetes label | `clickhouseGroup=server` | | Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` | | Recommended resources | CPU: 16, Memory: 64Gi (dev); CPU: 48, Memory: 192Gi (prod) | | SSD cache | `hostPath` pointing at `/mnt` (Azure mounts local SSD there automatically); `isOnEmptyDir=false`, `hostPath=/mnt`; no DaemonSet or RAID required | **Temp storage by instance size (`Standard_D*ds_v5`, Intel Icelake):** | VM Size | vCPUs | RAM | Temp Storage | Cache headroom (\~75%) | | ------------------- | ----- | ------- | ------------ | ---------------------- | | `Standard_D16ds_v5` | 16 | 64 GiB | 600 GiB | \~450 GiB | | `Standard_D32ds_v5` | 32 | 128 GiB | 1200 GiB | \~900 GiB | | `Standard_D48ds_v5` | 48 | 192 GiB | 1800 GiB | \~1350 GiB | #### System / Operator Node Pool | Property | Value | | -------- | ------------------------------------------------------------------------------------- | | VM size | `Standard_D4s_v3` or larger | | Notes | Default system pool created with the cluster; no ClickHouse labels or taints required | For keeper and server node pools, **create one pool per availability zone** if you wish to support cluster autoscaler across zones. ### Bare Metal Assign nodes to distinct roles and label and taint them so the operator schedules ClickHouse pods correctly and nothing else lands on them. At least **three failure domains** are recommended so the operator can spread Keeper across them (topology spread with max skew 1 and a minimum of 3 domains on `topology.kubernetes.io/zone`). Map each domain to a real fault-isolation boundary such as a rack, chassis, or power feed where possible. #### Keeper Nodes | Property | Value | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Count | 3 (one per failure domain) | | Node label | `clickhouseGroup=keeper` | | Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` | | Zone label | a distinct `topology.kubernetes.io/zone` per node, see the [Kubernetes documentation on topology constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) for more context | | Storage | small persistent volume via a StorageClass (not NVMe) | | Recommended resources | CPU: 4, Memory: 16Gi | #### Server Nodes | Property | Value | | --------------------- | --------------------------------------------------------------------------------------- | | Count | 3+ (spread across the failure domains) | | Node label | `clickhouseGroup=server` | | Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` | | Zone label | spread across the failure domains | | Local NVMe SSD | Required for the filesystem cache; mounted at `/nvme/disk` (RAID-0 if multiple devices) | | Recommended resources | Sized to the workload | #### Operator Nodes At least one additional node for the operator and cluster add-ons. No ClickHouse labels or taints are required. *** ## Pod QoS: Guaranteed (recommended) ClickHouse server, keeper, and child compute group pods should run with the Kubernetes [**Guaranteed**](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed) QoS class. In practice, this means that for every container in the pod, `resources.requests` equals `resources.limits` for **both** CPU and memory, and both fields are set. Every deployment example in this documentation follows this pattern. ### Why ClickHouse's memory tracker relies on a stable memory ceiling. When `requests != limits`, the effective ceiling and the OOM-kill boundary diverge and behavior becomes hard to reason about: * If the memory tracker uses the request value, queries are killed while the node still has headroom (over-conservative). * If it uses the cgroup limit, the pod can exceed its request and be OOM-killed by the kernel with no ClickHouse-side warning. Guaranteed QoS also gives the pod the highest priority under node pressure -- see the [Kubernetes Pod QoS docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/) for the eviction and scheduling implications. ### Choosing a different QoS class The operator does not enforce Guaranteed QoS -- you can deliberately run ClickHouse under `Burstable` (or another class) if, for example, you are trading predictability for pack density. Before you do, make sure you understand the OOM trade-offs above and the wider Kubernetes resource model: * [Pod Quality of Service Classes](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/) * [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) For the applied examples, see any of the [deployment tutorials](/docs/cloud/clickhouse-private/tutorials/deploy-aws) or the [compute-compute separation guide](/docs/cloud/clickhouse-private/how-to/configure-compute-compute-separation). *** ## Networking Requirements ### AWS | Component | Specification | | --------------- | ---------------------------------------------- | | VPC CIDR | `10.20.0.0/16` (example) | | IPv6 | Not used | | Tenancy | Default | | Minimum AZs | 3 (for HA) | | Public subnets | 3 (one per AZ, `/20` each) | | Private subnets | 3 (one per AZ, `/18` each) | | NAT gateways | 1 per AZ | | VPC endpoints | S3 gateway | | DNS hostnames | Enabled | | DNS resolution | Enabled | | CNI | IPv4 required (e.g., Amazon VPC CNI with IRSA) | Example subnet layout (us-west-2): | Type | AZ | CIDR | | ------- | ---------- | ---------------- | | Public | us-west-2a | `10.20.192.0/20` | | Public | us-west-2b | `10.20.208.0/20` | | Public | us-west-2c | `10.20.224.0/20` | | Private | us-west-2a | `10.20.0.0/18` | | Private | us-west-2b | `10.20.64.0/18` | | Private | us-west-2c | `10.20.128.0/18` | ### GCP | Component | Specification | | -------------------------- | --------------------------------------------------------------------- | | VPC mode | Custom subnet | | GKE subnet range | `10.1.0.0/24` (example) | | Pod secondary range | `10.244.0.0/14` (must be on `/14` boundary; 3rd octet divisible by 4) | | Service secondary range | `10.252.0.0/20` (must be on `/20` boundary) | | Master IPv4 CIDR | `172.16.0.0/28` | | Private nodes | Enabled (internal IPs only) | | Private endpoint | Enabled (no public API; FedRAMP) | | Master authorized networks | GKE subnet + bastion subnet | | Cloud NAT | Required for outbound internet access from private nodes | | Private Google Access | Enabled on subnet | ### Azure | Component | Specification | | ------------------ | --------------------------- | | VNet address space | `10.20.0.0/16` (example) | | Node subnet | `10.20.0.0/22` | | Network plugin | Azure CNI with overlay mode | | Pod CIDR | `192.168.0.0/16` | | Service CIDR | `10.0.0.0/16` | | DNS service IP | `10.0.0.10` | | Availability zones | 3 (for HA) | ### Bare Metal | Component | Specification | | --------------------------- | ---------------------------------------------------------------------------- | | CNI | Any CNI providing standard IPv4 pod networking (Cilium, Calico, Canal, etc.) | | Failure domains | At least 3, expressed via `topology.kubernetes.io/zone` node labels | | Object storage reachability | S3 endpoint reachable from the ClickHouse pods | | Client access | Via port-forward or a load balancer / ingress | *** ## Storage Requirements ### AWS | Resource | Details | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | S3 bucket | Standard class, encryption enabled, same region as EKS cluster. **Do not enable S3 Lifecycle policies** -- see [Manage data lifecycle](/docs/cloud/clickhouse-private/how-to/manage-data-lifecycle) | | S3 bucket scope | One per ClickHouse cluster, or shared with unique `keyPrefix` per cluster | | EBS CSI driver | Required; install via Helm with IRSA | | StorageClass | `gp3-encrypted` (provided by Helm chart) or custom | | NVMe cache | "d" series instances (e.g., `m7gd.*`); mounted via EC2 launch template RAID script at `/nvme/disk` | | Cache sizing | `bytesPerGiRAM * pod_memory_limit` (e.g., 11.3Gi \* 250Gi = 2825Gi) | | VolumeSnapshot CRDs | Required by operator | ### GCP | Resource | Details | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GCS bucket | Standard class, uniform bucket-level access, same region as GKE cluster. **Do not enable Object Lifecycle Management** -- see [Manage data lifecycle](/docs/cloud/clickhouse-private/how-to/manage-data-lifecycle) | | GCS S3-compatible endpoint | `https://storage.googleapis.com` | | GCS region setting | `auto` | | StorageClass | `premium-rwo` (GKE built-in) or custom via Helm chart with `pd.csi.storage.gke.io` provisioner | | Topology key | `topology.gke.io/zone` (not `topology.kubernetes.io/zone`) | | Local SSD cache | `emptyDir` backed by ephemeral local SSD (`--ephemeral-storage-local-ssd`); GCP base config sets `isOnEmptyDir=true` | | VolumeSnapshot CRDs | Required by operator | ### Azure | Resource | Details | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Storage Account | Standard LRS, StorageV2, public blob access disabled, same region as AKS cluster. **Do not enable Blob Lifecycle Management rules** -- see [Manage data lifecycle](/docs/cloud/clickhouse-private/how-to/manage-data-lifecycle) | | Blob container scope | One per ClickHouse cluster, or shared with unique `keyPrefix` per cluster | | Azure Disk CSI driver | Pre-installed in AKS; provisioner: `disk.csi.azure.com` | | StorageClass | `managed-csi` (server, Standard SSD) and `managed-csi-premium` (keeper, Premium SSD); both pre-installed | | SSD cache | `emptyDir` backed by temp disk; AKS bind-mounts `/mnt/aks/kubelet/` to `/var/lib/kubelet` when `kubelet_disk_type=Temporary`; `isOnEmptyDir=true`; no DaemonSet or RAID required | | Cache sizing | `bytesPerGiRAM * pod_memory_limit` targeting \~75% of temp storage (e.g., 7Gi \* 64Gi = 448Gi for `Standard_D16ds_v5`) | | VolumeSnapshot CRDs | Required by operator | ### Bare Metal | Resource | Details | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Object storage | MinIO AIStor recommended on-prem. **Do not enable object lifecycle / expiration rules** -- see [Manage data lifecycle](/docs/cloud/clickhouse-private/how-to/manage-data-lifecycle) | | Object storage credentials | Static access key / secret key, supplied to server and keeper via `additionalEnvVars` from a Kubernetes Secret | | Server filesystem cache | Local NVMe SSD mounted at `/nvme/disk` (RAID-0 if multiple devices); mounted into Server pods by the operator as a HostPath (default) or EmptyDir volume, not through a StorageClass | | Keeper storage | Persistent volume via a StorageClass (any storage class with a zone-aware label); non-NVMe | | Cache sizing | `cacheDiskSize` set to 70-80% of usable NVMe, or `bytesPerGiRAM * pod_memory_limit` | | S3 region | Set `isOnPremiseInstance=true`; a region is not required for on-prem object stores | | VolumeSnapshot CRDs | Required by operator | *** ## IAM / Service Account Requirements ### AWS #### ECR Pull Role * **Role name:** `ClickHouseAirgapECRPullRole` * **Purpose:** Pull images from ClickHouse private ECR * **Permissions:** `ecr:GetDownloadUrlForLayer`, `ecr:BatchGetImage`, `ecr:BatchCheckLayerAvailability`, `ecr:DescribeImages`, `ecr:ListImages`, `ecr:GetAuthorizationToken` * **Action:** Provide the role ARN to the ClickHouse team #### ClickHouse Server/Keeper S3 Role * **One role per provisioned ClickHouse cluster** * **Naming convention:** `CH-S3-$NAME-$REGION-$ORDINAL-Role` * `$NAME`: cluster name (e.g., `default-xx-01`) * `$REGION`: shortened region (e.g., `uw2` for `us-west-2`) * `$ORDINAL`: reserved, set to `00` * Example: `CH-S3-default-xx-01-uw2-00-Role` * **Permissions:** `s3:*`, `s3:ListBucket` on the bucket resource * **Trust policy:** IRSA via `sts:AssumeRoleWithWebIdentity` with EKS cluster's OIDC provider #### EBS CSI Driver Role * **Example name:** `ClickHouse_EksEbsCsiDriverRole` * **Managed policy:** `arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy` * **Trust policy:** IRSA with EKS cluster's OIDC provider ### GCP #### Deployer Service Account | Role | Purpose | | --------------------------------------- | ------------------------------------------ | | `roles/compute.networkAdmin` | Create/manage VPC, subnets, NAT, routers | | `roles/container.admin` | Create/manage GKE clusters and node pools | | `roles/storage.admin` | Create/manage GCS buckets | | `roles/iam.serviceAccountAdmin` | Create/manage service accounts | | `roles/iam.serviceAccountUser` | Use service accounts for workload identity | | `roles/iam.workloadIdentityUser` | Configure workload identity bindings | | `roles/artifactregistry.admin` | Create/manage Artifact Registry repos | | `roles/resourcemanager.projectIamAdmin` | Manage IAM policies on the project | #### ClickHouse Workload Identity Service Account * **GCP SA:** `clickhouse@$PROJECT.iam.gserviceaccount.com` * **K8s SA (manual):** `clickhouse` in namespace `ns-$CLUSTER_NAME` * **K8s SA (helm-created):** `ch-$CLUSTER_NAME-sa` in namespace `ns-$CLUSTER_NAME` * **K8s SA annotation:** `iam.gke.io/gcp-service-account=` * **IAM binding member:** `serviceAccount:$PROJECT.svc.id.goog[/]` * **Bucket permissions:** `roles/storage.objectAdmin`, `roles/storage.legacyBucketReader` * **Artifact Registry permissions:** `roles/artifactregistry.reader` ### Azure #### Deployer Permissions | Role | Purpose | | --------------------------------------------- | ---------------------------------------------- | | `Contributor` on resource group | Create/manage VNet, AKS, ACR, Storage Accounts | | `User Access Administrator` on resource group | Assign roles to the managed identity | | `AcrPush` on ACR | Push images during setup | #### ClickHouse Managed Identity * **Type:** User-Assigned Managed Identity (UAMI) * **K8s SA (helm-created):** `ch-$CLUSTER_NAME-sa` in namespace `ns-$CLUSTER_NAME` * **K8s SA annotation:** `azure.workload.identity/client-id=` * **Federated credential subject:** `system:serviceaccount:ns-$CLUSTER_NAME:ch-$CLUSTER_NAME-sa` * **Role assignment:** `Storage Blob Data Contributor` on the Storage Account ### Bare Metal * **Object storage credentials:** Static access key / secret key for MinIO AIStor, held in a Kubernetes Secret (`object-storage-credentials`) and injected into server and keeper pods via `additionalEnvVars`. * **Registry pull secret:** `registry-credentials` docker-registry secret in the `clickhouse-operator-system` and cluster namespaces, if your registry requires authentication. * **Service account:** No cloud identity binding required. *** ## Cluster Naming Conventions The naming convention for ClickHouse clusters is consistent across all platforms: ``` $DESCRIPTOR-$LETTERS-$ORDINAL ``` | Component | Description | Example | | ------------- | ------------------------------------- | --------- | | `$DESCRIPTOR` | Descriptive name (letters only) | `default` | | `$LETTERS` | Reserved; any two letters | `xx` | | `$ORDINAL` | Incrementing ordinal starting at `01` | `01` | **Example name:** `default-xx-01` The cluster name must be **unique within the target Kubernetes cluster** and is used in: * Kubernetes namespace: `ns-$CLUSTER_NAME` * Kubernetes resources: prefixed with `c-$CLUSTER_NAME` * IAM role names (AWS): `CH-S3-$CLUSTER_NAME-$REGION-$ORDINAL-Role` * S3/GCS key prefixes: `ch-s3-` (unique per cluster) *** ## Required Kubernetes Components | Component | AWS | GCP | Azure | Bare Metal | | ------------------- | -------------------------------- | ------------------------------------ | --------------------------------- | -------------------------------------------------------------------------------------------------------- | | CNI (IPv4) | Amazon VPC CNI (recommended) | GKE built-in | Azure CNI overlay (built-in) | Any IPv4 CNI | | CSI driver | aws-ebs-csi-driver | GKE built-in (pd.csi.storage.gke.io) | AKS built-in (disk.csi.azure.com) | StorageClass for Keeper (any storage class with a zone-aware label); Server cache uses HostPath/EmptyDir | | DNS | CoreDNS | GKE built-in | AKS built-in (CoreDNS) | Any in-cluster DNS (e.g. CoreDNS) | | VolumeSnapshot CRDs | Required | Required | Required | Required | | Autoscaler | Cluster Autoscaler (recommended) | GKE node pool autoscaling | AKS cluster autoscaler | N/A | | Cert manager | Not required (unless FIPS) | Required for FIPS | Not required (unless FIPS) | Not required (unless FIPS) |