> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Integrate Teleport with ClickHouse Private

Set up Teleport Enterprise for secure database access to ClickHouse Private clusters in government/FIPS environments. This guide covers deploying Teleport on EKS, configuring database access, SAML authentication, and certificate rotation.

## Prerequisites

* The [FIPS/Government deployment](/docs/cloud/clickhouse-private/tutorials/deploy-government) has been completed with `SERVER_VERIFICATION_MODE=relaxed` and `KEEPER_VERIFICATION_MODE=relaxed` (required for Teleport mTLS)
* Teleport Enterprise license
* The server certificate SAN must include `DNS.2 = c-${CLUSTER_NAME}-server-any.ns-${CLUSTER_NAME}.svc.${KUBERNETES_DOMAIN}` (see [Generate FIPS certificates](/docs/cloud/clickhouse-private/how-to/configure-fips-certificates))

***

## Create a Node Group for Teleport

A dedicated node group for Teleport is optional. If you do not have Bottlerocket ARM FIPS nodes available, create a node group with the following configuration in the EKS console under Compute:

* **Name**: fips-arm64
* **Node IAM role**: create a new recommended role
* **Kubernetes taints**:
  * **Key**: `group`, **Value**: `fips-arm64`, **Effect**: `NoSchedule`
* **AMI type**: Bottlerocket ARM FIPS
* **Instance types**: m7g.large
* **Scaling**: Desired 2, Min 2, Max 3

***

## Install Teleport Cluster on EKS

### Create Namespace and License Secret

```bash theme={null}
kubectl create namespace teleport
kubectl label namespace teleport 'pod-security.kubernetes.io/enforce=baseline'

# Download the Teleport Enterprise license, then:
kubectl --namespace teleport create secret generic license --from-file=/Downloads/license.pem
```

### Generate FIPS Certificates for Teleport

Create the following script (`generate_fips_certs_in_container_for_teleport.sh`):

```bash theme={null}
#!/bin/bash
set -e

CLUSTER_FQDN=${CLUSTER_FQDN:-"teleport.example.com"}
NAMESPACE=${NAMESPACE:-"teleport"}
KUBERNETES_DOMAIN=${KUBERNETES_DOMAIN:-"cluster.local"}
COUNTRY=${COUNTRY:-"US"}
STATE=${STATE:-"YourState"}
CITY=${CITY:-"YourCity"}
ORG=${ORG:-"YourOrganization"}
ORG_UNIT=${ORG_UNIT:-"YourOrganizationalUnit"}
CN=${CN:-"YourRootCA"}

# Only run inside a container, prevent accidental execution on host.
if [ ! -f /.dockerenv ]; then
    echo "Not running inside Docker"
    exit 1
fi

dnf -y install openssl crypto-policies-scripts
fips-mode-setup --enable

mkdir -p /certs/{ca_teleport,server_teleport}

# Generate CA certificate
cd /certs/ca_teleport
openssl genrsa -out ca.key 3072

cat > ca.cnf << EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions     = v3_ca
prompt             = no

[ req_distinguished_name ]
C  = ${COUNTRY}
ST = ${STATE}
L  = ${CITY}
O  = ${ORG}
OU = ${ORG_UNIT}
CN = ${CN}

[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
EOF

openssl req -new -x509 -days 3650 -key ca.key -out ca.crt -config ca.cnf

# Generate server certificate
cd /certs/server_teleport
cat > server.cnf << EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions     = v3_req
prompt             = no

[ req_distinguished_name ]
C  = ${COUNTRY}
ST = ${STATE}
L  = ${CITY}
O  = ${ORG}
OU = ${ORG_UNIT}
CN = clickhouse-server

[ v3_req ]
subjectKeyIdentifier = hash
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = ${CLUSTER_FQDN}
DNS.2 = teleport.${NAMESPACE}.svc.${KUBERNETES_DOMAIN}
EOF

openssl genrsa -out server.key 3072
openssl req -new -key server.key -out server.csr -config server.cnf
openssl x509 -req -days 365 -in server.csr -CA /certs/ca_teleport/ca.crt -CAkey /certs/ca_teleport/ca.key \
    -CAcreateserial -out server.crt -extensions v3_req -extfile server.cnf

echo 'Verifying FIPS compliance of generated certificates:'
openssl version
openssl rsa -in /certs/server_teleport/server.key -text -noout | grep 'Private-Key'
openssl x509 -in /certs/server_teleport/server.crt -text -noout | grep 'Signature Algorithm'
echo 'FIPS-compliant certificates have been generated successfully!'
```

Run it (replace `<CLUSTER_FQDN>` with the FQDN of the Teleport service):

```bash theme={null}
export CLUSTER_FQDN=<CLUSTER_FQDN>
export NAMESPACE=teleport
export KUBERNETES_DOMAIN=cluster.local
export COUNTRY=US
export STATE=YourState
export CITY=YourCity
export ORG=YourOrganization
export ORG_UNIT=YourOrganizationalUnit
export CN=YourRootCA

docker run -it --rm \
    -e CLUSTER_FQDN=${CLUSTER_FQDN} \
    -e NAMESPACE=${NAMESPACE} \
    -e KUBERNETES_DOMAIN=${KUBERNETES_DOMAIN} \
    -e COUNTRY=${COUNTRY} \
    -e STATE=${STATE} \
    -e CITY=${CITY} \
    -e ORG=${ORG} \
    -e ORG_UNIT=${ORG_UNIT} \
    -e CN=${CN} \
    -v $(pwd):/certs \
    registry.access.redhat.com/ubi8/ubi \
    /certs/generate_fips_certs_in_container_for_teleport.sh
```

### Create Certificate Secrets

```bash theme={null}
kubectl create secret generic teleport-proxy-cert \
  --from-file=tls.crt=server_teleport/server.crt \
  --from-file=tls.key=server_teleport/server.key \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -
```

### Create Helm Values

Create `eks_teleport_values.yaml`. Replace the placeholders:

* `<CLUSTER_NAME>`: The FQDN of the cluster
* `<VPN_SUBNETS>`: VPN private subnets, comma-delimited (e.g., `private-subnet-us-east-2a,private-subnet-us-east-2b,private-subnet-us-east-2c`)
* `<BUCKET_NAME>`: S3 bucket for Teleport session records
* `<NODE_SELECTOR>`: Node selector value (e.g., `fips-arm64`)

```yaml theme={null}
chartMode: aws
clusterName: "<CLUSTER_NAME>"

enterprise: true
enterpriseImage: public.ecr.aws/gravitational/teleport-ent-fips-distroless
authentication:
  localAuth: false

service:
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    service.beta.kubernetes.io/aws-load-balancer-subnets: "<VPN_SUBNETS>"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

proxyListenerMode: multiplex
aws:
  region: us-east-2
  backendTable: teleport-helm-backend
  auditLogTable: teleport-helm-events
  auditLogMirrorOnStdout: false
  sessionRecordingBucket: <BUCKET_NAME>
  backups: true
  dynamoAutoScaling: false

highAvailability:
  replicaCount: 2

nodeSelector:
  eks.amazonaws.com/nodegroup: <NODE_SELECTOR>

publicAddr: [<CLUSTER_NAME>:443, teleport.teleport.svc.cluster.local:443]

tls:
  existingSecretName: teleport-proxy-cert
  existingCASecretName: teleport-proxy-ca

tolerations:
- key: "group"
  operator: "Equal"
  value: "teleport"
  effect: "NoSchedule"

podSecurityPolicy:
  enabled: false

log:
  level: INFO
  output: stderr

operator:
  enabled: true
  installCRDs: dynamic

serviceAccount:
  create: true
  name: "teleport"

rbac:
  create: true
```

> This example deploys Teleport with FIPS on EKS. For other cloud providers, refer to the [Teleport documentation](https://goteleport.com/docs/zero-trust-access/deploy-a-cluster/helm-deployments/).

### Deploy Teleport Cluster

```bash theme={null}
helm repo add teleport https://charts.releases.teleport.dev
helm repo update

helm upgrade --install teleport teleport/teleport-cluster \
  --namespace teleport \
  --values eks_teleport_values.yaml \
  --version 18.2.2
```

### Get the Teleport Service External IP

```bash theme={null}
kubectl get svc teleport -n teleport -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
```

Take the external IP and create a CNAME record in Route53 or your DNS provider. If the load balancer does not have a hostname:

```bash theme={null}
kubectl get svc teleport -n teleport -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```

Then create an A record instead.

***

## Create a Teleport Provision Token

A provision token allows Teleport agents to join the cluster. Create `teleport_provision_token_for_clickhouse.yaml`:

```yaml theme={null}
apiVersion: resources.teleport.dev/v2
kind: TeleportProvisionToken
metadata:
  name: kube-agent-token
  namespace: teleport
spec:
  roles:
    - Db
  join_method: kubernetes
  kubernetes:
    type: in_cluster
    allow:
      - service_account: teleport-agent:teleport-kube-agent
```

```bash theme={null}
kubectl apply -f teleport_provision_token_for_clickhouse.yaml
```

***

## Set Up the Teleport Agent

### Create Agent Namespace

```bash theme={null}
kubectl create namespace teleport-agent
kubectl label namespace teleport-agent 'pod-security.kubernetes.io/enforce=baseline'
```

### Create Agent Secrets

Create a secret for the Teleport proxy CA in the agent namespace:

```bash theme={null}
kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f-
```

Create a secret for the database CA (concatenate CAs if using multiple databases with different CAs). Create a file called `db_cas.pem`, then:

```bash theme={null}
kubectl create secret generic db-ca \
  --from-file=ca.pem=db_cas.pem \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f-
```

### Configure Agent Values

Create `teleport_agent_values.yaml`. Replace:

* `<CLICKHOUSE_CLUSTER_NAME>`: Name of the provisioned cluster (e.g., `default-aa-01`)
* `<ENV>`: Static label for this database (e.g., `dev`)

```yaml theme={null}
roles: db
proxyAddr: teleport.teleport.svc.cluster.local:443
enterprise: true
enterpriseImage: public.ecr.aws/gravitational/teleport-ent-fips-distroless
joinParams:
  method: kubernetes
  tokenName: kube-agent-token
tls:
  existingCASecretName: teleport-proxy-ca
databases:
  - name: <CLICKHOUSE_CLUSTER_NAME>-https
    uri: c-<CLICKHOUSE_CLUSTER_NAME>-server-any.<CLICKHOUSE_CLUSTER_NAME>.svc.cluster.local:8443
    protocol: clickhouse-https
    tls:
      ca_cert_file: "/etc/teleport-tls-db/db-ca/ca.pem"
    static_labels:
      env: <ENV>
  - name: <CLICKHOUSE_CLUSTER_NAME>-native
    uri: c-<CLICKHOUSE_CLUSTER_NAME>-server-any.<CLICKHOUSE_CLUSTER_NAME>.svc.cluster.local:9440
    protocol: clickhouse
    tls:
      ca_cert_file: "/etc/teleport-tls-db/db-ca/ca.pem"
    static_labels:
      env: <ENV>
extraVolumes:
  - name: db-ca
    secret:
      secretName: db-ca
extraVolumeMounts:
  - name: db-ca
    mountPath: /etc/teleport-tls-db/db-ca
    readOnly: true
```

> The YAML creates 2 database resources for 1 database because ClickHouse supports 2 protocols (HTTPS on 8443 and native TLS on 9440). Remove whichever you do not need.

### Deploy the Agent

```bash theme={null}
VERSION="18.2.2"
helm upgrade --install teleport-kube-agent teleport/teleport-kube-agent \
  --namespace teleport-agent \
  --version ${VERSION} \
  -f teleport_agent_values.yaml
```

***

## Add Teleport CA to ClickHouse Trusted List

Fetch the Teleport root CA and add it to the ClickHouse server certificate secret:

```bash theme={null}
kubectl exec -it -n teleport deployment/teleport-auth -- tctl auth export --type=db-client > teleport_ca.crt

cat teleport_ca.crt  # Ensure it is not empty

CLUSTER_NAME="<CLUSTER_NAME>"
NAMESPACE="ns-${CLUSTER_NAME}"

kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath="{.data.ca\.crt}" | base64 -d > current_ca.crt
cat current_ca.crt teleport_ca.crt > combined_ca.crt

kubectl create secret generic -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret \
  --from-file=server.crt=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.crt}' | base64 -d) \
  --from-file=server.key=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.key}' | base64 -d) \
  --from-file=client.crt=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.crt}' | base64 -d) \
  --from-file=client.key=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.key}' | base64 -d) \
  --from-file=ca.crt=combined_ca.crt \
  --namespace=${NAMESPACE} \
  --dry-run=client -o yaml | kubectl apply -f -

rm teleport_ca.crt current_ca.crt combined_ca.crt
```

Restart the ClickHouse server pods to apply the new CA.

***

## Configure SAML Authentication

### Create a Database Access Role

Create `saml_clickhouse_user_role.yaml`. Replace the user trait key with the attribute from your identity provider (e.g., `http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress`):

```yaml theme={null}
apiVersion: resources.teleport.dev/v1
kind: TeleportRoleV8
metadata:
  name: saml-clickhouse-user-dev-role
  namespace: teleport
spec:
  allow:
    db_labels:
      env: dev
    db_users:
      - '{{external["http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"]}}'
    db_names:
    - '*'
```

```bash theme={null}
kubectl apply -f saml_clickhouse_user_role.yaml
```

For more information on roles, see the [Teleport documentation](https://goteleport.com/docs/reference/infrastructure-as-code/operator-resources/resources-teleport-dev-rolesv8/).

### Create a SAML Connector

Create `saml_connector.yaml`. Replace `<DOMAIN>` with your Route53 domain and `<Entity Descriptor>` with the value from your identity provider:

```yaml theme={null}
# Example Auth0 integration -- rename and adjust role mappings as needed
kind: TeleportSAMLConnector
apiVersion: resources.teleport.dev/v2
metadata:
  name: samlconnector
  namespace: teleport
spec:
  acs: https://teleport.<DOMAIN>/v1/webapi/saml/acs/samlconnector
  service_provider_issuer: https://teleport.<DOMAIN>
  display: Auth0
  attributes_to_roles:
  - name: http://schemas.xmlsoap.org/claims/Roles
    value: clickhouse-dev
    roles:
    - saml-clickhouse-user-dev-role
  - name: http://schemas.xmlsoap.org/claims/Roles
    value: admin
    roles:
    - access
    - editor
  entity_descriptor: <Entity Descriptor>
```

For more information on SAML connectors, see the [Teleport documentation](https://goteleport.com/docs/reference/infrastructure-as-code/operator-resources/resources-teleport-dev-samlconnectors/).

```bash theme={null}
kubectl apply -f saml_connector.yaml
```

***

## Create Database Users

Exec into a ClickHouse server pod:

```bash theme={null}
kubectl exec -ti <CLICKHOUSE_SERVER_POD> -n ns-<CLUSTER_NAME> -- /bin/bash
```

Connect to the database:

```bash theme={null}
cat <<EOF > client-config.xml
<config>
    <openSSL>
        <client>
            <caConfig>/etc/clickhouse-server/certs/ca.crt</caConfig>
        </client>
    </openSSL>
</config>
EOF
PASSWORD='My super secret p@$$w0rd'
clickhouse client --host localhost --port 9440 --password "$PASSWORD" --secure --config=client-config.xml
```

Create a user authenticated via SSL certificate CN (replace `<DB_USER>` with the username -- typically the user's email address from the identity provider):

```sql theme={null}
CREATE USER '<DB_USER>' IDENTIFIED WITH ssl_certificate CN '<DB_USER>';
GRANT SELECT ON default.* TO '<DB_USER>';
```

***

## Connect via Teleport

Add the Teleport CA to your system's trusted CAs, or use the `--insecure` flag for development.

```bash theme={null}
# Log in to Teleport
tsh login --proxy=teleport.<DOMAIN>:443 --auth=<AUTH_TYPE> --user=<TELEPORT_USER> teleport.<DOMAIN>

# Create a proxy tunnel
tsh proxy db --tunnel <CLICKHOUSE_CLUSTER_NAME>-native --db-user=<DB_USER> --port 59215 &

# Connect
clickhouse client --host localhost --port 59215 --user <DB_USER>
```

Replace:

* `<DOMAIN>`: Your Route53 domain
* `<AUTH_TYPE>`: Authentication type (e.g., `samlconnector`)
* `<TELEPORT_USER>`: Teleport username
* `<DB_USER>`: Database username
* `<CLICKHOUSE_CLUSTER_NAME>`: ClickHouse cluster name (e.g., `default-aa-01`)

***

## Day 2: Certificate Rotation

Teleport certificates expire and must be rotated before expiry.

### Rotate Teleport Proxy Certificates

1. **Generate new certificates** using the script from [Generate FIPS Certificates for Teleport](#generate-fips-certificates-for-teleport). Ensure `CLUSTER_FQDN` and other variables match your cluster.

   > The script must run on a FIPS-enabled host. Docker automatically mounts `/proc`, so as long as the container runs on a FIPS-enabled host, the generated certificates will be FIPS-compliant.

2. **Append the new CA to the existing bundle** in both namespaces:

   ```bash theme={null}
   kubectl get secret teleport-proxy-ca -n teleport -o jsonpath='{.data.ca\.pem}' | base64 -d > current_teleport_ca.pem
   cat current_teleport_ca.pem ca_teleport/ca.crt > combined_ca.pem

   kubectl create secret generic teleport-proxy-ca \
     --from-file=ca.pem=combined_ca.pem \
     --namespace=teleport \
     --dry-run=client -o yaml | kubectl apply -f -

   kubectl create secret generic teleport-proxy-ca \
     --from-file=ca.pem=combined_ca.pem \
     --namespace=teleport-agent \
     --dry-run=client -o yaml | kubectl apply -f -
   ```

3. **Rolling restart Teleport**:

   ```bash theme={null}
   kubectl rollout restart deployment -n teleport
   kubectl rollout restart deployment -n teleport-agent
   ```

4. **Add the new Teleport CA to ClickHouse trusted list**:

   ```bash theme={null}
   kubectl exec -it -n teleport deployment/teleport-auth -- tctl auth export --type=db-client > teleport_ca.crt

   CLICKHOUSE_CLUSTER_NAME="<CLICKHOUSE_CLUSTER_NAME>"
   NAMESPACE="ns-${CLICKHOUSE_CLUSTER_NAME}"

   kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath="{.data.ca\.crt}" | base64 -d > current_ca.crt
   cat current_ca.crt teleport_ca.crt > combined_ca.crt

   kubectl create secret generic -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret \
     --from-file=server.crt=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.crt}' | base64 -d) \
     --from-file=server.key=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.key}' | base64 -d) \
     --from-file=client.crt=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.crt}' | base64 -d) \
     --from-file=client.key=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.key}' | base64 -d) \
     --from-file=ca.crt=combined_ca.crt \
     --namespace=${NAMESPACE} \
     --dry-run=client -o yaml | kubectl apply -f -

   rm -f teleport_ca.crt current_ca.crt combined_ca.crt

   kubectl rollout restart statefulset -n ${NAMESPACE}
   ```

5. **Replace the proxy certificate**:

   ```bash theme={null}
   kubectl create secret generic teleport-proxy-cert \
     --from-file=tls.crt=server_teleport/server.crt \
     --from-file=tls.key=server_teleport/server.key \
     --namespace=teleport \
     --dry-run=client -o yaml | kubectl apply -f -

   kubectl rollout restart deployment -n teleport
   ```

6. **Remove the old CA** from the bundle:

   ```bash theme={null}
   kubectl create secret generic teleport-proxy-ca \
     --from-file=ca.pem=ca_teleport/ca.crt \
     --namespace=teleport \
     --dry-run=client -o yaml | kubectl apply -f -

   kubectl create secret generic teleport-proxy-ca \
     --from-file=ca.pem=ca_teleport/ca.crt \
     --namespace=teleport-agent \
     --dry-run=client -o yaml | kubectl apply -f -

   kubectl rollout restart deployment -n teleport
   kubectl rollout restart deployment -n teleport-agent
   ```

7. **Clean up**:

   ```bash theme={null}
   rm -f current_teleport_ca.pem combined_ca.pem
   ```

***

## Reference

* [Teleport Documentation](https://goteleport.com/docs/)