Integrate Teleport with ClickHouse Private

Set up Teleport Enterprise for secure database access to ClickHouse Private clusters in government/FIPS environments. This guide covers deploying Teleport on EKS, configuring database access, SAML authentication, and certificate rotation.

Prerequisites

The FIPS/Government deployment has been completed with SERVER_VERIFICATION_MODE=relaxed and KEEPER_VERIFICATION_MODE=relaxed (required for Teleport mTLS)
Teleport Enterprise license
The server certificate SAN must include DNS.2 = c-${CLUSTER_NAME}-server-any.ns-${CLUSTER_NAME}.svc.${KUBERNETES_DOMAIN} (see Generate FIPS certificates)

Create a Node Group for Teleport

A dedicated node group for Teleport is optional. If you do not have Bottlerocket ARM FIPS nodes available, create a node group with the following configuration in the EKS console under Compute:

Name: fips-arm64
Node IAM role: create a new recommended role
Kubernetes taints:
- Key: group, Value: fips-arm64, Effect: NoSchedule
AMI type: Bottlerocket ARM FIPS
Instance types: m7g.large
Scaling: Desired 2, Min 2, Max 3

Install Teleport Cluster on EKS

Create Namespace and License Secret

kubectl create namespace teleport
kubectl label namespace teleport 'pod-security.kubernetes.io/enforce=baseline'

# Download the Teleport Enterprise license, then:
kubectl --namespace teleport create secret generic license --from-file=/Downloads/license.pem

Generate FIPS Certificates for Teleport

Create the following script (generate_fips_certs_in_container_for_teleport.sh):

#!/bin/bash
set -e

CLUSTER_FQDN=${CLUSTER_FQDN:-"teleport.example.com"}
NAMESPACE=${NAMESPACE:-"teleport"}
KUBERNETES_DOMAIN=${KUBERNETES_DOMAIN:-"cluster.local"}
COUNTRY=${COUNTRY:-"US"}
STATE=${STATE:-"YourState"}
CITY=${CITY:-"YourCity"}
ORG=${ORG:-"YourOrganization"}
ORG_UNIT=${ORG_UNIT:-"YourOrganizationalUnit"}
CN=${CN:-"YourRootCA"}

# Only run inside a container, prevent accidental execution on host.
if [ ! -f /.dockerenv ]; then
    echo "Not running inside Docker"
    exit 1
fi

dnf -y install openssl crypto-policies-scripts
fips-mode-setup --enable

mkdir -p /certs/{ca_teleport,server_teleport}

# Generate CA certificate
cd /certs/ca_teleport
openssl genrsa -out ca.key 3072

cat > ca.cnf << EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions     = v3_ca
prompt             = no

[ req_distinguished_name ]
C  = ${COUNTRY}
ST = ${STATE}
L  = ${CITY}
O  = ${ORG}
OU = ${ORG_UNIT}
CN = ${CN}

[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
EOF

openssl req -new -x509 -days 3650 -key ca.key -out ca.crt -config ca.cnf

# Generate server certificate
cd /certs/server_teleport
cat > server.cnf << EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions     = v3_req
prompt             = no

[ req_distinguished_name ]
C  = ${COUNTRY}
ST = ${STATE}
L  = ${CITY}
O  = ${ORG}
OU = ${ORG_UNIT}
CN = clickhouse-server

[ v3_req ]
subjectKeyIdentifier = hash
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = ${CLUSTER_FQDN}
DNS.2 = teleport.${NAMESPACE}.svc.${KUBERNETES_DOMAIN}
EOF

openssl genrsa -out server.key 3072
openssl req -new -key server.key -out server.csr -config server.cnf
openssl x509 -req -days 365 -in server.csr -CA /certs/ca_teleport/ca.crt -CAkey /certs/ca_teleport/ca.key \
    -CAcreateserial -out server.crt -extensions v3_req -extfile server.cnf

echo 'Verifying FIPS compliance of generated certificates:'
openssl version
openssl rsa -in /certs/server_teleport/server.key -text -noout | grep 'Private-Key'
openssl x509 -in /certs/server_teleport/server.crt -text -noout | grep 'Signature Algorithm'
echo 'FIPS-compliant certificates have been generated successfully!'

Run it (replace <CLUSTER_FQDN> with the FQDN of the Teleport service):

export CLUSTER_FQDN=<CLUSTER_FQDN>
export NAMESPACE=teleport
export KUBERNETES_DOMAIN=cluster.local
export COUNTRY=US
export STATE=YourState
export CITY=YourCity
export ORG=YourOrganization
export ORG_UNIT=YourOrganizationalUnit
export CN=YourRootCA

docker run -it --rm \
    -e CLUSTER_FQDN=${CLUSTER_FQDN} \
    -e NAMESPACE=${NAMESPACE} \
    -e KUBERNETES_DOMAIN=${KUBERNETES_DOMAIN} \
    -e COUNTRY=${COUNTRY} \
    -e STATE=${STATE} \
    -e CITY=${CITY} \
    -e ORG=${ORG} \
    -e ORG_UNIT=${ORG_UNIT} \
    -e CN=${CN} \
    -v $(pwd):/certs \
    registry.access.redhat.com/ubi8/ubi \
    /certs/generate_fips_certs_in_container_for_teleport.sh

Create Certificate Secrets

kubectl create secret generic teleport-proxy-cert \
  --from-file=tls.crt=server_teleport/server.crt \
  --from-file=tls.key=server_teleport/server.key \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

Create Helm Values

Create eks_teleport_values.yaml. Replace the placeholders:

<CLUSTER_NAME>: The FQDN of the cluster
<VPN_SUBNETS>: VPN private subnets, comma-delimited (e.g., private-subnet-us-east-2a,private-subnet-us-east-2b,private-subnet-us-east-2c)
<BUCKET_NAME>: S3 bucket for Teleport session records
<NODE_SELECTOR>: Node selector value (e.g., fips-arm64)

chartMode: aws
clusterName: "<CLUSTER_NAME>"

enterprise: true
enterpriseImage: public.ecr.aws/gravitational/teleport-ent-fips-distroless
authentication:
  localAuth: false

service:
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    service.beta.kubernetes.io/aws-load-balancer-subnets: "<VPN_SUBNETS>"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

proxyListenerMode: multiplex
aws:
  region: us-east-2
  backendTable: teleport-helm-backend
  auditLogTable: teleport-helm-events
  auditLogMirrorOnStdout: false
  sessionRecordingBucket: <BUCKET_NAME>
  backups: true
  dynamoAutoScaling: false

highAvailability:
  replicaCount: 2

nodeSelector:
  eks.amazonaws.com/nodegroup: <NODE_SELECTOR>

publicAddr: [<CLUSTER_NAME>:443, teleport.teleport.svc.cluster.local:443]

tls:
  existingSecretName: teleport-proxy-cert
  existingCASecretName: teleport-proxy-ca

tolerations:
- key: "group"
  operator: "Equal"
  value: "teleport"
  effect: "NoSchedule"

podSecurityPolicy:
  enabled: false

log:
  level: INFO
  output: stderr

operator:
  enabled: true
  installCRDs: dynamic

serviceAccount:
  create: true
  name: "teleport"

rbac:
  create: true

This example deploys Teleport with FIPS on EKS. For other cloud providers, refer to the Teleport documentation.

Deploy Teleport Cluster

helm repo add teleport https://charts.releases.teleport.dev
helm repo update

helm upgrade --install teleport teleport/teleport-cluster \
  --namespace teleport \
  --values eks_teleport_values.yaml \
  --version 18.2.2

Get the Teleport Service External IP

kubectl get svc teleport -n teleport -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Take the external IP and create a CNAME record in Route53 or your DNS provider. If the load balancer does not have a hostname:

kubectl get svc teleport -n teleport -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Then create an A record instead.

Create a Teleport Provision Token

A provision token allows Teleport agents to join the cluster. Create teleport_provision_token_for_clickhouse.yaml:

apiVersion: resources.teleport.dev/v2
kind: TeleportProvisionToken
metadata:
  name: kube-agent-token
  namespace: teleport
spec:
  roles:
    - Db
  join_method: kubernetes
  kubernetes:
    type: in_cluster
    allow:
      - service_account: teleport-agent:teleport-kube-agent

kubectl apply -f teleport_provision_token_for_clickhouse.yaml

Set Up the Teleport Agent

Create Agent Namespace

kubectl create namespace teleport-agent
kubectl label namespace teleport-agent 'pod-security.kubernetes.io/enforce=baseline'

Create Agent Secrets

Create a secret for the Teleport proxy CA in the agent namespace:

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f-

Create a secret for the database CA (concatenate CAs if using multiple databases with different CAs). Create a file called db_cas.pem, then:

kubectl create secret generic db-ca \
  --from-file=ca.pem=db_cas.pem \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f-

Configure Agent Values

Create teleport_agent_values.yaml. Replace:

<CLICKHOUSE_CLUSTER_NAME>: Name of the provisioned cluster (e.g., default-aa-01)
<ENV>: Static label for this database (e.g., dev)

roles: db
proxyAddr: teleport.teleport.svc.cluster.local:443
enterprise: true
enterpriseImage: public.ecr.aws/gravitational/teleport-ent-fips-distroless
joinParams:
  method: kubernetes
  tokenName: kube-agent-token
tls:
  existingCASecretName: teleport-proxy-ca
databases:
  - name: <CLICKHOUSE_CLUSTER_NAME>-https
    uri: c-<CLICKHOUSE_CLUSTER_NAME>-server-any.<CLICKHOUSE_CLUSTER_NAME>.svc.cluster.local:8443
    protocol: clickhouse-https
    tls:
      ca_cert_file: "/etc/teleport-tls-db/db-ca/ca.pem"
    static_labels:
      env: <ENV>
  - name: <CLICKHOUSE_CLUSTER_NAME>-native
    uri: c-<CLICKHOUSE_CLUSTER_NAME>-server-any.<CLICKHOUSE_CLUSTER_NAME>.svc.cluster.local:9440
    protocol: clickhouse
    tls:
      ca_cert_file: "/etc/teleport-tls-db/db-ca/ca.pem"
    static_labels:
      env: <ENV>
extraVolumes:
  - name: db-ca
    secret:
      secretName: db-ca
extraVolumeMounts:
  - name: db-ca
    mountPath: /etc/teleport-tls-db/db-ca
    readOnly: true

The YAML creates 2 database resources for 1 database because ClickHouse supports 2 protocols (HTTPS on 8443 and native TLS on 9440). Remove whichever you do not need.

Deploy the Agent

VERSION="18.2.2"
helm upgrade --install teleport-kube-agent teleport/teleport-kube-agent \
  --namespace teleport-agent \
  --version ${VERSION} \
  -f teleport_agent_values.yaml

Add Teleport CA to ClickHouse Trusted List

Fetch the Teleport root CA and add it to the ClickHouse server certificate secret:

kubectl exec -it -n teleport deployment/teleport-auth -- tctl auth export --type=db-client > teleport_ca.crt

cat teleport_ca.crt  # Ensure it is not empty

CLUSTER_NAME="<CLUSTER_NAME>"
NAMESPACE="ns-${CLUSTER_NAME}"

kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath="{.data.ca\.crt}" | base64 -d > current_ca.crt
cat current_ca.crt teleport_ca.crt > combined_ca.crt

kubectl create secret generic -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret \
  --from-file=server.crt=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.crt}' | base64 -d) \
  --from-file=server.key=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.key}' | base64 -d) \
  --from-file=client.crt=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.crt}' | base64 -d) \
  --from-file=client.key=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.key}' | base64 -d) \
  --from-file=ca.crt=combined_ca.crt \
  --namespace=${NAMESPACE} \
  --dry-run=client -o yaml | kubectl apply -f -

rm teleport_ca.crt current_ca.crt combined_ca.crt

Restart the ClickHouse server pods to apply the new CA.

Configure SAML Authentication

Create a Database Access Role

Create saml_clickhouse_user_role.yaml. Replace the user trait key with the attribute from your identity provider (e.g., http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress):

apiVersion: resources.teleport.dev/v1
kind: TeleportRoleV8
metadata:
  name: saml-clickhouse-user-dev-role
  namespace: teleport
spec:
  allow:
    db_labels:
      env: dev
    db_users:
      - '{{external["http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"]}}'
    db_names:
    - '*'

kubectl apply -f saml_clickhouse_user_role.yaml

For more information on roles, see the Teleport documentation.

Create a SAML Connector

Create saml_connector.yaml. Replace <DOMAIN> with your Route53 domain and <Entity Descriptor> with the value from your identity provider:

# Example Auth0 integration -- rename and adjust role mappings as needed
kind: TeleportSAMLConnector
apiVersion: resources.teleport.dev/v2
metadata:
  name: samlconnector
  namespace: teleport
spec:
  acs: https://teleport.<DOMAIN>/v1/webapi/saml/acs/samlconnector
  service_provider_issuer: https://teleport.<DOMAIN>
  display: Auth0
  attributes_to_roles:
  - name: http://schemas.xmlsoap.org/claims/Roles
    value: clickhouse-dev
    roles:
    - saml-clickhouse-user-dev-role
  - name: http://schemas.xmlsoap.org/claims/Roles
    value: admin
    roles:
    - access
    - editor
  entity_descriptor: <Entity Descriptor>

For more information on SAML connectors, see the Teleport documentation.

kubectl apply -f saml_connector.yaml

Create Database Users

Exec into a ClickHouse server pod:

kubectl exec -ti <CLICKHOUSE_SERVER_POD> -n ns-<CLUSTER_NAME> -- /bin/bash

Connect to the database:

cat <<EOF > client-config.xml
<config>
    <openSSL>
        <client>
            <caConfig>/etc/clickhouse-server/certs/ca.crt</caConfig>
        </client>
    </openSSL>
</config>
EOF
PASSWORD='My super secret p@$$w0rd'
clickhouse client --host localhost --port 9440 --password "$PASSWORD" --secure --config=client-config.xml

Create a user authenticated via SSL certificate CN (replace <DB_USER> with the username — typically the user’s email address from the identity provider):

CREATE USER '<DB_USER>' IDENTIFIED WITH ssl_certificate CN '<DB_USER>';
GRANT SELECT ON default.* TO '<DB_USER>';

Connect via Teleport

Add the Teleport CA to your system’s trusted CAs, or use the --insecure flag for development.

# Log in to Teleport
tsh login --proxy=teleport.<DOMAIN>:443 --auth=<AUTH_TYPE> --user=<TELEPORT_USER> teleport.<DOMAIN>

# Create a proxy tunnel
tsh proxy db --tunnel <CLICKHOUSE_CLUSTER_NAME>-native --db-user=<DB_USER> --port 59215 &

# Connect
clickhouse client --host localhost --port 59215 --user <DB_USER>

Replace:

<DOMAIN>: Your Route53 domain
<AUTH_TYPE>: Authentication type (e.g., samlconnector)
<TELEPORT_USER>: Teleport username
<DB_USER>: Database username
<CLICKHOUSE_CLUSTER_NAME>: ClickHouse cluster name (e.g., default-aa-01)

Day 2: Certificate Rotation

Teleport certificates expire and must be rotated before expiry.

Rotate Teleport Proxy Certificates

Generate new certificates using the script from Generate FIPS Certificates for Teleport. Ensure CLUSTER_FQDN and other variables match your cluster.
The script must run on a FIPS-enabled host. Docker automatically mounts /proc, so as long as the container runs on a FIPS-enabled host, the generated certificates will be FIPS-compliant.

Append the new CA to the existing bundle in both namespaces:

kubectl get secret teleport-proxy-ca -n teleport -o jsonpath='{.data.ca\.pem}' | base64 -d > current_teleport_ca.pem
cat current_teleport_ca.pem ca_teleport/ca.crt > combined_ca.pem

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=combined_ca.pem \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=combined_ca.pem \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f -

Rolling restart Teleport:

kubectl rollout restart deployment -n teleport
kubectl rollout restart deployment -n teleport-agent

Add the new Teleport CA to ClickHouse trusted list:

kubectl exec -it -n teleport deployment/teleport-auth -- tctl auth export --type=db-client > teleport_ca.crt

CLICKHOUSE_CLUSTER_NAME="<CLICKHOUSE_CLUSTER_NAME>"
NAMESPACE="ns-${CLICKHOUSE_CLUSTER_NAME}"

kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath="{.data.ca\.crt}" | base64 -d > current_ca.crt
cat current_ca.crt teleport_ca.crt > combined_ca.crt

kubectl create secret generic -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret \
  --from-file=server.crt=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.crt}' | base64 -d) \
  --from-file=server.key=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.key}' | base64 -d) \
  --from-file=client.crt=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.crt}' | base64 -d) \
  --from-file=client.key=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.key}' | base64 -d) \
  --from-file=ca.crt=combined_ca.crt \
  --namespace=${NAMESPACE} \
  --dry-run=client -o yaml | kubectl apply -f -

rm -f teleport_ca.crt current_ca.crt combined_ca.crt

kubectl rollout restart statefulset -n ${NAMESPACE}

Replace the proxy certificate:

kubectl create secret generic teleport-proxy-cert \
  --from-file=tls.crt=server_teleport/server.crt \
  --from-file=tls.key=server_teleport/server.key \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deployment -n teleport

Remove the old CA from the bundle:

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deployment -n teleport
kubectl rollout restart deployment -n teleport-agent

Clean up:

rm -f current_teleport_ca.pem combined_ca.pem

Reference

Teleport Documentation

ClickHouse Cloud

Bring Your Own Cloud

ClickHouse Private

Integrate Teleport with ClickHouse Private

Prerequisites

Create a Node Group for Teleport

Install Teleport Cluster on EKS

Create Namespace and License Secret

Generate FIPS Certificates for Teleport

Create Certificate Secrets

Create Helm Values

Deploy Teleport Cluster

Get the Teleport Service External IP

Create a Teleport Provision Token

Set Up the Teleport Agent

Create Agent Namespace

Create Agent Secrets

Configure Agent Values

Deploy the Agent

Add Teleport CA to ClickHouse Trusted List

Configure SAML Authentication

Create a Database Access Role

Create a SAML Connector

Create Database Users

Connect via Teleport

Day 2: Certificate Rotation

Rotate Teleport Proxy Certificates

Reference

​Prerequisites

​Create a Node Group for Teleport

​Install Teleport Cluster on EKS

​Create Namespace and License Secret

​Generate FIPS Certificates for Teleport

​Create Certificate Secrets

​Create Helm Values

​Deploy Teleport Cluster

​Get the Teleport Service External IP

​Create a Teleport Provision Token

​Set Up the Teleport Agent

​Create Agent Namespace

​Create Agent Secrets

​Configure Agent Values

​Deploy the Agent

​Add Teleport CA to ClickHouse Trusted List

​Configure SAML Authentication

​Create a Database Access Role

​Create a SAML Connector

​Create Database Users

​Connect via Teleport

​Day 2: Certificate Rotation

​Rotate Teleport Proxy Certificates

​Reference

Prerequisites

Create a Node Group for Teleport

Install Teleport Cluster on EKS

Create Namespace and License Secret

Generate FIPS Certificates for Teleport

Create Certificate Secrets

Create Helm Values

Deploy Teleport Cluster

Get the Teleport Service External IP

Create a Teleport Provision Token

Set Up the Teleport Agent

Create Agent Namespace

Create Agent Secrets

Configure Agent Values

Deploy the Agent

Add Teleport CA to ClickHouse Trusted List

Configure SAML Authentication

Create a Database Access Role

Create a SAML Connector

Create Database Users

Connect via Teleport

Day 2: Certificate Rotation

Rotate Teleport Proxy Certificates

Reference