> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Move the filesystem cache to a StorageClass

The filesystem cache on each Server replica can be backed by a
PersistentVolumeClaim provisioned from a StorageClass. Any StorageClass works,
whether it provisions node-local NVMe or network-attached storage, although
node-local consistently gives the best cache performance.

Setting it up on a new instance is a normal install. Moving an instance that is
already running onto a StorageClass, or off it again, takes an extra manual
step, and this page covers that case.

> [!NOTE]
>
> Requires ClickHouse operator version `1.23234.1` or later. The operator image and the `clickhouse-operator-helm` chart share the same version.

## Why the extra step is needed

A StorageClass-backed cache comes from the StatefulSet's
`volumeClaimTemplates`, and that field is
[immutable](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/).
The Operator cannot add it to, or remove it from, a StatefulSet that already
exists. The StatefulSet has to be replaced instead.

> [!WARNING]
>
> Each replica is unavailable for a few minutes during this procedure. Plan for
> reduced capacity while it runs, and make sure clients reconnect and retry.
>
> The Operator applies the new cache mount to the existing StatefulSet as soon as
> you upgrade, before you get a chance to replace it. The mount has nothing
> behind it until the StatefulSet is recreated, so the pod is deleted and cannot
> be recreated:
>
> ```
> FailedCreate  Pod "<instance>-server-<id>-0" is invalid:
>   spec.containers[0].volumeMounts[5].name: Not found: "ch-cache-volume"
> ```
>
> The steps below bring each replica back.

## Move onto a StorageClass

**1. Set the StorageClass in your Helm values**

```yaml theme={null}
server:
  ssdCacheConfiguration:
    cacheDiskSize: "400Gi"
    isOnEmptyDir: false
    storage:
      storageClassName: <your-storage-class>
      size: "400Gi"
```

`accessModes` and `volumeMode` default to `ReadWriteOnce` and `Filesystem`. The
cache is per-replica and never shared, so these suit most drivers.

Size the claim at or above the cache budget you give ClickHouse. ClickHouse
fills to that budget whatever the volume's actual size, so an undersized claim
fails writes with `ENOSPC` instead of evicting from the cache.

Upgrade the release. The replicas will start failing as described above.

**2. Recreate each Server StatefulSet, one replica at a time**

Each Server replica runs in its own StatefulSet with a single pod. List them:

```sh theme={null}
kubectl get statefulsets -n <namespace> \
  -l app.kubernetes.io/name=clickhouse-server,app=<instance-name>-server
```

Delete one and wait for the Operator to rebuild it:

```sh theme={null}
kubectl delete statefulset <sts-name> -n <namespace>
kubectl rollout status statefulset/<sts-name> -n <namespace>
```

The Operator recreates it from scratch, this time with the cache claim template
in place, which takes a couple of minutes. Confirm the replica is healthy
before moving on to the next one.

> [!NOTE]
>
> Deleting the StatefulSet does not delete your data. All data is stored in
> object storage, so this migration to a StorageClass-backed filesystem cache is
> safe to perform.

**3. Verify**

The recreated StatefulSet should list both claim templates:

```sh theme={null}
kubectl get statefulset <sts-name> -n <namespace> \
  -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{"\n"}{end}'
# ch-storage-volume
# ch-cache-volume
```

And each replica should have a bound cache claim:

```sh theme={null}
kubectl get pvc -n <namespace> -l app=<instance-name>-server
```

Once every replica has been recreated the instance returns to `Running`:

```sh theme={null}
kubectl get clickhousecluster -n <namespace> <instance-name>
```

## Move back off a StorageClass

The same constraint applies in reverse. Once a StatefulSet carries a cache
claim template the Operator cannot remove it, so clearing
`storage.storageClassName` and choosing another backing is not complete until
each StatefulSet has been recreated. Follow the same procedure.

The old `ch-cache-volume` claims stay bound afterwards, because the Operator
only collects cache volumes whose replica has gone away. Delete them once every
replica is healthy to release the capacity:

```sh theme={null}
kubectl delete pvc -n <namespace> ch-cache-volume-<sts-name>-0
```
