Configure and operate the filesystem cache

For optimal performance we recommend the following configuration for the disk backing the filesystem cache. If you followed the installation tutorial for your Cloud Provider these recommendations are already set. The ClickHouse OSS Usage recommendations provides more in-depth discussions of hardware & OS configuration. The recommendations below are specifically tailored to Private Deployments. For more details on the filesystem cache, see ClickHouse Filesystem cache for SharedMergeTree.

This guide assumes a traditional Linux distribution. On specialized Linux-based operating systems such as Bottlerocket, the required commands and configuration steps will differ, although the general recommendations still hold.

Disks We recommend to always use node-local NVMe disks to back the filesystem cache, otherwise query performance will be impacted by the lower throughput and higher latency of network attached storage. Good query performance cannot be guaranteed if deviating from this recommendation. RAID For multiple NVMe disks backing the Filesystem cache it is recommended to set-up a RAID 0 array. This provides the best read & write performance, as data is striped evenly across all disks. This doesn’t provide any redundancy or fault-tolerance. For the cache this is perfectly safe: the data on the ClickHouse Server lives on Object Storage and the cache is just there for increased performance. Additionally a higher Chunk size is recommended as the average ClickHouse server write is large. A RAID chunk size defines the amount of data sequentially written to a single drive before moving to the next in an array. To create a RAID 0 array you can first find the NVMe devices with nvme list and create an array with:

sudo mdadm --create --verbose /dev/md0 \
  --level=0 \
  --chunk=512 # kilobytes
  --raid-devices=2 \
  /dev/nvme1n1 /dev/nvme2n1 # devices found with nvme list

which will create a new “device” /dev/md0, backed by the RAID array with a chunk size of 512 Kb. Filesystem ClickHouse is most tested with the ext4 filesystem. When mounting the filesystem as ext4, make sure to set to noatime option to prevent linux from automatically updating a file’s “last accessed time” (atime) when it is read. This improves the performance for Read-heavy workloads. Additionally the nobarrier option improves the performance by allowing the NVMe to re-order writes. This is potentially a durability issue as data can be lost on a crash, however, the cache isn’t used to durably store data and the source-of-truth of ClickHouse data lives in Object Storage. Therefore, it is also possible to configure the fsck-pass to 0 (skip), there is no need for filesystem repair. To configure this, change the /etc/fstab file:

UUID=<device UUID>  <mount-path>  ext4  defaults,noatime,nobarrier  0 0

To get the device UUID use the blkid command. Use the mount command to remount the devices.

EmptyDir vs HostPath

For almost all cases we recommend using EmptyDir volumes. HostPath mounts are an option we don’t recommend due to the added operational complexity and security risks. Once the node-local NVMe disks are formatted and available for consumption in a Kubernetes cluster, the next decision to take is how to consume the disks. The ClickHouse Operator supports two modes currently to mount the NVMe disks into the ClickHouse Server pods:

HostPath volumes, creating a direct mapping from the Server pod to the Host (default for AWS)
EmptyDir volumes, backed by Kubernetes’ Local Ephemeral Storage (default for other Cloud Providers)

Our general recommendation is to prefer EmptyDir backed volumes, for the following reasons: HostPath volumes don’t automatically clean up the underlying data on pod reschedules. This normally is a benefit (as ClickHouse Servers rescheduled on the same node will have a warm cache). However it can lead to Full Disk errors in cases where the Cache path changes (e.g. due to a misconfiguration). HostPath volumes have numerous security flaws, increasing the potential attack surface. Several security tools (e.g. Kyverno) have policies to disallow HostPath volumes inside of Kubernetes. Additionally, a lot of Cloud Providers (such as GCP and Azure) by default have features enabling Local Ephemeral Storage automatically. This makes it an easy to use option that reduces the operational complexity.

Useful queries for the Filesystem cache

ClickHouse has several system tables that can provide insights into the Filesystem cache:

system.filesystem_cache_settings allows for introspection of the current configuration/
system.filesystem_cache has information on all currently cached objects.
system.filesystem_cache_log contains a log with all cache interactions.

Additionally the SYSTEM DROP FILESYSTEM CACHE query can be used to drop the entire filesystem cache. This can be useful in cases where the filesystem cache is completely full and ClickHouse is returning CANNOT_WRITE_TO_FILE_DESCRIPTOR errors due to the disk being full.

ClickHouse Cloud

Bring Your Own Cloud

ClickHouse Private

Configure and operate the filesystem cache

EmptyDir vs HostPath

Useful queries for the Filesystem cache

​EmptyDir vs HostPath

​Useful queries for the Filesystem cache

EmptyDir vs HostPath

Useful queries for the Filesystem cache