ClickHouse Filesystem Cache for Shared Merge Tree

In ClickHouse Cloud and ClickHouse Private, SharedMergeTree stores all its primary data in object storage (S3, GCS, Azure Blob). Because object storage has high latency and per-request cost, a local filesystem cache sits in front of it. This cache is an important performance component of the shared-storage architecture. The filesystem cache caches byte ranges of files in S3 (in chunks of a few Mb). As ClickHouse by default compresses data, the cache stores the data compressed. By default the filesystem cache acts as a read-through cache, meaning the cache is lazily populated when queries are executed. By enabling cache_on_write_operations the filesystem cache acts as a write-through cache. Newly written data will automatically be cached in the filesystem cache. Background merges can be further tweaked with the filesystem_cache_skip_download_if_exceeds_per_query_cache_write_limit & filesystem_cache_max_download_size settings. As merges cause both a read and write of data (reading the to-be merged parts, combining them into a single part), merges can quickly pollute the cache with data. By tweaking the settings above, you can control how much data can be loaded into the cache by a single query. By default the filesystem cache uses an LRU (Least Recently Used) eviction strategy. ClickHouse also supports SLRU (segmented LRU), which protects frequently-used entries from being evicted by a single large scan The filesystem cache sits between Object Storage and the query (see the hierarchy below), unless specifically bypassed with per-query settings such as enable_filesystem_cache.

Query
  └─ Mark cache / Uncompressed cache / Query cache   (RAM, per-node)
       └─ Filesystem cache                            (local NVMe SSD)
            └─ Object storage (S3/GCS/Azure)          (source of truth)

The filesystem cache exists for two main purposes:

Hiding Latency from Object Storage. A round trip to Object Storage could take tens of milliseconds, whereas NVMe disk latency is measured in microseconds.
Reduce Costs & prevent Throttling. Object Storage often charges per GET request. Hitting the filesystem cache prevents a GET request, thereby reducing the costs. It also prevents many GET requests from triggering throttling on the Object Storage.

As the cache is checked on each query by default, the performance of the disk matters and we cannot guarentee good performance if ClickHouse isn’t using local NVMe disks as the backing disk for the filesystem cache. Network attached storage has lower throughput and higher latency for read (and write) operations. This will negatively impact the performance of queries. The general recommendation is to size the cache at 70-80% of the total disk size. ClickHouse can use the cache disk for data spilling to disk (e.g. a JOIN / SORT or aggregation query that doesn’t fit in memory), which can cause ClickHouse to use more than the configured cache disk size.