Querying in ClickHouse CloudThe data in this system table is held locally on each node in ClickHouse Cloud. Obtaining a complete view of all data, therefore, requires the
clusterAllReplicas function. See here for further details.Description
Contains metrics that are calculated periodically in the background. For example, the amount of RAM in use. A metric is either scalar (a single number in thevalue column) or key-value: broken down per entity, such as per CPU core, block device, network interface, or disk. Key-value metrics are represented as a single row with a Map in the key_values column (for example, CPUFrequencyMHz maps every CPU core number to its frequency), and their value column is NaN.
Columns
metric(String) — Metric name.value(Float64) — Metric value. For key-value metrics (broken down per CPU core, block device, disk, …) it is NaN, and the values are in thekey_valuescolumn.key_values(Map(LowCardinality(String), Float64)) — Values of a key-value metric, e.g. keyed by the CPU core number or the block device name. Empty for scalar metrics.description(String - Metric description)
Example
Metric descriptions
The descriptions below are generated from the C++ source byutils/generate-async-metrics-docs. The single source of truth is the string literal next to each metric registration in src/Common/AsynchronousMetrics.cpp, src/Interpreters/ServerAsynchronousMetrics.cpp, and src/Coordination/KeeperAsynchronousMetrics.cpp. Metric names that include a variable part (currently only the HTTP connection pool group) are shown with a *name* placeholder; the running server reports them with the concrete name substituted in.
AsynchronousHeavyMetricsCalculationTimeSpent
Time in seconds spent for calculation of asynchronous heavy (tables related) metrics (this is the overhead of asynchronous metrics).AsynchronousHeavyMetricsUpdateInterval
Heavy (tables related) metrics update intervalAsynchronousMetricsCalculationTimeSpent
Time in seconds spent for calculation of asynchronous metrics (this is the overhead of asynchronous metrics).AsynchronousMetricsUpdateInterval
Metrics update intervalAsyncLoggingQueueSize
Number of async messages queued pending for logging, keyed by the logging channel name.BlockActiveTime
Time in seconds every block device had the IO requests queued. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockActiveTimePerOp
Similar to theBlockActiveTime metric, but the value is divided to the number of IO operations to count the per-operation time. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: /sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockDiscardBytes
Number of discarded bytes on every block device. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockDiscardMerges
Number of discard operations requested from every block device and merged together by the OS IO scheduler. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockDiscardOps
Number of discard operations requested from every block device. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockDiscardTime
Time in seconds spend in discard operations requested from every block device, summed across all the operations. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockInFlightOps
This value counts the number of I/O requests that have been issued to the device driver but have not yet completed, for every block device. It does not include IO requests that are in the queue but not yet issued to the device driver. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockQueueTime
This value counts the number of milliseconds that IO requests have waited on every block device. If there are multiple IO requests waiting, this value will increase as the product of the number of milliseconds times the number of requests waiting. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockQueueTimePerOp
Similar to theBlockQueueTime metric, but the value is divided to the number of IO operations to count the per-operation time. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: /sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockReadBytes
Number of bytes read from every block device. It can be lower than the number of bytes read from the filesystem due to the usage of the OS page cache, that saves IO. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockReadMerges
Number of read operations requested from every block device and merged together by the OS IO scheduler. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockReadOps
Number of read operations requested from every block device. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockReadTime
Time in seconds spend in read operations requested from every block device, summed across all the operations. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockWriteBytes
Number of bytes written to every block device. It can be lower than the number of bytes written to the filesystem due to the usage of the OS page cache, that saves IO. A write to the block device may happen later than the corresponding write to the filesystem due to write-through caching. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockWriteMerges
Number of write operations requested from every block device and merged together by the OS IO scheduler. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockWriteOps
Number of write operations requested from every block device. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
BlockWriteTime
Time in seconds spend in write operations requested from every block device, summed across all the operations. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source:/sys/block. See https://www.kernel.org/doc/Documentation/block/stat.txt
CGroupMaxCPU
The maximum number of CPU cores according to CGroups.CGroupMemoryTotal
The total amount of memory in cgroup, in bytes. If stated zero, the limit is the same as OSMemoryTotal.CGroupMemoryUsed
The amount of memory used in cgroup, in bytes. On cgroup v2 this is anon + sock + non-reclaimable kernel memory; on cgroup v1 this is RSS. In both cases the kernel OS page cache (file-backed cache) is excluded.CGroupMemoryUsedWithoutPageCache
The amount of memory used in cgroup, in bytes, excluding the ClickHouse userspace page cache. This is CGroupMemoryUsed minus the userspace page cache size. When userspace page cache is disabled, this value equals CGroupMemoryUsed.CGroupSystemTime
The ratio of time the CPU core was running OS kernel (system) code.CGroupSystemTimeNormalized
The value is similar toCGroupSystemTime but divided by the number of available CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
CGroupUserTime
The ratio of time the CPU core was running userspace code. This includes also the time when the CPU was under-utilized due to the reasons internal to the CPU (memory loads, pipeline stalls, branch mispredictions, running another SMT core).CGroupUserTimeNormalized
The value is similar toCGroupUserTime but divided by the number of available CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
CompiledExpressionCacheBytesMax
Configured maximum bytes of the compiled-expression cache. Available headroom = this minusCompiledExpressionCacheBytes (CurrentMetric).
CompiledExpressionCacheCountMax
Configured maximum entries of the compiled-expression cache. Available headroom = this minusCompiledExpressionCacheCount (CurrentMetric).
CPUFrequencyMHz
The current frequency of every CPU core, in MHz, keyed by the CPU core number. Most of the modern CPUs adjust the frequency dynamically for power saving and Turbo Boosting.DeadBlobsQueueEstimate
Estimated number of blobs enqueued for removal from the disk object storage (the blob manager dead queue), keyed by the disk name. Disks without blob replication report 0.DictionaryMaxUpdateDelay
The maximum delay (in seconds) of dictionary updateDictionaryTotalFailedUpdates
Number of errors since last successful loading in all dictionaries.DiskAvailable
Available bytes on every disk (virtual filesystem), keyed by the disk name. Remote filesystems may not provide this information and can show a large value like 16 EiB.DiskGetObjectThrottlerAvailable
Number of GetObject requests that can be currently issued without hitting throttling limit on every disk (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information.DiskGetObjectThrottlerRPS
GetObject Request throttling limit on every disk in requests per second (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information.DiskPutObjectThrottlerAvailable
Number of PutObject requests that can be currently issued without hitting throttling limit on every disk (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information.DiskPutObjectThrottlerRPS
PutObject Request throttling limit on every disk in requests per second (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information.DiskTotal
The total size in bytes of every disk (virtual filesystem), keyed by the disk name. Remote filesystems may not provide this information and can show a large value like 16 EiB.DiskUnreserved
Available bytes on every disk (virtual filesystem) without the reservations for merges, fetches, and moves, keyed by the disk name. Remote filesystems may not provide this information and can show a large value like 16 EiB.DiskUsed
Used bytes on every disk (virtual filesystem), keyed by the disk name. Remote filesystems do not always provide this information.EDACCorrectable
The number of correctable ECC memory errors, keyed by the memory controller number. A high number of this value indicates bad RAM which has to be immediately replaced, because in presence of a high number of corrected errors, a number of silent errors may happen as well, leading to data corruption. Source:/sys/devices/system/edac/mc/
EDACUncorrectable
The number of uncorrectable ECC memory errors, keyed by the memory controller number. A non-zero number of this value indicates bad RAM which has to be immediately replaced, because it indicates potential data corruption. Source:/sys/devices/system/edac/mc/
ExecutableUserDefinedFunctionMemoryResidentBytes
Sum of the resident set size (VmRSS) over all live processes of executable and executable_pool user-defined functions and their descendant processes, in bytes. Idle executable_pool workers are included. Shared pages are counted once per process, so the sum is an upper bound that can exceed the unique physical memory footprint of the UDF processes.ExecutableUserDefinedFunctionProcesses
Number of live processes spawned for executable and executable_pool user-defined functions, including their descendant processes.FilesystemCacheBytes
Total bytes in thecache virtual filesystem. This cache is hold on disk.
FilesystemCacheCapacity
Total capacity in thecache virtual filesystem. This cache is hold on disk.
FilesystemCacheFiles
Total number of cached file segments in thecache virtual filesystem. This cache is hold on disk.
FilesystemLogsPathAvailableBytes
Available bytes on the volume where ClickHouse logs path is mounted. If this value approaches zero, you should tune the log rotation in the configuration file.FilesystemLogsPathAvailableINodes
The number of available inodes on the volume where ClickHouse logs path is mounted.FilesystemLogsPathTotalBytes
The size of the volume where ClickHouse logs path is mounted, in bytes. It’s recommended to have at least 10 GB for logs.FilesystemLogsPathTotalINodes
The total number of inodes on the volume where ClickHouse logs path is mounted.FilesystemLogsPathUsedBytes
Used bytes on the volume where ClickHouse logs path is mounted.FilesystemLogsPathUsedINodes
The number of used inodes on the volume where ClickHouse logs path is mounted.FilesystemMainPathAvailableBytes
Available bytes on the volume where the main ClickHouse path is mounted.FilesystemMainPathAvailableINodes
The number of available inodes on the volume where the main ClickHouse path is mounted. If it is close to zero, it indicates a misconfiguration, and you will get ‘no space left on device’ even when the disk is not full.FilesystemMainPathTotalBytes
The size of the volume where the main ClickHouse path is mounted, in bytes.FilesystemMainPathTotalINodes
The total number of inodes on the volume where the main ClickHouse path is mounted. If it is less than 25 million, it indicates a misconfiguration.FilesystemMainPathUsedBytes
Used bytes on the volume where the main ClickHouse path is mounted.FilesystemMainPathUsedINodes
The number of used inodes on the volume where the main ClickHouse path is mounted. This value mostly corresponds to the number of files.GRPCRejectedConnections
Number of rejected connections for the GRPC protocol.GRPCThreads
Number of threads in the server of the GRPC protocol.HashTableStatsCacheEntries
The number of entries in the cache of hash table sizes. The cache for hash table sizes is used for predictive optimization of GROUP BY.HashTableStatsCacheHits
The number of times the prediction of a hash table size was correct.HashTableStatsCacheMisses
The number of times the prediction of a hash table size was incorrect.HTTPConnectionPoolgroup_nameTCPRcvBufTotalBytes
Total kernel TCP receive buffer memory (sk_rmem_alloc) across all HTTP connection pool sockets.HTTPConnectionPoolgroup_nameTCPSndBufTotalBytes
Total kernel TCP transmit buffer memory (sk_wmem_alloc) across all HTTP connection pool sockets.HTTPRejectedConnections
Number of rejected connections for the HTTP interface (without TLS).HTTPSecureRejectedConnections
Number of rejected connections for the HTTPS interface.HTTPSecureThreads
Number of threads in the server of the HTTPS interface.HTTPThreads
Number of threads in the server of the HTTP interface (without TLS).InterserverRejectedConnections
Number of rejected connections for the replicas communication protocol (without TLS).InterserverSecureRejectedConnections
Number of rejected connections for the replicas communication protocol (with TLS).InterserverSecureThreads
Number of threads in the server of the replicas communication protocol (with TLS).InterserverThreads
Number of threads in the server of the replicas communication protocol (without TLS).jemalloc.active
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.allocated
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.arenas.all.dirty_purged
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.arenas.all.muzzy_purged
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.arenas.all.pactive
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.arenas.all.pdirty
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.arenas.all.pmuzzy
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.arenas.dirty_decay_ms
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.background_thread.num_runs
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.background_thread.num_threads
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.background_thread.run_intervals
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.cache_arena.pactive
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.cache_arena.pdirty
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.epoch
An internal incremental update number of the statistics of jemalloc (Jason Evans’ memory allocator), used in all otherjemalloc metrics.
jemalloc.fragmentation
The difference betweenjemalloc.active and jemalloc.allocated — represents internal fragmentation in the memory allocator (active pages that are not fully utilized by allocations).
jemalloc.jit_arena.active_bytes
Active bytes in the dedicated jemalloc JIT arena. Includes both (a) LLVM heap state (TargetMachine and its target-specific Subtarget, IR modules, optimization-pass analyses, RuntimeDyld relocation tables) and (b) the page-aligned reserved blocks thatJITModuleMemoryManager allocates via posix_memalign for executable/data sections (intercepted into je_posix_memalign, so it goes through jemalloc). CompiledExpressionCacheBytes (CurrentMetric) is a subset of this — it tracks only the reserved page-block capacity for executable/data sections, not the actual code/data bytes in use within those blocks.
jemalloc.jit_arena.dirty_bytes
Dirty bytes in the JIT arena that are eligible for purging back to the OS.jemalloc.jit_arena.pactive
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.jit_arena.pdirty
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.mapped
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.mergetree_arena.active_bytes
Active bytes summed across the dedicated jemalloc MergeTree arena pool (jemalloc.mergetree_arena.count arenas). Holds long-lived MergeTree heap state: per-part metadata (SerializationInfoByName, MergeTreeDataPartChecksums tree, the Poco::LRUCache<String, ColumnSize> delegates inside each IMergeTreeDataPart, the per-part ColumnSize/IndexSize maps, MinMaxIndex, VersionMetadataOnDisk, and the MergeTreeDataPart{Compact,Wide} object itself), metadata shared across parts of a table (NamesAndTypesList, column_name_to_position, the serializations map and ColumnsSubstreams, see SharedPartColumns.h) plus per-table metadata (StorageInMemoryMetadata / ColumnsDescription / VirtualColumnsDescription clones set up by setProperties, the serialization_hints aggregation, and the shared_part_columns_cache). Active parts and outdated parts pending cleanup both contribute. Disjoint from the cache arena and JIT arena. The per-part columns system.parts.primary_key_bytes_in_memory[_allocated] and system.parts.index_granularity_bytes_in_memory[_allocated] are subsets of this metric (when their values are non-zero). The primary index is allocated here even when it is owned by PrimaryIndexCache (deliberate: re-homing it at the cache boundary could fail after a part is already committed), so PrimaryIndexCacheBytes overlaps with this metric.
jemalloc.mergetree_arena.count
Number of dedicated jemalloc arenas for long-lived MergeTree metadata, controlled by thejemalloc_merge_tree_arenas server setting. 0 means the dedicated arena pool is disabled and metadata is allocated in the default per-CPU arenas; 1 is a single shared arena; N > 1 is a pool sharded by CPU. See jemalloc.mergetree_arena.active_bytes.
jemalloc.mergetree_arena.dirty_bytes
Dirty bytes summed across the dedicated jemalloc MergeTree arena pool that are eligible for purging back to the OS.jemalloc.mergetree_arena.pactive
Active pages summed across the dedicated jemalloc MergeTree arena pool.jemalloc.mergetree_arena.pdirty
Dirty pages summed across the dedicated jemalloc MergeTree arena pool.jemalloc.metadata
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.metadata_thp
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.prof.active
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.prof.lg_sample
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.prof.thread_active_init
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.resident
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmljemalloc.retained
An internal metric of the low-level memory allocator (jemalloc). See https://jemalloc.net/jemalloc.3.htmlJitter
The difference in time the thread for calculation of the asynchronous metrics was scheduled to wake up and the time it was in fact, woken up. A proxy-indicator of overall system latency and responsiveness.KeeperApproximateDataSize
The approximate data size of ClickHouse Keeper, in bytes.KeeperAvgLatency
Average request latency of ClickHouse Keeper.KeeperCommitLogsCacheEntries
Number of decoded log entries currently buffered ahead of the commit thread by the changelog read-ahead readerKeeperCommitLogsCacheSize
Total size of decoded log entries currently buffered ahead of the commit thread by the changelog read-ahead readerKeeperEphemeralsCount
The number of ephemeral nodes in ClickHouse Keeper.KeeperFollowers
The number of followers of ClickHouse Keeper.KeeperIsExceedingMemorySoftLimitHit
1 if ClickHouse Keeper is exceeding the memory soft limit, 0 otherwise.KeeperIsFollower
1 if ClickHouse Keeper is a follower, 0 otherwise.KeeperIsLeader
1 if ClickHouse Keeper is a leader, 0 otherwise.KeeperIsObserver
1 if ClickHouse Keeper is an observer, 0 otherwise.KeeperIsStandalone
1 if ClickHouse Keeper is in a standalone mode, 0 otherwise.KeeperLastCommittedLogIdx
Index of the last committed log in ClickHouse Keeper.KeeperLastLeaderElectionTime
Duration in milliseconds of the most recent locally observed no-leader window that ended when this ClickHouse Keeper instance became leader. Leadership transfers that do not expose a sampled no-leader state are not recorded.0 if this instance is not the active leader or has no recorded completed window.
KeeperLastLeaderUnavailableTime
Duration in milliseconds of the most recent locally observed no-leader window completed by this ClickHouse Keeper leader.0 if this instance is not the active leader or has no recorded completed no-leader window.
KeeperLastLogIdx
Index of the last log stored in ClickHouse Keeper.KeeperLastLogTerm
Raft term of the last log stored in ClickHouse Keeper.KeeperLastSnapshotIdx
Index of the last log present in the last created snapshot.KeeperLatestLogsCacheEntries
Number of entries stored in the in-memory cache for latest logsKeeperLatestLogsCacheSize
Total size of in-memory cache for latest logsKeeperLatestSnapshotSize
The uncompressed size in bytes of the latest snapshot created by ClickHouse Keeper.KeeperMaxFileDescriptorCount
The maximum number of open file descriptors in ClickHouse Keeper.-1 if the value cannot be determined.
KeeperMaxLatency
Maximum request latency of ClickHouse Keeper.KeeperMinLatency
Minimal request latency of ClickHouse Keeper.KeeperOpenFileDescriptorCount
The number of open file descriptors in ClickHouse Keeper.-1 if the value cannot be determined.
KeeperPacketsReceived
Number of packets received by ClickHouse Keeper.KeeperPacketsSent
Number of packets sent by ClickHouse Keeper.KeeperPathsWatched
The number of different paths watched by the clients of ClickHouse Keeper.KeeperSessionWithWatches
The number of client sessions of ClickHouse Keeper having watches.KeeperSyncedFollowers
The number of followers of ClickHouse Keeper who are also in-sync.KeeperTargetCommitLogIdx
Index until which logs can be committed in ClickHouse Keeper.KeeperTCPRejectedConnections
Number of rejected connections for the Keeper TCP protocol (without TLS).KeeperTCPSecureRejectedConnections
Number of rejected connections for the Keeper TCP protocol (with TLS).KeeperTCPSecureThreads
Number of threads in the server of the Keeper TCP protocol (with TLS).KeeperTCPThreads
Number of threads in the server of the Keeper TCP protocol (without TLS).KeeperWatchCount
The number of watches in ClickHouse Keeper.KeeperZnodeCount
The number of nodes (data entries) in ClickHouse Keeper.KeeperZxid
The current transaction id number (zxid) in ClickHouse Keeper.LoadAverage1
The whole system load, averaged with exponential smoothing over 1 minute. The load represents the number of threads across all the processes (the scheduling entities of the OS kernel), that are currently running by CPU or waiting for IO, or ready to run but not being scheduled at this point of time. This number includes all the processes, not only clickhouse-server. The number can be greater than the number of CPU cores, if the system is overloaded, and many processes are ready to run but waiting for CPU or IO.LoadAverage15
The whole system load, averaged with exponential smoothing over 15 minutes. The load represents the number of threads across all the processes (the scheduling entities of the OS kernel), that are currently running by CPU or waiting for IO, or ready to run but not being scheduled at this point of time. This number includes all the processes, not only clickhouse-server. The number can be greater than the number of CPU cores, if the system is overloaded, and many processes are ready to run but waiting for CPU or IO.LoadAverage5
The whole system load, averaged with exponential smoothing over 5 minutes. The load represents the number of threads across all the processes (the scheduling entities of the OS kernel), that are currently running by CPU or waiting for IO, or ready to run but not being scheduled at this point of time. This number includes all the processes, not only clickhouse-server. The number can be greater than the number of CPU cores, if the system is overloaded, and many processes are ready to run but waiting for CPU or IO.LongestRunningMerge
Elapsed time in seconds of the longest currently running background merge.MaxPartCountForPartition
Maximum number of parts per partition across all partitions of all tables of MergeTree family. Values larger than 300 indicates misconfiguration, overload, or massive data loading.MemoryCode
The amount of virtual memory mapped for the pages of machine code of the server process, in bytes.MemoryDataAndStack
The amount of virtual memory mapped for the use of stack and for the allocated memory, in bytes. It is unspecified whether it includes the per-thread stacks and most of the allocated memory, that is allocated with the ‘mmap’ system call. This metric exists only for completeness reasons. I recommend to use theMemoryResident metric for monitoring.
MemoryResident
The amount of physical memory used by the server process, in bytes.MemoryResidentMax
Maximum amount of physical memory used by the server process, in bytes.MemoryResidentWithoutPageCache
The amount of physical memory used by the server process, excluding userspace page cache, in bytes. This provides a more accurate view of actual memory usage when userspace page cache is utilized. When userspace page cache is disabled, this value equals MemoryResident.MemoryShared
The amount of memory used by the server process, that is also shared by another processes, in bytes. ClickHouse does not use shared memory, but some memory can be labeled by OS as shared for its own reasons. This metric does not make a lot of sense to watch, and it exists only for completeness reasons.MemoryThreadStacksCount
Number of pthread stack VMAs tagged with[anon:clickhouse_stack] in /proc/self/smaps. Refreshed on the heavy-metrics cadence. Requires Linux 5.17 or newer; absent on older kernels (see the MEMORY_THREAD_STACKS_METRIC_UNAVAILABLE entry in system.warnings).
Number of the task’s VM regions tagged VM_MEMORY_STACK (excluding the inaccessible guard regions). Refreshed on the heavy-metrics cadence.
MemoryThreadStacksResident
Approximate resident set size of pthread stacks, summed fromRss: of /proc/self/smaps VMAs tagged with [anon:clickhouse_stack] via prctl(PR_SET_VMA_ANON_NAME). Refreshed on the heavy-metrics cadence. Requires Linux 5.17 or newer; absent on older kernels (see the MEMORY_THREAD_STACKS_METRIC_UNAVAILABLE entry in system.warnings).
Approximate resident set size of pthread stacks, summed from the resident pages of the task’s VM regions tagged VM_MEMORY_STACK (excluding the inaccessible guard regions). Refreshed on the heavy-metrics cadence.
MemoryThreadStacksVirtual
Approximate virtual size of pthread stacks, summed fromSize: of /proc/self/smaps VMAs tagged with [anon:clickhouse_stack]. Refreshed on the heavy-metrics cadence. Requires Linux 5.17 or newer; absent on older kernels (see the MEMORY_THREAD_STACKS_METRIC_UNAVAILABLE entry in system.warnings).
Approximate virtual size of pthread stacks, summed from the sizes of the task’s VM regions tagged VM_MEMORY_STACK (excluding the inaccessible guard regions). Refreshed on the heavy-metrics cadence.
MemoryVirtual
The size of the virtual address space allocated by the server process, in bytes. The size of the virtual address space is usually much greater than the physical memory consumption, and should not be used as an estimate for the memory consumption. The large values of this metric are totally normal, and makes only technical sense.MissingBlobsQueueEstimate
Estimated number of blobs awaiting replication to other locations of the disk (the blob manager missing queue), keyed by the disk name. Disks without blob replication report 0.MySQLRejectedConnections
Number of rejected connections for the MySQL compatibility protocol.MySQLThreads
Number of threads in the server of the MySQL compatibility protocol.NetworkReceiveBytes
Number of bytes received via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.NetworkReceiveDrop
Number of bytes a packet was dropped while received via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.NetworkReceiveErrors
Number of times error happened receiving via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.NetworkReceivePackets
Number of network packets received via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.NetworkSendBytes
Number of bytes sent via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.NetworkSendDrop
Number of times a packed was dropped while sending via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.NetworkSendErrors
Number of times error (e.g. TCP retransmit) happened while sending via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.NetworkSendPackets
Number of network packets sent via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.NetworkTCPReceiveQueue
Total size of receive queues of network sockets used on the server across TCPv4 and TCPv6.NetworkTCPSocketRemoteAddresses
Total number of unique remote addresses of network sockets used on the server across TCPv4 and TCPv6.NetworkTCPSockets
Total number of network sockets used on the server across TCPv4 and TCPv6, in all states.NetworkTCPSockets_description
Total number of network sockets in the specific state on the server across TCPv4 and TCPv6.NetworkTCPTransmitQueue
Total size of transmit queues of network sockets used on the server across TCPv4 and TCPv6.NetworkTCPUnrecoveredRetransmits
Total size of current retransmits (unrecovered at this moment) of network sockets used on the server across TCPv4 and TCPv6.NumberOfDatabases
Total number of databases on the server.NumberOfDetachedByUserParts
The total number of parts detached from MergeTree tables by users with theALTER TABLE DETACH query (as opposed to unexpected, broken or ignored parts). The server does not care about detached parts and they can be removed.
NumberOfDetachedParts
The total number of parts detached from MergeTree tables. A part can be detached by a user with theALTER TABLE DETACH query or by the server itself it the part is broken, unexpected or unneeded. The server does not care about detached parts and they can be removed.
NumberOfPendingMutations
The total number of mutations that are in left to be mutated.NumberOfPendingMutationsOverExecutionTime
The total number of mutations which have data part left to be mutated over the specified max_pending_mutations_execution_time_to_warn setting.NumberOfTables
Total number of tables summed across the databases on the server, excluding the databases that cannot contain MergeTree tables. The excluded database engines are those who generate the set of tables on the fly, likeLazy, MySQL, PostgreSQL, SQlite.
NumberOfTablesSystem
Total number of tables in the system database on the server stored in tables of MergeTree family.OSContextSwitches
The number of context switches that the system underwent on the host machine. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.OSCPUOverload
Relative CPU deficit, calculated as: how many threads are waiting for CPU relative to the number of threads, using CPU. If it is greater than zero, the server would benefit from more CPU. If it is significantly greater than zero, the server could become unresponsive. The metric is accumulated between the updates of asynchronous metrics.OSGuestNiceTime
The ratio of time spent running a virtual CPU for guest operating systems under the control of the Linux kernel, when a guest was set to a higher priority (Seeman procfs). This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. This metric is irrelevant for ClickHouse, but still exists for completeness. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
OSGuestNiceTimeCPU
The ratio of time every CPU core spent running a virtual CPU for guest operating systems with higher priority, same asOSGuestNiceTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSGuestNiceTimeNormalized
The value is similar toOSGuestNiceTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
OSGuestTime
The ratio of time spent running a virtual CPU for guest operating systems under the control of the Linux kernel (Seeman procfs). This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. This metric is irrelevant for ClickHouse, but still exists for completeness. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
OSGuestTimeCPU
The ratio of time every CPU core spent running a virtual CPU for guest operating systems, same asOSGuestTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSGuestTimeNormalized
The value is similar toOSGuestTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
OSIdleTime
The ratio of time the CPU core was idle (not even ready to run a process waiting for IO) from the OS kernel standpoint. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. This does not include the time when the CPU was under-utilized due to the reasons internal to the CPU (memory loads, pipeline stalls, branch mispredictions, running another SMT core). The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].OSIdleTimeCPU
The ratio of time every CPU core was idle, same asOSIdleTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSIdleTimeNormalized
The value is similar toOSIdleTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
OSInterrupts
The number of interrupts on the host machine. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.OSIOWaitTime
The ratio of time the CPU core was not running the code but when the OS kernel did not run any other process on this CPU as the processes were waiting for IO. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].OSIOWaitTimeCPU
The ratio of time every CPU core was waiting for IO, same asOSIOWaitTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSIOWaitTimeNormalized
The value is similar toOSIOWaitTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
OSIrqTime
The ratio of time spent for running hardware interrupt requests on the CPU. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. A high number of this metric may indicate hardware misconfiguration or a very high network load. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].OSIrqTimeCPU
The ratio of time spent for running hardware interrupt requests on every CPU core, same asOSIrqTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSIrqTimeNormalized
The value is similar toOSIrqTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
OSMemoryAvailable
The amount of memory available to be used by programs, in bytes. This is very similar to theOSMemoryFreePlusCached metric. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSMemoryBuffers
The amount of memory used by OS kernel buffers, in bytes. This should be typically small, and large values may indicate a misconfiguration of the OS. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.OSMemoryCached
The amount of memory used by the OS page cache, in bytes. Typically, almost all available memory is used by the OS page cache - high values of this metric are normal and expected. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.OSMemoryFreePlusCached
The amount of free memory plus OS page cache memory on the host system, in bytes. This memory is available to be used by programs. The value should be very similar toOSMemoryAvailable. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSMemoryFreeWithoutCached
The amount of free memory on the host system, in bytes. This does not include the memory used by the OS page cache memory, in bytes. The page cache memory is also available for usage by programs, so the value of this metric can be confusing. See theOSMemoryAvailable metric instead. For convenience we also provide the OSMemoryFreePlusCached metric, that should be somewhat similar to OSMemoryAvailable. See also https://www.linuxatemyram.com/. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSMemorySwapCached
The amount of memory in swap that was also loaded in RAM. Swap should be disabled on production systems. If the value of this metric is large, it indicates a misconfiguration. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.OSMemoryTotal
The total amount of memory on the host system, in bytes.OSNiceTime
The ratio of time the CPU core was running userspace code with higher priority. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].OSNiceTimeCPU
The ratio of time every CPU core was running userspace code with higher priority, same asOSNiceTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSNiceTimeNormalized
The value is similar toOSNiceTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
OSOpenFiles
The total number of opened files on the host machine. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.OSProcessesBlocked
Number of threads blocked waiting for I/O to complete (man procfs). This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSProcessesCreated
The number of processes created. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.OSProcessesRunning
The number of runnable (running or ready to run) threads by the operating system. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.OSSoftIrqTime
The ratio of time spent for running software interrupt requests on the CPU. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. A high number of this metric may indicate inefficient software running on the system. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].OSSoftIrqTimeCPU
The ratio of time spent for running software interrupt requests on every CPU core, same asOSSoftIrqTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSSoftIrqTimeNormalized
The value is similar toOSSoftIrqTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
OSStealTime
The ratio of time spent in other operating systems by the CPU when running in a virtualized environment. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Not every virtualized environments present this metric, and most of them don’t. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].OSStealTimeCPU
The ratio of time every CPU core spent in other operating systems when running in a virtualized environment, same asOSStealTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSStealTimeNormalized
The value is similar toOSStealTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
OSSystemTime
The ratio of time the CPU core was running OS kernel (system) code. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].OSSystemTimeCPU
The ratio of time every CPU core was running OS kernel (system) code, same asOSSystemTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSSystemTimeNormalized
The value is similar toOSSystemTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
OSThreadsRunnable
The total number of ‘runnable’ threads, as the OS kernel scheduler seeing it.OSThreadsTotal
The total number of threads, as the OS kernel scheduler seeing it.OSUptime
The uptime of the host server (the machine where ClickHouse is running), in seconds.OSUserTime
The ratio of time the CPU core was running userspace code. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. This includes also the time when the CPU was under-utilized due to the reasons internal to the CPU (memory loads, pipeline stalls, branch mispredictions, running another SMT core). The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].OSUserTimeCPU
The ratio of time every CPU core was running userspace code, same asOSUserTime but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
OSUserTimeNormalized
The value is similar toOSUserTime but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
PageCacheMaxBytes
Current limit on the size of userspace page cache, in bytes.PostgreSQLRejectedConnections
Number of rejected connections for the PostgreSQL compatibility protocol.PostgreSQLThreads
Number of threads in the server of the PostgreSQL compatibility protocol.ProcessSignalQueueLimit
Total limit of signal queue (once it reaches ProcessSignalQueueSize, you may get CANNOT_CREATE_TIMER errors)ProcessSignalQueueSize
Size of signal queue (pending signals, timers for query profiling)PrometheusRejectedConnections
Number of rejected connections for the Prometheus endpoint. Note: prometheus endpoints can be also used via the usual HTTP/HTTPs ports.PrometheusThreads
Number of threads in the server of the Prometheus endpoint. Note: prometheus endpoints can be also used via the usual HTTP/HTTPs ports.PSI_type_stall_type
Microseconds of stall time since last measurement.Upstream docs can be found https://docs.kernel.org/accounting/psi.html for the metrics and how to interpret themQueriesMemoryUsage
Total memory currently used by all running queries on the server, in bytes. Useful for attributing memory pressure to the concurrent query load.QueriesPeakMemoryUsage
Sum of per-user query memory peaks across all users tracked inProcessList, in bytes. Each user’s peak is the high-water mark of that user’s memory tracker, which is reset when the user has no running queries. This is therefore an aggregate of currently-tracked per-user peaks, not a single server-wide peak of all queries since startup.
ReaderExecutorModeledCostMsPerRequestedMiB
Experimental ReaderExecutor read-path efficiency: modeled cost (ms) per MiB of requested bytes over the last update interval, instance-wide — the ratio of the deltas of ProfileEvents ReaderExecutorModeledCostMicroseconds and ReaderExecutorDeliveredBytes. Lower is better: the bandwidth floor is ~20 (a clean source read), cache hits trend to 0, over-fetch and incomplete connections push it up. 0 means no executor reads in the interval.ReplicasMaxAbsoluteDelay
Maximum difference in seconds between the most fresh replicated part and the most fresh data part still to be replicated, across Replicated tables. A very high value indicates a replica with no data.ReplicasMaxInsertsInQueue
Maximum number of INSERT operations in the queue (still to be replicated) across Replicated tables.ReplicasMaxMergesInQueue
Maximum number of merge operations in the queue (still to be applied) across Replicated tables.ReplicasMaxQueueSize
Maximum queue size (in the number of operations like get, merge) across Replicated tables.ReplicasMaxRelativeDelay
Maximum difference between the replica delay and the delay of the most up-to-date replica of the same table, across Replicated tables.ReplicasSumInsertsInQueue
Sum of INSERT operations in the queue (still to be replicated) across Replicated tables.ReplicasSumMergesInQueue
Sum of merge operations in the queue (still to be applied) across Replicated tables.ReplicasSumQueueSize
Sum queue size (in the number of operations like get, merge) across Replicated tables.Silkcounter_name
Value of the eponymous low-level counter of the silk fiber runtime, accumulated since the fiber scheduler initialization. Counters with Time in the name are in nanoseconds.TCPRejectedConnections
Number of rejected connections for the TCP protocol (without TLS).TCPSecureRejectedConnections
Number of rejected connections for the TCP protocol (with TLS).TCPSecureThreads
Number of threads in the server of the TCP protocol (with TLS).TCPThreads
Number of threads in the server of the TCP protocol (without TLS).Temperature
The temperature reported by every sensor, in ℃. A sensor can return an unrealistic value. The keys are the thermal zone numbers (source:/sys/class/thermal) or the names of the hardware monitors and their sensors (source: /sys/class/hwmon).
TotalBytesOfMergeTreeTables
Total amount of bytes (compressed, including data and indices) stored in all tables of MergeTree family.TotalBytesOfMergeTreeTablesSystem
Total amount of bytes (compressed, including data and indices) stored in tables of MergeTree family in the system database.TotalIndexGranularityBytesInMemory
The total amount of memory (in bytes) used by index granules (only takes active parts into account).TotalIndexGranularityBytesInMemoryAllocated
The total amount of memory (in bytes) reserved for index granules (only takes active parts into account).TotalPartsOfMergeTreeTables
Total amount of data parts in all tables of MergeTree family. Numbers larger than 10 000 will negatively affect the server startup time and it may indicate unreasonable choice of the partition key.TotalPartsOfMergeTreeTablesSystem
Total amount of data parts in tables of MergeTree family in the system database.TotalPrimaryKeyBytesInMemory
The total amount of memory (in bytes) used by primary key values (only takes active parts into account).TotalPrimaryKeyBytesInMemoryAllocated
The total amount of memory (in bytes) reserved for primary key values (only takes active parts into account).TotalProjectionIndexGranularityBytesInMemory
The total amount of memory (in bytes) used by projection index granularity (only takes active parts into account).TotalProjectionIndexGranularityBytesInMemoryAllocated
The total amount of memory (in bytes) reserved for projection index granularity (only takes active parts into account).TotalProjectionPrimaryKeyBytesInMemory
The total amount of memory (in bytes) used by projection primary key values (only takes active parts into account).TotalProjectionPrimaryKeyBytesInMemoryAllocated
The total amount of memory (in bytes) reserved for projection primary key values (only takes active parts into account).TotalRowsOfMergeTreeTables
Total amount of rows (records) stored in all tables of MergeTree family.TotalRowsOfMergeTreeTablesSystem
Total amount of rows (records) stored in tables of MergeTree family in the system database.TotalUncompressedBytesOfMergeTreeTables
Total amount of uncompressed bytes, as reported by the part checksums, stored in all tables of MergeTree family. It is the same source as thetotal_bytes_uncompressed column of system.tables, and it does not include files that are stored uncompressed, such as marks and primary key indices.
TotalUncompressedBytesOfMergeTreeTablesSystem
Total amount of uncompressed bytes, as reported by the part checksums, stored in tables of MergeTree family in the system database. It is the same source as thetotal_bytes_uncompressed column of system.tables, and it does not include files that are stored uncompressed, such as marks and primary key indices.
TrackedMemory
Memory tracked by ClickHouse (should be equal to MemoryTracking metric), in bytes.UntrackedMemory
Sum of per-thread buffers of recent allocations and deallocations that have not yet been propagated to TrackedMemory.Uptime
The server uptime in seconds. It includes the time spent for server initialization before accepting connections.VMMaxMapCount
The maximum number of memory mappings a process may have (/proc/sys/vm/max_map_count).VMNumMaps
The current number of memory mappings of the process (/proc/self/maps). If it is close to the maximum (VMMaxMapCount), you should increase the limit for vm.max_map_count in /etc/sysctl.confZooKeeperClientLastZXIDSeen
The last ZXID seen by the current ZooKeeper client session. This value increases monotonically as the client observes transactions from ZooKeeper. See Also- Monitoring — Base concepts of ClickHouse monitoring.
- system.metrics — Contains instantly calculated metrics.
- system.events — Contains a number of events that have occurred.
- system.metric_log — Contains a history of metrics values from tables
system.metricsandsystem.events.