Querying in ClickHouse CloudThe data in this system table is held locally on each node in ClickHouse Cloud. Obtaining a complete view of all data, therefore, requires the
clusterAllReplicas function. See here for further details.Description
Contains information about the number of events that have occurred in the system. For example, in the table, you can find how manySELECT queries were processed since the ClickHouse server started.
Event descriptions
ACMEAPIRequests
Number of ACME API requests issued.ACMECertificateOrders
Number of ACME certificate orders issued.AdaptiveAggregationBucketsRetired
Number of two-level buckets whose working memory (arena slot, staged-chunk references) was retired right after their merge-and-convert completed, ahead of the whole merge finishing.AdaptiveAggregationDrainedRecords
How many delayed records the adaptive aggregation drained into the shared table at merge time.AdaptiveAggregationGiveUps
How many threads gave up on freezing in the adaptive aggregation because their stream held few distinct keys.AdaptiveAggregationLocalFreezes
How many local hash tables the adaptive aggregation froze at the freeze threshold.AdaptiveAggregationPressureDrainedRecords
How many staged records the adaptive aggregation drained early under memory pressure.AdaptiveAggregationPressureStandDowns
How many threads in the adaptive aggregation left the learning phase for good because memory pressure crossed the external aggregation threshold, so their local tables spill through the ordinary external aggregation.AdaptiveAggregationPressureSweeps
How many times the adaptive aggregation drained staged records early because of memory pressure.AdaptiveAggregationProbeBypasses
How many threads stopped probing their frozen local table in the adaptive aggregation because almost no row hit it.AdaptiveAggregationResidueReleases
How many times the adaptive aggregation wrote its shared drain table out because a thread back on the baseline algorithm was about to spill on account of it.AdaptiveAggregationSealedChunks
How many coalesced chunks the adaptive aggregation sealed from buffered staging batches.AdaptiveAggregationSealNormalizations
How many staged argument columns the adaptive aggregation seal normalized from a wrapped representation (Const, Replicated, Sparse, LowCardinality) to the dense form the drain consumes.AdaptiveAggregationSharedTableSpills
How many times the adaptive aggregation wrote its shared drain table out because it reached the part bound under memory pressure.AdaptiveAggregationSpillBacklogSheds
How many times the adaptive aggregation shed the staged backlog because a thread on the baseline algorithm - one the thaw put back there, or one that stood down on its own - was about to spill while staged records were still resident.AdaptiveAggregationStagedBytes
How many key bytes the adaptive aggregation staged for the merge-time drain.AdaptiveAggregationStagedChunkPiecesOverBound
How many pieces of a cut staged chunk the adaptive aggregation published still over the bound the chunk was cut to meet. A chunk is cut along bucket boundaries, and record by record inside a bucket that is over the bound on its own, so this is a single record whose staged bytes alone are over the bound; such a piece is claimed whole, and its drain can overshoot the pressure part by that much.AdaptiveAggregationStagedChunkSplits
How many staged chunks the adaptive aggregation cut along bucket boundaries at publication because one chunk’s drain was estimated over the pressure part bound.AdaptiveAggregationStagedClaimsClosedAtBound
How many claims of staged chunks by the adaptive aggregation’s drains were closed before a chunk whose addition would have taken the batch to the pressure part bound, leaving that chunk to the next claim. The bound holds for the batch as drained, so two chunks each just under it - two pieces of a cut chunk that are single records over half a part - are drained into two tables rather than one; only a first chunk that is over the bound alone is claimed regardless.AdaptiveAggregationStagedRecords
How many delayed records the adaptive aggregation staged before deduplication.AdaptiveAggregationStagedRecordsMerged
How many staged records the adaptive aggregation merged away as duplicate keys at publish and at the seal.AdaptiveAggregationThaws
How many times the adaptive aggregation thawed the local tables because the staged stream proved repeat-dominated.AddressesDiscovered
Total count of new addresses in DNS resolve results for HTTP connectionsAddressesExpired
Total count of expired addresses which is no longer presented in DNS resolve results for HTTP connectionsAddressesMarkedAsFailed
Total count of addresses which have been marked as faulty due to connection errors for HTTP connectionsAggregatingSortedMilliseconds
Total time spent while aggregating sorted columnsAggregationBucketTopKConversions
Number of two-level buckets converted through the bucket-local Top-K selection (the aggregationBucketTopK plan optimization).AggregationConvertedToTwoLevel
How many times a single-level aggregation hash table was converted to two-level at runtime.AggregationHashTablesInitializedAsTwoLevel
How many hash tables were inited as two-level for aggregation.AggregationOptimizedEqualRangesOfKeys
For how many blocks optimization of equal ranges of keys was appliedAggregationPreallocatedElementsInHashTables
How many elements were preallocated in hash tables for aggregation.AggregationTopKHeapsFrozen
How many top-K aggregation heaps were frozen, falling back to regular aggregation. Either the heap rejected almost nothing within its observation window (e.g. the number of distinct grouping keys does not exceed the LIMIT), or a tie-set at the heap’s boundary - which can never be evicted - overgrew it (seeenable_group_by_top_k_optimization).
AggregationTopKKeysEvicted
How many grouping keys were evicted from the bounded top-K heap during aggregation (seeenable_group_by_top_k_optimization).
AggregationTopKKeysPruned
How many evicted grouping keys were also erased from the intermediate hash table, with their aggregate states destroyed (seeenable_group_by_top_k_optimization). Lower than AggregationTopKKeysEvicted when the aggregation method cannot erase keys, or when only a prefix of the key is ranked: the heap then still skips rows, but the hash table keeps every admitted group.
AggregationTopKRowsSkipped
How many rows were skipped during aggregation because their grouping key could not enter the top-K result (seeenable_group_by_top_k_optimization).
AIAPICalls
Number of HTTP requests dispatched to AI providers.AIInputTokens
Total prompt tokens consumed across all AI function calls in the query.AIORead
Number of reads with Linux or FreeBSD AIO interfaceAIOReadBytes
Number of bytes read with Linux or FreeBSD AIO interfaceAIOutputTokens
Total completion tokens consumed across all AI function calls in the query.AIOWrite
Number of writes with Linux or FreeBSD AIO interfaceAIOWriteBytes
Number of bytes written with Linux or FreeBSD AIO interfaceAIRowsProcessed
Number of rows that received an AI result.AIRowsSkipped
Number of rows that received a default value due to quota or error.AllUsersThrottlerBytes
Bytes passed through ‘max_network_bandwidth_for_all_users’ throttler.AllUsersThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_network_bandwidth_for_all_users’ throttling.AnalyzePatchRangesMicroseconds
Total time spent analyzing index of patch partsApplyPatchesMicroseconds
Total time spent applying patch parts to blocksApplyPatchMergeOnKeyMicroseconds
Total time spent inside applyPatchesMergeOnKey sort-key merge loopsArenaAllocBytes
Number of bytes allocated for memory Arena (used for GROUP BY and similar operations)ArenaAllocChunks
Number of chunks allocated for memory Arena (used for GROUP BY and similar operations)ASTFuzzerOracleChecks
Number of oracle checks attempted by the server-side AST fuzzer.ASTFuzzerOracleMismatches
Number of oracle mismatches detected by the server-side AST fuzzer.ASTFuzzerOracleTLPAggregateChecks
Number of TLP Aggregate (State/Merge) oracle checks attempted by the server-side AST fuzzer.ASTFuzzerQueries
Number of fuzzed queries attempted by the server-side AST fuzzer.ASTFuzzerSkippedBackupRestore
Number of fuzzed BACKUP/RESTORE queries the server-side AST fuzzer skipped instead of executing.ASTFuzzerSkippedReplicatedDDLInternal
Number of times the server-side AST fuzzer skipped fuzzing because an internal replicated-database DDL execution (a live ZooKeeperMetadataTransaction) was in flight on the context.AsynchronousReaderIgnoredBytes
Number of bytes ignored during asynchronous readingAsynchronousReadWaitMicroseconds
Time spent in waiting for asynchronous reads in asynchronous local read.AsynchronousRemoteReadWaitMicroseconds
Time spent in waiting for asynchronous remote reads.AsyncInsertBytes
Data size in bytes of asynchronous INSERT queries.AsyncInsertCacheHits
Number of times a duplicate hash id has been found in asynchronous INSERT hash id cache.AsyncInsertFlush
Number of flushes of the asynchronous INSERT queue. Every flush inserts a batch of asynchronous INSERT queries as a single part. Compare with AsyncInsertQuery to see how many queries are batched together on average.AsyncInsertQuery
Same as InsertQuery, but only for asynchronous INSERT queries.AsyncInsertRows
Number of rows inserted by asynchronous INSERT queries.AsyncLoaderSpawnFailures
Number of times the async loader could not spawn a worker because the global thread pool could not provide a thread. Queued jobs are then run by a worker the pool already has: one that is running, or one resuming from a wait on a job of another pool.AsyncLoaderWaitMicroseconds
Total time a query was waiting for async loader jobs.AsyncLoggingConsoleDroppedMessages
How many messages have been dropped from the console log due to the async log queue being fullAsyncLoggingConsoleTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the console logAsyncLoggingErrorFileLogDroppedMessages
How many messages have been dropped from error file log due to the async log queue being fullAsyncLoggingErrorFileLogTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the error file logAsyncLoggingFileLogDroppedMessages
How many messages have been dropped from the file log due to the async log queue being fullAsyncLoggingFileLogTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the file logAsyncLoggingSyslogDroppedMessages
How many messages have been dropped from the syslog due to the async log queue being fullAsyncLoggingSyslogTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the syslogAsyncLoggingTextLogDroppedMessages
How many messages have been dropped from text_log due to the async log queue being fullAsyncLoggingTextLogTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the text_logAzureCommitBlockList
Number of Azure blob storage API CommitBlockList callsAzureCopyObject
Number of Azure blob storage API CopyObject callsAzureCreateContainer
Number of Azure blob storage API CreateContainer calls.AzureDeleteObjects
Number of Azure blob storage API DeleteObject(s) calls.AzureGetObject
Number of Azure API GetObject calls.AzureGetProperties
Number of Azure blob storage API GetProperties calls.AzureGetRequestThrottlerBlocked
Number of Azure GET requests blocked by throttler.AzureGetRequestThrottlerCount
Number of Azure GET requests passed through throttler: blocked and not blocked.AzureGetRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform Azure GET request throttling.AzureListObjects
Number of Azure blob storage API ListObjects calls.AzurePutRequestThrottlerBlocked
Number of Azure PUT requests blocked by throttler.AzurePutRequestThrottlerCount
Number of Azure PUT requests passed through throttler: blocked and not blocked.AzurePutRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform Azure PUT request throttling.AzureReadMicroseconds
Total time spent waiting for Azure read requests.AzureReadRequestsCount
Number of Azure read requests.AzureReadRequestsErrors
Number of Azure read request errors.AzureReadRequestsRedirects
Number of Azure read request redirects.AzureReadRequestsThrottling
Number of Azure read requests throttled.AzureStageBlock
Number of Azure blob storage API StageBlock callsAzureUpload
Number of Azure blob storage API Upload callsAzureWriteMicroseconds
Total time spent waiting for Azure write requests.AzureWriteRequestsCount
Number of Azure write requests.AzureWriteRequestsErrors
Number of Azure write request errors.AzureWriteRequestsRedirects
Number of Azure write request redirects.AzureWriteRequestsThrottling
Number of Azure write requests throttled.BackgroundLoadingMarksTasks
Number of background tasks for loading marksBackupEntriesCollectorForTablesDataMicroseconds
Time spent making backup entries for tables dataBackupEntriesCollectorMicroseconds
Time spent making backup entriesBackupEntriesCollectorRunPostTasksMicroseconds
Time spent running post tasks after making backup entriesBackupLockFileReads
How many times the ‘.lock’ file was read while making backupBackupPreparingFileInfosMicroseconds
Time spent preparing file infos for backup entriesBackupReadLocalBytesToCalculateChecksums
Total size of files read locally to calculate checksums for backup entriesBackupReadLocalFilesToCalculateChecksums
Number of files read locally to calculate checksums for backup entriesBackupReadMetadataMicroseconds
Time spent reading backup metadata from .backup fileBackupReadRemoteBytesToCalculateChecksums
Total size of files read from remote disks to calculate checksums for backup entriesBackupReadRemoteFilesToCalculateChecksums
Number of files read from remote disks to calculate checksums for backup entriesBackupsOpenedForRead
Number of backups opened for readingBackupsOpenedForUnlock
Number of backups opened for unlockingBackupsOpenedForWrite
Number of backups opened for writingBackupThrottlerBytes
Bytes passed through ‘max_backup_bandwidth_for_server’ throttler.BackupThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_backup_bandwidth_for_server’ throttling.BackupWriteMetadataMicroseconds
Time spent writing backup metadata to .backup fileBlobCopierThreadLockBlobsErrors
Number of blobs lock errors occurred during BlobCopier executionBlobCopierThreadLockedBlobs
Number of blobs returned from metadata storageBlobCopierThreadRecordBlobsErrors
Number of blobs recording errors occurred during BlobCopier executionBlobCopierThreadRecordedBlobs
Number of blobs which replication by BlobCopier was recorded in metadata storageBlobCopierThreadReplicateBlobsErrors
Number of blobs replication errors occurred during BlobCopier executionBlobCopierThreadReplicatedBlobs
Number of blobs replicated by BlobCopierBlobCopierThreadRuns
Number of BlobCopier thread executesBlobKillerThreadLockBlobsErrors
Number of blobs lock errors occurred during BlobKiller executionBlobKillerThreadLockedBlobs
Number of blobs returned from metadata storageBlobKillerThreadRecordBlobsErrors
Number of blobs recording errors occurred during BlobKiller executionBlobKillerThreadRecordedBlobs
Number of blobs which removal by BlobKiller was recorded in metadata storageBlobKillerThreadRemoveBlobsErrors
Number of blobs removal errors occurred during BlobKiller executionBlobKillerThreadRemovedBlobs
Number of blobs removed by BlobKillerBlobKillerThreadRemoveTasks
Number of remove tasks created by BlobKillerBlobKillerThreadRuns
Number of BlobKiller thread executesBuildPatchesJoinMicroseconds
Total time spent building indexes and hash tables for applying patch parts with Join modeBuildPatchesMergeMicroseconds
Total time spent building indexes for applying patch parts with Merge modeCachedReadBufferCacheWriteBytes
Bytes written from source (remote fs, etc) to filesystem cacheCachedReadBufferCacheWriteMicroseconds
Time spent writing data into filesystem cacheCachedReadBufferCacheWriteStopped
Number of times writing a file segment into the filesystem cache was stopped mid-download (space reservation or the cache write failed), after which the read continues bypassing the cacheCachedReadBufferCreateBufferMicroseconds
Prepare buffer timeCachedReadBufferPredownloadedBytes
Bytes read from filesystem cache source. Cache segments are read from left to right as a whole, it might be that we need to predownload some part of the segment irrelevant for the current task just to get to the needed dataCachedReadBufferPredownloadedFromSourceBytes
Bytes read from filesystem cache source for predownload (from remote fs, etc)CachedReadBufferPredownloadedFromSourceMicroseconds
Time reading from filesystem cache source for predownload (from remote filesystem, etc)CachedReadBufferReadFromCacheBytes
Bytes read from filesystem cacheCachedReadBufferReadFromCacheHits
Number of times the read from filesystem cache hit the cache.CachedReadBufferReadFromCacheMicroseconds
Time reading from filesystem cacheCachedReadBufferReadFromCacheMisses
Number of times the read from filesystem cache miss the cache.CachedReadBufferReadFromSourceBytes
Bytes read from filesystem cache source (from remote fs, etc)CachedReadBufferReadFromSourceMicroseconds
Time reading from filesystem cache source (from remote filesystem, etc)CachedReadBufferWaitReadBufferMicroseconds
Time spend waiting for internal read buffer (includes cache waiting)CachedWriteBufferCacheWriteBytes
Bytes written from source (remote fs, etc) to filesystem cacheCachedWriteBufferCacheWriteMicroseconds
Time spent writing data into filesystem cacheCachedWriteBufferCacheWriteStopped
Number of times write-through caching was stopped (space reservation or the cache write failed, or a covering segment was being evicted), after which the write continues without populating the cacheCachedWriteBufferCoveringSegmentShrinkFailed
Number of times write-through caching had to stop because a covering file segment which stayed behind the offset being written at could not be shrunk (it is held by someone else, so the hole in it cannot be resolved) or was concurrently evictedCachedWriteBufferCoveringSegmentShrunk
Number of times write-through caching shrunk a covering file segment which stayed behind the offset being written at (the ranges in between went to another distributed cache server), in order to continue caching in a new file segmentCacheWarmerBytesDownloaded
Amount of data fetched into filesystem cache by dedicated background threads.CacheWarmerDataPartsDownloaded
Number of data parts that were fully fetched by CacheWarmer.CannotRemoveEphemeralNode
Number of times an error happened while trying to remove ephemeral node. This is not an issue, because our implementation of ZooKeeper library guarantee that the session will expire and the node will be removed.CannotWriteToWriteBufferDiscard
Number of stack traces dropped by query profiler or signal handler because pipe is full or cannot write to pipe.CoalescingSortedMilliseconds
Total time spent while coalescing sorted columnsCollapsingSortedMilliseconds
Total time spent while collapsing sorted columnsCommonBackgroundExecutorTaskCancelMicroseconds
Time spent in cancel() for Common executor tasks.CommonBackgroundExecutorTaskExecuteStepMicroseconds
Time spent in executeStep() for Common executor tasks.CommonBackgroundExecutorTaskResetMicroseconds
Time spent resetting task for Common executor.CommonBackgroundExecutorWaitMicroseconds
Time spent waiting for completion in Common executor.CompiledFunctionExecute
Number of times a compiled function was executed.CompileExpressionsBytes
Number of bytes used for expressions compilation.CompileExpressionsMicroseconds
Total time spent for compilation of expressions to LLVM code.CompileFunction
Number of times a compilation of generated LLVM code (to create fused function for complex expressions) was initiated.CompileRegexpFunction
Number of times a regular expression was JIT-compiled to machine code.CompressedReadBufferBlocks
Number of compressed blocks (the blocks of data that are compressed independent of each other) read from compressed sources (files, network).CompressedReadBufferBytes
Number of uncompressed bytes (the number of bytes after decompression) read from compressed sources (files, network).CompressedReadBufferChecksumDoesntMatch
Number of times the compressed block checksum did not match.CompressedReadBufferChecksumDoesntMatchMicroseconds
Total time spent detecting bit-flips due to compressed block checksum mismatches.CompressedReadBufferChecksumDoesntMatchSingleBitMismatch
Number of times a compressed block checksum mismatch was caused by a single-bit difference.ConcurrencyControlDownscales
Total number of CPU downscaling eventsConcurrencyControlPreemptedMicroseconds
Total time a query was waiting due to preemption of CPU slots.ConcurrencyControlPreemptions
Total number of CPU preemptionsConcurrencyControlQueriesDelayed
Number of queries delayed by CPU slot capacity pressure. Contention indicator — excludes lazy-upscale waits on an uncontested server.ConcurrencyControlSlotsAcquired
Total number of CPU slots acquiredConcurrencyControlSlotsAcquiredNonCompeting
Total number of noncompeting CPU slot acquiredConcurrencyControlSlotsDelayed
Number of CPU slots not granted at allocation time due to capacity pressure. Excludes slots held back by lazy allocation.ConcurrencyControlSlotsGranted
Number of CPU slot granted according to guarantee of 1 thread per query and for queries with setting ‘use_concurrency_control’ = 0ConcurrencyControlUpscales
Total number of CPU upscaling eventsConcurrencyControlWaitMicroseconds
Total time a query was waiting on resource requests for CPU slots.ConcurrentQuerySlotsAcquired
Total number of query slots acquiredConcurrentQueryWaitMicroseconds
Total time a query was waiting for a query slotsConnectionPoolIsFullMicroseconds
Total time spent waiting for a slot in connection pool.ContextLock
Number of times the lock of Context was acquired or tried to acquire. This is global lock.ContextLockWaitMicroseconds
Context lock wait time in microsecondsCoordinatedMergesMergeAssignmentRequest
Total number of merge assignment requestsCoordinatedMergesMergeAssignmentRequestMicroseconds
Total time spend in merge assignment clientCoordinatedMergesMergeAssignmentResponse
Total number of merge assignment requestsCoordinatedMergesMergeAssignmentResponseMicroseconds
Total time spend in merge assignment handlerCoordinatedMergesMergeCoordinatorFetchMetadataMicroseconds
Total time spend on fetching fresh metadata inside merge coordinatorCoordinatedMergesMergeCoordinatorFilterMicroseconds
Total time spend on filtering prepared merges inside merge coordinatorCoordinatedMergesMergeCoordinatorLockStateExclusivelyCount
Total number of exclusive captures of coordinator state lockCoordinatedMergesMergeCoordinatorLockStateExclusivelyMicroseconds
Total time spend on locking coordinator state mutex exclusivelyCoordinatedMergesMergeCoordinatorLockStateForShareCount
Total number of for share captures of coordinator state lockCoordinatedMergesMergeCoordinatorLockStateForShareMicroseconds
Total time spend on locking coordinator state mutex for shareCoordinatedMergesMergeCoordinatorSelectMergesMicroseconds
Total time spend on finding merge using merge selectors inside merge coordinatorCoordinatedMergesMergeCoordinatorUpdateCount
Total number of merge coordinator updatesCoordinatedMergesMergeCoordinatorUpdateMicroseconds
Total time spend on updating merge coordinator stateCoordinatedMergesMergeWorkerUpdateCount
Total number merge worker updatesCoordinatedMergesMergeWorkerUpdateMicroseconds
Total time spend on updating local state of assigned merges on workerCreatedLogEntryForMerge
Successfully created log entry to merge parts in ReplicatedMergeTree.CreatedLogEntryForMutation
Successfully created log entry to mutate parts in ReplicatedMergeTree.CreatedReadBufferDirectIO
Number of times a read buffer with O_DIRECT was created for reading data (while choosing among other read methods).CreatedReadBufferDirectIOFailed
Number of times a read buffer with O_DIRECT was attempted to be created for reading data (while choosing among other read methods), but the OS did not allow it (due to lack of filesystem support or other reasons) and we fallen back to the ordinary reading method.CreatedReadBufferMMap
Number of times a read buffer using ‘mmap’ was created for reading data (while choosing among other read methods).CreatedReadBufferMMapFailed
Number of times a read buffer with ‘mmap’ was attempted to be created for reading data (while choosing among other read methods), but the OS did not allow it (due to lack of filesystem support or other reasons) and we fallen back to the ordinary reading method.CreatedReadBufferOrdinary
Number of times ordinary read buffer was created for reading data (while choosing among other read methods).DataAfterMergeDiffersFromReplica
Number of times data after merge is not byte-identical to the data on another replicas. There could be several reasons:- Using newer version of compression library after server update.
- Using another compression method.
- Non-deterministic compression algorithm (highly unlikely).
- Non-deterministic merge algorithm due to logical error in code.
- Data corruption in memory due to bug in code.
- Data corruption in memory due to hardware issue.
- Manual modification of source data after server startup.
- Manual modification of checksums stored in ZooKeeper.
- Part format related settings like ‘enable_mixed_granularity_parts’ are different on different replicas. The server successfully detected this situation and will download merged part from the replica to force the byte-identical result.
DataAfterMutationDiffersFromReplica
Number of times data after mutation is not byte-identical to the data on other replicas. In addition to the reasons described in ‘DataAfterMergeDiffersFromReplica’, it is also possible due to non-deterministic mutation.DefaultImplementationForNullsRows
Number of rows processed by default implementation for nulls in function executionDefaultImplementationForNullsRowsWithNulls
Number of rows which contain null values processed by default implementation for nulls in function executionDelayedInserts
Number of times the INSERT of a block to a MergeTree table was throttled due to high number of active data parts for partition.DelayedInsertsMilliseconds
Total number of milliseconds spent while the INSERT of a block to a MergeTree table was throttled due to high number of active data parts for partition.DelayedMutations
Number of times the mutation of a MergeTree table was throttled due to high number of unfinished mutations for table.DelayedMutationsMilliseconds
Total number of milliseconds spent while the mutation of a MergeTree table was throttled due to high number of unfinished mutations for table.DeltaLakeDeltaLogExistenceChecks
Number of object-storage listings issued to check whether a DeltaLake_delta_log exists (e.g. during CREATE TABLE).
DeltaLakePartitionPrunedFiles
Number of skipped files during DeltaLake partition pruningDeltaLakeScannedFiles
Number of files scanned during DeltaLake scan callbacksDeltaLakeSnapshotInitializations
Number of times a DeltaLake table snapshot was initialized (loaded from object storage)DictCacheKeysExpired
Number of keys looked up in the dictionaries of ‘cache’ types and found in the cache but they were obsolete.DictCacheKeysHit
Number of keys looked up in the dictionaries of ‘cache’ types and found in the cache.DictCacheKeysNotFound
Number of keys looked up in the dictionaries of ‘cache’ types and not found.DictCacheKeysRequested
Number of keys requested from the data source for the dictionaries of ‘cache’ types.DictCacheKeysRequestedFound
Number of keys requested from the data source for dictionaries of ‘cache’ types and found in the data source.DictCacheKeysRequestedMiss
Number of keys requested from the data source for dictionaries of ‘cache’ types but not found in the data source.DictCacheLockReadNs
Number of nanoseconds spend in waiting for read lock to lookup the data for the dictionaries of ‘cache’ types.DictCacheLockWriteNs
Number of nanoseconds spend in waiting for write lock to update the data for the dictionaries of ‘cache’ types.DictCacheRequests
Number of bulk requests to the external data sources for the dictionaries of ‘cache’ types.DictCacheRequestTimeNs
Number of nanoseconds spend in querying the external data sources for the dictionaries of ‘cache’ types.DirectorySync
Number of times the F_FULLFSYNC/fsync/fdatasync function was called for directories.DirectorySyncElapsedMicroseconds
Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for directories.DiskAzureCommitBlockList
Number of Disk Azure blob storage API CommitBlockList callsDiskAzureCopyObject
Number of Disk Azure blob storage API CopyObject callsDiskAzureCreateContainer
Number of Disk Azure blob storage API CreateContainer calls.DiskAzureDeleteObjects
Number of Azure blob storage API DeleteObject(s) calls.DiskAzureGetObject
Number of Disk Azure API GetObject calls.DiskAzureGetProperties
Number of Disk Azure blob storage API GetProperties calls.DiskAzureGetRequestThrottlerBlocked
Number of Azure disk GET requests blocked by throttler.DiskAzureGetRequestThrottlerCount
Number of Azure disk GET requests passed through throttler: blocked and not blocked.DiskAzureGetRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform Azure disk GET request throttling.DiskAzureListObjects
Number of Disk Azure blob storage API ListObjects calls.DiskAzurePutRequestThrottlerBlocked
Number of Azure disk PUT requests blocked by throttler.DiskAzurePutRequestThrottlerCount
Number of Azure disk PUT requests passed through throttler: blocked and not blocked.DiskAzurePutRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform Azure disk PUT request throttling.DiskAzureReadMicroseconds
Total time spent waiting for Azure disk read requests.DiskAzureReadRequestsCount
Number of Azure disk read requests.DiskAzureReadRequestsErrors
Number of Azure disk read request errors.DiskAzureReadRequestsRedirects
Number of Azure disk read request redirects.DiskAzureReadRequestsThrottling
Number of Azure disk read requests throttled.DiskAzureStageBlock
Number of Disk Azure blob storage API StageBlock callsDiskAzureUpload
Number of Disk Azure blob storage API Upload callsDiskAzureWriteMicroseconds
Total time spent waiting for Azure disk write requests.DiskAzureWriteRequestsCount
Number of Azure disk write requests.DiskAzureWriteRequestsErrors
Number of Azure disk write request errors.DiskAzureWriteRequestsRedirects
Number of Azure disk write request redirects.DiskAzureWriteRequestsThrottling
Number of Azure disk write requests throttled.DiskConnectionsCreated
Number of created connections for diskDiskConnectionsElapsedMicroseconds
Total time spend on creating connections for diskDiskConnectionsErrors
Number of cases when creation of a connection for disk is failedDiskConnectionsExpired
Number of expired connections for diskDiskConnectionsPreserved
Number of preserved connections for diskDiskConnectionsReset
Number of reset connections for diskDiskConnectionsReused
Number of reused connections for diskDiskObjectStorageWaitBlobRemovalMicroseconds
Time spent waiting for pending blob removal after committing metadata transactionDiskPlainRewritableAzureDirectoryCreated
Number of directories created by the ‘plain_rewritable’ metadata storage for AzureObjectStorage.DiskPlainRewritableAzureDirectoryRemoved
Number of directories removed by the ‘plain_rewritable’ metadata storage for AzureObjectStorage.DiskPlainRewritableLegacyLayoutDiskCount
Number of the ‘plain_rewritable’ disks with legacy layout.DiskPlainRewritableLocalDirectoryCreated
Number of directories created by the ‘plain_rewritable’ metadata storage for LocalObjectStorage.DiskPlainRewritableLocalDirectoryRemoved
Number of directories removed by the ‘plain_rewritable’ metadata storage for LocalObjectStorage.DiskPlainRewritableS3DirectoryCreated
Number of directories created by the ‘plain_rewritable’ metadata storage for S3ObjectStorage.DiskPlainRewritableS3DirectoryRemoved
Number of directories removed by the ‘plain_rewritable’ metadata storage for S3ObjectStorage.DiskPlainRewritableUndoStageRetries
Number of times a step of reversing a failed ‘plain_rewritable’ metadata transaction had to be repeated because object storage rejected it.DiskReadElapsedMicroseconds
Total time spent waiting for read syscall. This include reads from page cache.DiskS3AbortMultipartUpload
Number of DiskS3 API AbortMultipartUpload calls.DiskS3CompleteMultipartUpload
Number of DiskS3 API CompleteMultipartUpload calls.DiskS3CopyObject
Number of DiskS3 API CopyObject calls.DiskS3CreateMultipartUpload
Number of DiskS3 API CreateMultipartUpload calls.DiskS3DeleteObjects
Number of DiskS3 API DeleteObject(s) calls.DiskS3GetObject
Number of DiskS3 API GetObject calls.DiskS3GetObjectTagging
Number of DiskS3 API GetObjectTagging calls.DiskS3GetRequestThrottlerBlocked
Number of DiskS3 GET and SELECT requests blocked by throttler.DiskS3GetRequestThrottlerCount
Number of DiskS3 GET and SELECT requests passed through throttler: blocked and not blocked.DiskS3GetRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform DiskS3 GET and SELECT request throttling.DiskS3HeadObject
Number of DiskS3 API HeadObject calls.DiskS3ListObjects
Number of DiskS3 API ListObjects calls.DiskS3PutObject
Number of DiskS3 API PutObject calls.DiskS3PutRequestThrottlerBlocked
Number of DiskS3 PUT, COPY, POST and LIST requests blocked by throttler.DiskS3PutRequestThrottlerCount
Number of DiskS3 PUT, COPY, POST and LIST requests passed through throttler: blocked and not blocked.DiskS3PutRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform DiskS3 PUT, COPY, POST and LIST request throttling.DiskS3ReadMicroseconds
Time of GET and HEAD requests to DiskS3 storage.DiskS3ReadRequestAttempts
Number of attempts for GET and HEAD requests to DiskS3 storage, including the initial try and any retries, but excluding retries performed internally by the S3 retry strategyDiskS3ReadRequestRetryableErrors
Number of retryable errors for GET and HEAD requests to DiskS3 storage, excluding retries performed internally by the S3 retry strategyDiskS3ReadRequestsCount
Number of GET and HEAD requests to DiskS3 storage.DiskS3ReadRequestsErrors
Number of non-throttling errors in GET and HEAD requests to DiskS3 storage.DiskS3ReadRequestsRedirects
Number of redirects in GET and HEAD requests to DiskS3 storage.DiskS3ReadRequestsThrottling
Number of 429 and 503 errors in GET and HEAD requests to DiskS3 storage.DiskS3UploadPart
Number of DiskS3 API UploadPart calls.DiskS3UploadPartCopy
Number of DiskS3 API UploadPartCopy calls.DiskS3WriteMicroseconds
Time of POST, DELETE, PUT and PATCH requests to DiskS3 storage.DiskS3WriteRequestAttempts
Number of attempts for POST, DELETE, PUT and PATCH requests to DiskS3 storage, including the initial try and any retries, but excluding retries performed internally by the retry strategyDiskS3WriteRequestRetryableErrors
Number of retryable errors for POST, DELETE, PUT and PATCH requests to DiskS3 storage, excluding retries performed internally by the retry strategyDiskS3WriteRequestsCount
Number of POST, DELETE, PUT and PATCH requests to DiskS3 storage.DiskS3WriteRequestsErrors
Number of non-throttling errors in POST, DELETE, PUT and PATCH requests to DiskS3 storage.DiskS3WriteRequestsRedirects
Number of redirects in POST, DELETE, PUT and PATCH requests to DiskS3 storage.DiskS3WriteRequestsThrottling
Number of 429 and 503 errors in POST, DELETE, PUT and PATCH requests to DiskS3 storage.DiskWriteElapsedMicroseconds
Total time spent waiting for write syscall. This include writes to page cache.DistinctTransformsAbandonedDeduplication
How many deduplication transforms dropped their hash table and stopped deduplicating because the observed input was almost entirely unique and a consumer downstream deduplicates anyway: the preliminaryDISTINCT (see allow_preliminary_distinct_abandoning) and the per-stream pre-deduplication in front of an IN-subquery set fill.
DistinctTransformsSwitchedToPassThrough
How many preliminaryDISTINCT transforms freed their hash table and started passing every row through because query memory exceeded the external DISTINCT threshold, or projected hashing allocations would exceed the remaining threshold budget (see max_bytes_before_external_distinct): the final DISTINCT, which spills to disk, resolves the duplicates instead.
DistrCacheBackgroundWrites
Distributed Cache write buffer event. Number of cache writes that ran on a background thread (a background write slot was acquired)DistrCacheConnectAttempts
Distributed Cache connection event. The number of connection attempts to distributed cacheDistrCacheConnectErrors
Distributed Cache client event. Number of failures to connect to a distributed cache server before making a request (counted once per request, after all connect attempts are exhausted, unlike per-attempt DistrCacheUnsuccessfulConnectAttempts). A timeout while waiting for a free pooled connection is counted here as well. Failures to reconnect during request creation are counted in DistrCacheMakeRequestErrors insteadDistrCacheConnectMicroseconds
Distributed Cache connection event. The time spent to connect to distributed cacheDistrCacheEstimatedInflightBytesOnSeek
Distributed Cache read buffer event. The estimated in-flight bytes at the seeks which considered discarding the connection (compare with DistrCacheUnusedDataPacketsBytesReadRangeIdChanged)DistrCacheFallbackReadMicroseconds
Distributed Cache read buffer event. Time spend reading from fallback buffer instead of distributed cacheDistrCacheGetClientMicroseconds
Distributed Cache connection event. Time spent getting client for distributed cacheDistrCacheGetResponseMicroseconds
Distributed Cache client event. Time spend to wait for response from distributed cacheDistrCacheHashRingRebuilds
Distributed Cache registry event. Number of distributed cache hash ring rebuildsDistrCacheInlineWrites
Distributed Cache write buffer event. Number of cache writes that ran inline on the calling thread (writes of temporary data, which has no object storage write to overlap with)DistrCacheLockRegistryMicroseconds
Distributed Cache registry event. Time spent to take DistributedCacheRegistry lockDistrCacheMakeRequestErrors
Distributed Cache client event. Number of distributed cache errors when making a requestDistrCacheNextImplMicroseconds
Distributed Cache read buffer event. Time spend in ReadBufferFromDistributedCache::nextImplDistrCacheObjectStorageWriteBytes
Distributed Cache write buffer event. Number of bytes written to object storageDistrCacheObjectStorageWriteMicroseconds
Distributed Cache write buffer event. Time spent writing to object storageDistrCacheOpenedConnections
Distributed Cache connection event. The number of open connections to distributed cacheDistrCacheOpenedConnectionsBypassingPool
Distributed Cache connection event. The number of open connections to distributed cache bypassing poolDistrCachePrecomputeRangesMicroseconds
Distributed Cache read buffer event. Time spent to precompute read rangesDistrCacheRangeChange
Distributed Cache read buffer event. Number of times we changed read range because of seek/last_position changeDistrCacheRangeResetBackward
Distributed Cache read buffer event. Number of times we reset read range because of seek/last_position changeDistrCacheRangeResetForward
Distributed Cache read buffer event. Number of times we reset read range because of seek/last_position changeDistrCacheReadBytesFromFallbackBuffer
Distributed Cache read buffer event. Bytes read from fallback bufferDistrCacheReadBytesSkippedOnSeek
Distributed Cache read buffer event. In-flight bytes discarded on a forward seek to reuse the open streamDistrCacheReadBytesUnconsumedOnEndRequest
Distributed Cache read buffer event. Bytes of the requested read range left unconsumed when the request was ended earlyDistrCacheReadErrors
Distributed Cache client event. Total number of distributed cache errors during read requests (equals DistrCacheReadErrorsRetriable + DistrCacheReadErrorsNonRetriable)DistrCacheReadErrorsNonRetriable
Distributed Cache client event. Number of non-retriable distributed cache errors during read requestsDistrCacheReadErrorsRetriable
Distributed Cache client event. Number of retriable distributed cache errors during read requests (counted on each occurrence, including the ones recovered by a retry)DistrCacheReadMicroseconds
Distributed Cache read buffer event. Time spent reading from distributed cacheDistrCacheReadRequestRetries
Distributed Cache read buffer event. The number of read request retries after a retriable error (including retries of the initial request creation)DistrCacheReadRequestRetriesExhausted
Distributed Cache read buffer event. Number of times a retriable read error did not succeed within the read request retries limitDistrCacheReadThrottlerBytes
Bytes passed through ‘max_distributed_cache_read_bandwidth_for_server’ throttler.DistrCacheReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_distributed_cache_read_bandwidth_for_server’ throttling.DistrCacheReceivedCredentialsRefreshPackets
Distributed Cache client event. Total number of received RefreshCredentials packets received from distributed cacheDistrCacheReceivedDataPackets
Distributed Cache client event. Total number of received data packets received from distributed cacheDistrCacheReceivedDataPacketsBytes
Distributed Cache client event. The number of bytes in Data packets received from distributed cacheDistrCacheReceivedErrorPackets
Distributed Cache client event. Total number of received Error packets received from distributed cacheDistrCacheReceivedOkPackets
Distributed Cache client event. Total number of received Ok packets received from distributed cacheDistrCacheReceivedSlowDownPackets
Distributed Cache client event. Total number of received SlowDown packets received from distributed cache (the server rejected a new connection because this client has too many open connections relative to its weight)DistrCacheReceivedStopPackets
Distributed Cache client event. Total number of received Stop packets received from distributed cacheDistrCacheReceiveResponseErrors
Distributed Cache client event. Number of distributed cache errors when receiving response a requestDistrCacheRegistryUpdateMicroseconds
Distributed Cache registry event. Time spent updating distributed cache registryDistrCacheRegistryUpdates
Distributed Cache registry event. Number of distributed cache registry updatesDistrCacheRemoveOutdatedMicroseconds
Distributed Cache connection event. The time spent removing outdated connections from the poolDistrCacheReusedConnections
Distributed Cache connection event. The number of reused connections to distributed cacheDistrCacheSeekBufferedBytesShortfall
Distributed Cache read buffer event. The number of bytes the locally buffered data fell short of the gap when a skip was declinedDistrCacheSeekCancelledRequests
Distributed Cache read buffer event. The number of times a seek cancelled the open request instead of reusing the streamDistrCacheSeekReRequestFirstPacketMicroseconds
Distributed Cache read buffer event. Time between requesting a new read range on the open stream because of a seek and receiving its first packetDistrCacheSeekReRequests
Distributed Cache read buffer event. The number of times a seek requested a new read range on the same connectionDistrCacheSeekSkipDeclinedGapNotBuffered
Distributed Cache read buffer event. The number of forward seeks which could not skip because the gap was not buffered locallyDistrCacheSeekSkips
Distributed Cache read buffer event. The number of forward seeks which skipped the gap on the open streamDistrCacheSentDataPackets
Distributed Cache client event. Total number of data packets sent to distributed cacheDistrCacheSentDataPacketsBytes
Distributed Cache client event. The number of bytes in Data packets sent to distributed cacheDistrCacheServerAckRequestPackets
Distributed Cache server event. Number of AckRequest packets in DistributedCacheServerDistrCacheServerCachedReadBufferCacheHits
Distributed Cache server event. The number of times distributed cache hit the cache while reading from filesystem cacheDistrCacheServerCachedReadBufferCacheMisses
Distributed Cache server event. The number of times distributed cache missed the cache while reading from filesystem cacheDistrCacheServerCachedReadBufferCachePredownloadBytes
Distributed Cache server event. The number of bytes read from object storage for predownload in distributed cache while reading from filesystem cacheDistrCacheServerCachedReadBufferCacheReadBytes
Distributed Cache server event. The number of bytes read from cache in distributed cache while reading from filesystem cacheDistrCacheServerCachedReadBufferCacheWrittenBytes
Distributed Cache server event. The number of bytes written to cache in distributed cache while reading from filesystem cacheDistrCacheServerCachedReadBufferObjectStorageReadBytes
Distributed Cache server event. The number of bytes read from object storage in distributed cache while reading from filesystem cacheDistrCacheServerConnectionRegistryClientLockWaitMicroseconds
Distributed Cache server event. Time spent waiting for a per-client connection bucket mutex on contention (register/unregister/identify/eviction)DistrCacheServerConnectionRegistryLockWaitMicroseconds
Distributed Cache server event. Time spent waiting for the global connection registry mutex on contention (identify, setClientWeight, eviction, and unregister when a reindex or bucket removal is due)DistrCacheServerConnectionsDroppedOverLimit
Distributed Cache server event. The number of connections closed because the number of connections exceeded the limit (closed from the client with the maximum number of connections relative to its weight)DistrCacheServerConnectionsRejectedOverLimit
Distributed Cache server event. The number of new connections rejected with a SlowDown packet because the connecting client exceeded its share of connectionsDistrCacheServerContinueRequestPackets
Distributed Cache server event. Number of ContinueRequest packets in DistributedCacheServerDistrCacheServerCredentialsRefresh
Distributed Cache server event. The number of expired credentials were refreshedDistrCacheServerEndRequestPackets
Distributed Cache server event. Number of EndRequest packets in DistributedCacheServerDistrCacheServerNewS3CachedClients
Distributed Cache server event. The number of new cached s3 clientsDistrCacheServerProcessRequestMicroseconds
Distributed Cache server event. Time spent processing request on DistributedCache server sideDistrCacheServerReceivedCredentialsRefreshPackets
Distributed Cache server event. Number of RefreshCredentials client packets in DistributedCacheServerDistrCacheServerReusedS3CachedClients
Distributed Cache server event. The number of reused cached s3 clientsDistrCacheServerShutdownConnectionsMicroseconds
Distributed Cache server event. Time spent in shutdownAllConnections waiting for active connections to finish (or be forcefully closed) during server shutdownDistrCacheServerSkipped
Distributed Cache server event. The number of times distributed cache server was skipped because of previous failed connection attemptsDistrCacheServerStartRequestPackets
Distributed Cache server event. Number of StartRequest packets in DistributedCacheServerDistrCacheServerSwitches
Distributed Cache read buffer event. Number of server switches between distributed cache servers in read/write-through cacheDistrCacheServerUpdates
Distributed Cache event. The number of server switches during read or write because the hash-chosen server changed (e.g. due to server deregistration)DistrCacheSkippedWrites
Distributed Cache write buffer event. Number of write buffers that skipped the write-through cache write because no background write slot was free (the limit set by thedistributed_cache_write_pool_size server setting was reached) or a background write could not be started. The data is still written to object storage
DistrCacheStaleReconnections
Distributed Cache connection event. The number of reconnections due to stale (peer-closed) pooled connectionsDistrCacheStartRangeMicroseconds
Distributed Cache read buffer event. Time spent to start a new read range with distributed cacheDistrCacheSuccessfulConnectAttempts
Distributed Cache connection event. The number of successful connection attempts to distributed cacheDistrCacheSuccessfulRegistryUpdates
Distributed Cache registry event. The number of successful server registry updatesDistrCacheSystemTablesErrors
Distributed Cache client event. Number of distributed cache errors during introspection system table requests (system.distributed_cache, system.distributed_cache_metrics, system.distributed_cache_events). These requests are not retried, so retriable and non-retriable errors are counted togetherDistrCacheTemporaryFilesBytesWritten
Distributed Cache connection event. Number of bytes written to temporary files created in distributed cacheDistrCacheTemporaryFilesCreated
Distributed Cache connection event. Number of temporary files created in distributed cacheDistrCacheUnsuccessfulConnectAttempts
Distributed Cache connection event. The number of unsuccessful connection attempts to distributed cacheDistrCacheUnsuccessfulRegistryUpdates
Distributed Cache registry event. The number of unsuccessful server registry updatesDistrCacheUnusedDataPacketsBytes
Distributed Cache client event. The number of bytes in Data packets which were ignoredDistrCacheUnusedDataPacketsBytesReadRangeIdChanged
Distributed Cache client event. The number of bytes in ignored Data packets (part of DistrCacheUnusedDataPacketsBytes) that arrived for the tracked request but for an older read range id (a newer read range was requested on the same request)DistrCacheUnusedDataPacketsBytesUnknownRequest
Distributed Cache client event. The number of bytes in ignored Data packets (part of DistrCacheUnusedDataPacketsBytes) that arrived for a request the client no longer tracks (e.g. reading was stopped via EndRequest after the server had already sent more data)DistrCacheUnusedPackets
Distributed Cache client event. Number of skipped unused packets from distributed cacheDistrCacheUnusedPacketsBufferAllocations
Distributed Cache client event. The number of extra buffer allocations in case we could not reuse existing bufferDistrCacheWriteBytes
Distributed Cache write buffer event. Number of bytes written to distributed cacheDistrCacheWriteErrors
Distributed Cache client event. Total number of distributed cache errors during write requests (equals DistrCacheWriteErrorsRetriable + DistrCacheWriteErrorsNonRetriable). A server-initiated write-through cacheStop is flow control, not an error, and is not counted
DistrCacheWriteErrorsNonRetriable
Distributed Cache client event. Number of non-retriable distributed cache errors during write requestsDistrCacheWriteErrorsRetriable
Distributed Cache client event. Number of retriable distributed cache errors during write requests (counted on each occurrence, including the ones recovered by a retry)DistrCacheWriteMicroseconds
Distributed Cache write buffer event. Time spent in WriteBufferFromDistributedCache::writeToFileSegmentDistrCacheWriteRequestRetries
Distributed Cache write buffer event. The number of write request retries after a retriable errorDistrCacheWriteRequestRetriesExhausted
Distributed Cache write buffer event. Number of times a retriable write error did not succeed within the write request retries limitDistrCacheWriteThrottlerBytes
Bytes passed through ‘max_distributed_cache_write_bandwidth_for_server’ throttler.DistrCacheWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_distributed_cache_write_bandwidth_for_server’ throttling.DistributedAsyncInsertionFailures
Number of failures for asynchronous insertion into a Distributed table (with ‘distributed_foreground_insert’ = 0)DistributedConnectionConnectCount
Number of connects to other servers done during distributed query execution. Happens when new connection is established instead of using existing from pool.DistributedConnectionFailAtAll
Total count when distributed connection fails after all retries finished.DistributedConnectionFailTry
Total count when distributed connection fails with retry.DistributedConnectionMissingTable
Number of times we rejected a replica from a distributed query, because it did not contain a table needed for the query.DistributedConnectionReconnectCount
Number of reconnects to other servers done during distributed query execution. It can happen when a stale connection has been acquired from connection poolDistributedConnectionSkipReadOnlyReplica
Number of replicas skipped during INSERT into Distributed table due to replicas being read-onlyDistributedConnectionStaleReplica
Number of times we rejected a replica from a distributed query, because some table needed for a query had replication lag higher than the configured threshold.DistributedConnectionTries
Total count of distributed connection attempts.DistributedConnectionUsable
Total count of successful distributed connections to a usable server (with required table, but maybe stale).DistributedDelayedInserts
Number of times the INSERT of a block to a Distributed table was throttled due to high number of pending bytes.DistributedDelayedInsertsMilliseconds
Total number of milliseconds spent while the INSERT of a block to a Distributed table was throttled due to high number of pending bytes.DistributedIndexAnalysisMicroseconds
Total time spent during distributed index analysisDistributedIndexAnalysisMissingParts
Number of missing parts during distributed index analysis that will be resolved locallyDistributedIndexAnalysisParts
Number of parts send for distributed index analysisDistributedIndexAnalysisReplicaFallback
Number of times distributed index analysis failed on one of replicas with fallback to local replicaDistributedIndexAnalysisReplicaUnavailable
Number of times distributed index analysis failed on one of replicas without fallback (failed during connect)DistributedIndexAnalysisScheduledReplicas
Number of replicas (local replica will be accounted once) to which distributed index analysis has been scheduledDistributedPlanHostsUsed
Number of distinct hosts that were assigned at least one task when executing a query with make_distributed_plan.DistributedPlanLocalExecution
Set to 1 when a make_distributed_plan query was executed in-process via the local executor (distributed_plan_execute_locally) instead of being dispatched to remote workers.DistributedPlanRemoteTasks
Number of tasks dispatched to remote workers when executing a query with make_distributed_plan. A non-zero value means the query was actually executed distributedly.DistributedPlanWorkerPartsPruned
Number of the assigned data parts the worker’s own index analysis pruned, so the worker does not read them. The coordinator selects parts without that analysis, so it can assign a part no row of the query can match.DistributedPlanWorkerPartsReceived is the sum of this and DistributedPlanWorkerPartsScanned.
DistributedPlanWorkerPartsReceived
Number of data parts the coordinator assigned to a worker of a distributed query plan, summed over the bucketed reads a query runs on that worker. A part assigned to several buckets counts once per read.DistributedPlanWorkerPartsScanned
Number of the assigned data parts the worker keeps to read.DistributedRejectedInserts
Number of times the INSERT of a block to a Distributed table was rejected with ‘Too many bytes’ exception due to high number of pending bytes.DistributedShardsSkipped
Number of shards silently skipped in a query to a Distributed table (or the remote/cluster table functions) whenskip_unavailable_shards is enabled, either because the shard was unavailable or because it raised an exception covered by skip_unavailable_shards_mode.
DistributedSyncInsertionTimeoutExceeded
A timeout has exceeded while waiting for shards during synchronous insertion into a Distributed table (with ‘distributed_foreground_insert’ = 1)DNSError
Total count of errors in DNS resolutionDuplicatedAsyncInserts
Number of async inserts in the INSERTed block to a ReplicatedMergeTree table was deduplicated.DuplicatedInsertedBlocks
Number of the synchronous inserts to a *MergeTree table was deduplicated.DuplicationDataHashComputations
Number of column-wise data-hash computations performed while deduplicating INSERTed blocks to *MergeTree tables (cache misses in the unified deduplication hash).DuplicationElapsedMicroseconds
Total time spent checking for duplication of INSERTed blocks to *MergeTree tables.EncryptionHeaderCacheHits
Number of times encryption header bytes were found in the encryption header cache, so the source read of the header was skipped.EncryptionHeaderCacheMisses
Number of times encryption header bytes were not found in the encryption header cache, so they were read from the source.EngineFileLikeReadFiles
Number of files read in table engines working with files (like File/S3/URL/HDFS).ExecutableUserDefinedFunctionElapsedMicroseconds
Wall clock time spent executing executable and executable_pool user-defined functions, in microseconds.ExecutableUserDefinedFunctionInputBytes
Total bytes written to the stdin of executable and executable_pool user-defined function child processes.ExecutableUserDefinedFunctionInvocations
Number of executable and executable_pool user-defined function invocations.ExecutableUserDefinedFunctionOutputBytes
Total bytes read from the stdout of executable and executable_pool user-defined function child processes.ExecutableUserDefinedFunctionPeakMemoryByteSeconds
Per-process peak memory used by executable and executable_pool user-defined function child processes integrated over wall clock time, in byte-seconds.ExecutableUserDefinedFunctionPoolWaitMicroseconds
Time spent waiting on tryBorrowObject when the executable_pool user-defined function pool is exhausted, in microseconds.ExecutableUserDefinedFunctionSystemTimeMicroseconds
Kernel mode CPU time consumed by executable and executable_pool user-defined function child processes, in microseconds.ExecutableUserDefinedFunctionUserTimeMicroseconds
User mode CPU time consumed by executable and executable_pool user-defined function child processes, in microseconds.ExecuteShellCommand
Number of shell command executions.ExternalAggregationCompressedBytes
Number of bytes written to disk for aggregation in external memory.ExternalAggregationMerge
Number of times temporary files were merged for aggregation in external memory.ExternalAggregationUncompressedBytes
Amount of data (uncompressed, before compression) written to disk for aggregation in external memory.ExternalAggregationWritePart
Number of times a temporary file was written to disk for aggregation in external memory.ExternalDataSourceLocalCacheReadBytes
Bytes read from local cache buffer in RemoteReadBufferCacheExternalDistinctCompressedBytes
Number of compressed bytes written for DISTINCT in external memory.ExternalDistinctMerge
Number of times temporary files were merged for DISTINCT in external memory.ExternalDistinctUncompressedBytes
Amount of data (uncompressed, before compression) written for DISTINCT in external memory.ExternalDistinctWritePart
Number of times a temporary file was written to disk for DISTINCT in external memory.ExternalJoinCompressedBytes
Number of compressed bytes written for JOIN in external memory.ExternalJoinMerge
Number of times temporary files were merged for JOIN in external memory.ExternalJoinUncompressedBytes
Amount of data (uncompressed, before compression) written for JOIN in external memory.ExternalJoinWritePart
Number of times a temporary file was written to disk for JOIN in external memory.ExternalProcessingCompressedBytesTotal
Number of compressed bytes written by external processing (sorting/aggregating/joining)ExternalProcessingFilesTotal
Number of files used by external processing (sorting/aggregating/joining)ExternalProcessingUncompressedBytesTotal
Amount of data (uncompressed, before compression) written by external processing (sorting/aggregating/joining)ExternalSortCompressedBytes
Number of compressed bytes written for sorting in external memory.ExternalSortMerge
Number of times temporary files were merged for sorting in external memory.ExternalSortUncompressedBytes
Amount of data (uncompressed, before compression) written for sorting in external memory.ExternalSortWritePart
Number of times a temporary file was written to disk for sorting in external memory.FailedAsyncInsertQuery
Number of failed ASYNC INSERT queries.FailedInitialQuery
Number of failed initial queries.FailedInitialSelectQuery
Same as FailedInitialQuery, but only for SELECT queries.FailedInsertQuery
Same as FailedQuery, but only for INSERT queries.FailedInternalInsertQuery
Same as FailedInternalQuery, but only for INSERT queries.FailedInternalQuery
Number of failed internal queries.FailedInternalSelectQuery
Same as FailedInternalQuery, but only for SELECT queries.FailedQuery
Number of total failed queries, both internal and user queries.FailedSelectQuery
Same as FailedQuery, but only for SELECT queries.FetchBackgroundExecutorTaskCancelMicroseconds
Time spent in cancel() for Fetch executor tasks.FetchBackgroundExecutorTaskExecuteStepMicroseconds
Time spent in executeStep() for Fetch executor tasks.FetchBackgroundExecutorTaskResetMicroseconds
Time spent resetting task for Fetch executor.FetchBackgroundExecutorWaitMicroseconds
Time spent waiting for completion in Fetch executor.FiberStackAllocBytes
Number of bytes allocated for fiber stacks, including guard pages (guard pages are only used in debug and sanitizer builds).FiberStackAllocNanoseconds
Amount of time spent allocating fiber stacks. Nanoseconds are used because a single allocation is normally faster than a microsecond, unless the allocator or the kernel is contended.FiberStackAllocs
Number of stacks allocated for fibers (fibers are used for asynchronous communication with remote replicas).FiberStackFreeNanoseconds
Amount of time spent deallocating fiber stacks.FileOpen
Number of files opened.FileProgressCallbackInvocations
Number of times the per-queryFileProgressCallback was invoked (counts each per-file FileProgress event delivered to the client over native TCP or via clickhouse-local).
FileSegmentCompleteMicroseconds
Duration of FileSegment::complete() in filesystem cacheFileSegmentFailToIncreasePriority
Number of times the priority was not increased due to a high contention on the cache lockFileSegmentHolderCompleteMicroseconds
File segments holder complete() timeFileSegmentIncreasePriorityMicroseconds
File segment increase priority timeFileSegmentLockMicroseconds
Lock file segment timeFileSegmentRemoveMicroseconds
File segment remove() timeFileSegmentWaitMicroseconds
Wait on DOWNLOADING stateFileSegmentWaitTimeouts
Number of times waiting on a DOWNLOADING file segment timed out (seefilesystem_cache_wait_for_concurrent_download_timeout_milliseconds)
FileSegmentWriteMicroseconds
File segment write() timeFileSync
Number of times the F_FULLFSYNC/fsync/fdatasync function was called for files.FileSyncElapsedMicroseconds
Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for files.FilesystemCacheBackgroundDownloadQueuePush
Number of file segments sent for background download in filesystem cacheFilesystemCacheBackgroundEvictedBytes
Number of bytes evicted by background threadFilesystemCacheBackgroundEvictedFileSegments
Number of file segments evicted by background threadFilesystemCacheBackgroundRemovedInvalidatedEntries
Number of invalidated (lazily-removed) priority queue entries purged by the background cleanup threadFilesystemCacheCheckCorrectness
Number of times FileCache::assertCacheCorrectness was calledFilesystemCacheCheckCorrectnessMicroseconds
How much time does FileCache::assertCacheCorrectness takesFilesystemCacheClientsMapLockWaitMicroseconds
Time spent waiting on a contended per-client (user_id) usage map shard lock in the overcommit cache policy
FilesystemCacheCreatedKeyDirectories
Number of created key directoriesFilesystemCacheDowngradedFileSegments
Number of file segments downgraded (moved) from the protected to the probationary queue in SLRU cache policy. This is an internal move within the cache, not an eviction.FilesystemCacheEvictedBytes
Number of bytes evicted from filesystem cacheFilesystemCacheEvictedFileSegments
Number of file segments evicted from filesystem cacheFilesystemCacheEvictionReusedIterator
Number of filesystem cache iterator reusingFilesystemCacheEvictionSkippedEvictingFileSegments
Number of file segments skipped for eviction because of being in evicting stateFilesystemCacheEvictionSkippedFileSegments
Number of file segments skipped for eviction because of being in unreleasable stateFilesystemCacheEvictionSkippedMovingFileSegments
Number of file segments skipped for eviction because of being in moving stateFilesystemCacheEvictionTries
Number of filesystem cache eviction attemptsFilesystemCacheEvictMicroseconds
Filesystem cache eviction timeFilesystemCacheFailedEvictionCandidates
Number of file segments which unexpectedly failed to be evicted during dynamic filesystem cache evictionFilesystemCacheFailedReserveAttempts
Number of failed file cache space reservation attempts.FilesystemCacheFailToReserveSpaceBecauseOfCacheResize
Number of times space reservation was skipped due to the cache is being resizedFilesystemCacheFailToReserveSpaceBecauseOfLockContention
Number of times space reservation was skipped due to a high contention on the cache lockFilesystemCacheFreeSpaceKeepingThreadErrors
Number of exceptions caught by the background free space keeping job (collection, removal or finalization)FilesystemCacheFreeSpaceKeepingThreadRun
Number of times background thread executed free space keeping jobFilesystemCacheFreeSpaceKeepingThreadWorkMilliseconds
Time for which background thread executed free space keeping jobFilesystemCacheGetMicroseconds
Filesystem cache get() timeFilesystemCacheGetOrSetMicroseconds
Filesystem cache getOrSet() timeFilesystemCacheHoldFileSegments
Filesystem cache file segments count, which were holdFilesystemCacheIdleClientEvictions
Number of times all cache entries of an idle distributed-cache client (user_id) were purged because no access happened within idle_client_ttl_sec.
FilesystemCacheInvalidatedEntriesCleanupThreadWorkMilliseconds
Time for which the background thread executed the invalidated priority queue entries cleanup jobFilesystemCacheLoadMetadataMicroseconds
Time spent loading filesystem cache metadataFilesystemCacheLockKeyMicroseconds
Lock cache key timeFilesystemCacheLockMetadataMicroseconds
Lock filesystem cache metadata timeFilesystemCacheLockOriginPoolMicroseconds
Lock filesystem cache shared origin pool timeFilesystemCacheOvercommitCandidatesIterationSteps
Number of outer iterations the overcommit eviction loop ran beyond its first pass to satisfy a reservation’s deficitFilesystemCachePriorityReadLockMicroseconds
Lock filesystem cache time for read in priority queueFilesystemCachePriorityWriteLockMicroseconds
Lock filesystem cache time for write to priority queueFilesystemCacheReserveAttempts
Filesystem cache space reservation attemptFilesystemCacheReserveMicroseconds
Filesystem cache space reservation timeFilesystemCacheStateLockMicroseconds
Lock filesystem cache time for state lockFilesystemCacheUnusedHoldFileSegments
Filesystem cache file segments count, which were hold, but not used (because of seek or LIMIT n, etc)FilteringMarksWithPrimaryKeyMicroseconds
Time spent filtering parts by PK.FilteringMarksWithPrimaryKeyProcessedMarks
Total marks processed during PK analysis.FilteringMarksWithSecondaryKeysMicroseconds
Time spent filtering parts by skip indexes.FilterPartsByVirtualColumnsMicroseconds
Total time spent in filterPartsByVirtualColumns function.FilterTransformPassedBytes
Number of bytes that passed the filter in the queryFilterTransformPassedRows
Number of rows that passed the filter in the queryFunctionExecute
Number of SQL ordinary function calls (SQL functions are called on per-block basis, so this number represents the number of blocks).GatheredColumns
Number of columns gathered during the vertical stage of merges.GatheringColumnMilliseconds
Total time spent while gathering columns for vertical mergeGlobalMemoryLimitExceeded
Number of times the global memory limit was exceeded.GlobalThreadPoolExpansions
Counts the total number of times new threads have been added to the global thread pool. This metric indicates the frequency of expansions in the global thread pool to accommodate increased processing demands.GlobalThreadPoolJobs
Counts the number of jobs that have been pushed to the global thread pool.GlobalThreadPoolJobWaitTimeMicroseconds
Measures the elapsed time from when a job is scheduled in the thread pool to when it is picked up for execution by a worker thread. This metric helps identify delays in job processing, indicating the responsiveness of the thread pool to new tasks.GlobalThreadPoolLockWaitMicroseconds
Total time threads have spent waiting for locks in the global thread pool.GlobalThreadPoolShrinks
Counts the total number of times the global thread pool has shrunk by removing threads. This occurs when the number of idle threads exceeds max_thread_pool_free_size, indicating adjustments in the global thread pool size in response to decreased thread utilization.GlobalThreadPoolThreadCreationMicroseconds
Total time spent waiting for new threads to start.HardPageFaults
The number of hard page faults in query execution threads. High values indicate either that you forgot to turn off swap on your server, or eviction of memory pages of the ClickHouse binary during very high memory pressure, or successful usage of the ‘mmap’ read method for the tables data.HashJoinPreallocatedElementsInHashTables
How many elements were preallocated in hash tables for hash join.HedgedRequestsChangeReplica
Total count when timeout for changing replica expired in hedged requests.HTTPConnectionsCreated
Number of created client HTTP connectionsHTTPConnectionsElapsedMicroseconds
Total time spend on creating client HTTP connectionsHTTPConnectionsErrors
Number of cases when creation of a client HTTP connection failedHTTPConnectionsExpired
Number of expired client HTTP connectionsHTTPConnectionsPreserved
Number of preserved client HTTP connectionsHTTPConnectionsReset
Number of reset client HTTP connectionsHTTPConnectionsReused
Number of reused client HTTP connectionsHTTPServerConnectionsClosed
Number of closed server HTTP connections. Keep alive has not been negotiatedHTTPServerConnectionsCreated
Number of created server HTTP connectionsHTTPServerConnectionsExpired
Number of expired server HTTP connections.HTTPServerConnectionsPreserved
Number of preserved server HTTP connections. Connection kept alive successfullyHTTPServerConnectionsReset
Number of reset server HTTP connections. Server closes connectionHTTPServerConnectionsReused
Number of reused server HTTP connectionsIcebergIteratorInitializationMicroseconds
Total time spent on synchronous initialization of iceberg data iterators.IcebergMetadataFilesCacheHits
Number of times iceberg metadata files have been found in the cache.IcebergMetadataFilesCacheMisses
Number of times iceberg metadata files have not been found in the iceberg metadata cache and had to be read from (remote) disk.IcebergMetadataFilesCacheStaleMisses
Number of times iceberg metadata files have been found in the cache, but were considered stale and had to be read from (remote) disk.IcebergMetadataFilesCacheWeightLost
Approximate number of bytes evicted from the iceberg metadata cache.IcebergMetadataReadWaitTimeMicroseconds
Total time data readers spend waiting for iceberg metadata files to be read and parsed, summed across all reader threads.IcebergMetadataReturnedObjectInfos
Total number of returned object infos from iceberg iterator.IcebergMetadataUpdateMicroseconds
Total time spent updating iceberg storage metadata from state (buildStorageMetadataFromState).IcebergMinMaxIndexPrunedFiles
Number of skipped files by using MinMax index in IcebergIcebergMinMaxNonPrunedDeleteFiles
Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.IcebergMinMaxPrunedDeleteFiles
Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.IcebergPartitionPrunedFiles
Number of skipped files during Iceberg partition pruningIcebergPartitionPrunedManifestFiles
Number of Iceberg manifest files skipped without being read, using the partition summaries of the manifest listIcebergTrivialCountOptimizationApplied
Trivial count optimization applied while reading from IcebergIcebergVersionHintUsed
Number of times version-hint.text has been used.IgnoredColdParts
See setting ignore_cold_parts_seconds. Number of times read queries ignored very new parts that weren’t pulled into cache by CacheWarmer yet.IndexAnalysisRounds
Number of times index analysis was performed within the query.IndexBinarySearchAlgorithm
Number of times the binary search algorithm is used over the index marksIndexGenericExclusionSearchAlgorithm
Number of times the generic exclusion search algorithm is used over the index marksIndexGenericExclusionSearchStepLimitReached
Number of times the generic exclusion search over the index marks reached merge_tree_generic_exclusion_search_max_steps and accepted the remaining mark ranges without further splittingInitialQuery
Same as Query, but only counts initial queries (see is_initial_query).InitialSelectQuery
Same as InitialQuery, but only for SELECT queries.InsertedBytes
Number of bytes (uncompressed; for columns as they stored in memory) INSERTed to all tables.InsertedCompactParts
Number of parts inserted in Compact format.InsertedRows
Number of rows INSERTed to all tables.InsertedWideParts
Number of parts inserted in Wide format.InsertQueriesWithSubqueries
Count INSERT queries with all subqueriesInsertQuery
Same as Query, but only for INSERT queries.InsertQueryTimeMicroseconds
Total time of INSERT queries.InterfaceHTTPReceiveBytes
Number of bytes received through HTTP interfacesInterfaceHTTPSendBytes
Number of bytes sent through HTTP interfacesInterfaceInterserverReceiveBytes
Number of bytes received through interserver interfacesInterfaceInterserverSendBytes
Number of bytes sent through interserver interfacesInterfaceMySQLReceiveBytes
Number of bytes received through MySQL interfacesInterfaceMySQLSendBytes
Number of bytes sent through MySQL interfacesInterfaceNativeReceiveBytes
Number of bytes received through native interfacesInterfaceNativeSendBytes
Number of bytes sent through native interfacesInterfacePostgreSQLReceiveBytes
Number of bytes received through PostgreSQL interfacesInterfacePostgreSQLSendBytes
Number of bytes sent through PostgreSQL interfacesInterfacePrometheusReceiveBytes
Number of bytes received through Prometheus interfacesInterfacePrometheusSendBytes
Number of bytes sent through Prometheus interfacesIOBufferAllocBytes
Number of bytes allocated for IO buffers (for ReadBuffer/WriteBuffer).IOBufferAllocs
Number of allocations of IO buffers (for ReadBuffer/WriteBuffer).IOUringCQEsCompleted
Total number of successfully completed io_uring CQEsIOUringCQEsFailed
Total number of completed io_uring CQEs with failuresIOUringSQEsResubmitsAsync
Total number of asynchronous io_uring SQE resubmits performedIOUringSQEsResubmitsSync
Total number of synchronous io_uring SQE resubmits performedIOUringSQEsSubmitted
Total number of io_uring SQEs submittedJemallocFailedAllocationSampleTracking
Total number of times tracking of jemalloc allocation sample failedJemallocFailedDeallocationSampleTracking
Total number of times tracking of jemalloc deallocation sample failedJoinBuildPostProcessingMicroseconds
Elapsed time of post-processing steps after building the right JOIN side.JoinBuildRowStoreMicroseconds
Elapsed time transforming the right JOIN side payload into row-major format.JoinBuildTableRowCount
Total number of rows in the build table for a JOIN operation.JoinDelayedJoinedTransformBlockCount
Number of blocks emitted by DelayedJoinedBlocksWorkerTransform.JoinDelayedJoinedTransformRowCount
Number of rows emitted by DelayedJoinedBlocksWorkerTransform.JoinNonJoinedTransformBlockCount
Number of blocks emitted by NonJoinedBlocksTransform.JoinNonJoinedTransformRowCount
Number of non-joined rows emitted by NonJoinedBlocksTransform.JoinOptimizeMicroseconds
Total time spent executing JOIN plan optimizations.JoinProbeTableRowCount
Total number of rows in the probe table for a JOIN operation.JoinReorderMicroseconds
Total time spent executing JOIN reordering algorithm.JoinResultRowCount
Total number of rows in the result of a JOIN operation.JoinSpillingHashJoinSwitchedToGraceJoin
Number of times a (Concurrent)HashJoin was switched to GraceHashJoin due to memory limit in SpillingHashJoin.KafkaBackgroundReads
Number of background reads populating materialized views from Kafka since server startKafkaCommitFailures
Number of failed commits of consumed offsets to Kafka (usually is a sign of some data duplication)KafkaCommits
Number of successful commits of consumed offsets to Kafka (normally should be the same as KafkaBackgroundReads)KafkaConsumerErrors
Number of errors reported by librdkafka during pollsKafkaDirectReads
Number of direct selects from Kafka tables since server startKafkaMessagesFailed
Number of Kafka messages ClickHouse failed to parseKafkaMessagesPolled
Number of Kafka messages polled from librdkafka to ClickHouseKafkaMessagesProduced
Number of messages produced to KafkaKafkaMessagesRead
Number of Kafka messages already processed by ClickHouseKafkaMVNotReady
Number of failed attempts to stream data to a materialized view that is not readyKafkaProducerErrors
Number of errors during producing the messages to KafkaKafkaProducerFlushes
Number of explicit flushes to Kafka producerKafkaRebalanceAssignments
Number of partition assignments (the final stage of consumer group rebalance)KafkaRebalanceErrors
Number of failed consumer group rebalancesKafkaRebalanceRevocations
Number of partition revocations (the first stage of consumer group rebalance)KafkaRowsRead
Number of rows parsed from Kafka messagesKafkaRowsRejected
Number of parsed rows which were later rejected (due to rebalances / errors or similar reasons). Those rows will be consumed again after the rebalance.KafkaRowsWritten
Number of rows inserted into Kafka tablesKafkaWrites
Number of writes (inserts) to Kafka tablesKeeperAddWatchRequest
Number of add watches requestsKeeperBatchMaxCount
Number of batches that hit max_requests_batch_size limitKeeperBatchMaxReadCount
Number of batches that hit max_read_batch_size limitKeeperBatchMaxReadTotalSize
Number of batches that hit max_read_batch_bytes_size limitKeeperBatchMaxTotalSize
Number of batches that hit max_requests_batch_bytes_size limitKeeperChangelogFileSyncMicroseconds
Time spent in fsync for Keeper changelog (uncompressed logs only)KeeperChangelogLockWaitMicroseconds
Time spent waiting for acquiring Keeper changelog lockKeeperChangelogStartupReadBytes
Number of physical bytes (checksum + header + blob) read from changelog files by the parallel Keeper startup readerKeeperChangelogStartupReadEntries
Number of changelog records validated by the parallel Keeper startup readerKeeperChangelogStartupReadMicroseconds
Wall time spent in the parallel changelog read during Keeper startupKeeperChangelogStartupStitchMicroseconds
Wall time spent stitching per-file parallel startup read results into LogEntryStorage stateKeeperChangelogWrittenBytes
Number of bytes written to the changelog in KeeperKeeperCheckRequest
Number of check requestsKeeperCheckWatchRequest
Number of remove watches requestsKeeperCommits
Number of successful commitsKeeperCommitsFailed
Number of failed commitsKeeperCommitWaitElapsedMicroseconds
Time spent waiting for certain log to be committedKeeperCreateRequest
Number of create requestsKeeperExistsRequest
Number of exists requestsKeeperGetRequest
Number of get requestsKeeperLatency
Keeper latencyKeeperListRecursiveRequest
Number of get children recursive requestsKeeperListRequest
Number of list requestsKeeperLiveSessionsLockWaitMicroseconds
Time spent waiting to acquire Keeper live sessions lockKeeperLogsEntryReadFromCommitReadAhead
Number of log entries served to the commit thread from the commit read-ahead reader’s decoded window (fast-path pop or first-of-window drain pop)KeeperLogsEntryReadFromFile
Number of log entries in Keeper being read directly from the changelog fileKeeperLogsEntryReadFromLatestCache
Number of log entries in Keeper being read from latest logs cacheKeeperLogsReadAheadCursorsInstalled
Number of new fill cursors queued onto a Keeper changelog read-ahead reader (peer or commit) that the fill task was not already coveringKeeperLogsReadAheadFillDecodedEntries
Number of log entries decoded by the Keeper read-ahead fill taskKeeperLogsReadAheadFillReopens
Number of times the Keeper read-ahead fill reopened or seeked a changelog fileKeeperLogsReadAheadPlanEpochMismatches
Number of times a Keeper changelog read plan was discarded because a concurrent write_at truncation invalidated it before it could be servedKeeperLogsReadAheadReadersCreated
Number of Keeper changelog read-ahead readers (and their fill tasks) created; staying flat across file boundaries indicates a reader is being reused rather than torn down and recreatedKeeperLogsReadAheadScheduleRejected
Number of times a Keeper changelog read-ahead reader could not be created because the read-ahead thread pool queue was full; the caller falls back to a direct readKeeperLogsReadAheadTimeoutFallbacks
Number of times a Keeper changelog read-ahead serve request timed out waiting for the fill task and fell back to a direct read of the remaining tailKeeperLSMTCommittedEntryBytes
Serialized bytes of entries appended to the Keeper LSMT committed memtableKeeperLSMTCommittedMemtablesCreated
Number of committed memtables created by Keeper LSMTKeeperLSMTFileDeletionExceptions
Number of exceptions in Keeper LSMT file deletionsKeeperLSMTFlushes
Number of Keeper LSMT memtable flushes finished successfullyKeeperLSMTFlushExceptions
Number of exceptions in Keeper LSMT memtable flushesKeeperLSMTFlushWrittenCompressedBytes
Compressed bytes written to files by Keeper LSMT memtable flushesKeeperLSMTFlushWrittenUncompressedBytes
Uncompressed bytes written to files by Keeper LSMT memtable flushesKeeperLSMTGetBlockFromCache
Number of Keeper LSMT block accesses served by the block cacheKeeperLSMTGetBlockFromWeakPtr
Number of Keeper LSMT block accesses served by the weak pointer in the file’s block infoKeeperLSMTGetBlockLoadedGroup
Number of Keeper LSMT block accesses that had to load a block group from diskKeeperLSMTGetCommittedNodeFromMemory
Number of Keeper LSMT committed node lookups that found the node’s block already in memoryKeeperLSMTGetCommittedNodeLoadedBlock
Number of Keeper LSMT committed node lookups that had to load the node’s blockKeeperLSMTGetCommittedNodeNonexistent
Number of Keeper LSMT committed node lookups for nodes that don’t existKeeperLSMTGetUncommittedNodeHits
Number of Keeper LSMT node lookups that found the node in uncommitted memtablesKeeperLSMTGetUncommittedNodeMisses
Number of Keeper LSMT node lookups that didn’t find the node in uncommitted memtables and fell through to committed stateKeeperLSMTListFilterFalsePositives
Number of file scans not skipped by the Keeper LSMT parent paths filter that found no childrenKeeperLSMTListFilterSkipped
Number of file scans skipped by Keeper LSMT list operations because the parent paths filter ruled out any childrenKeeperLSMTListFilterTruePositives
Number of file scans not skipped by the Keeper LSMT parent paths filter that found childrenKeeperLSMTListNamesFromFiles
Child names (including tombstones) obtained from files by Keeper LSMT list operationsKeeperLSMTListNamesFromMemtables
Child names (including tombstones) obtained from memtables by Keeper LSMT list operationsKeeperLSMTListScannedBlocks
Blocks scanned by Keeper LSMT list operations in filesKeeperLSMTListScannedEntries
Entries scanned by Keeper LSMT list operations in filesKeeperLSMTLoadedBlocks
Number of blocks loaded from disk by Keeper LSMTKeeperLSMTLoadedUncompressedBytes
Uncompressed bytes of blocks loaded from disk by Keeper LSMTKeeperLSMTMergeConsumedFiles
Number of input files fully consumed and dropped by Keeper LSMT mergesKeeperLSMTMergeConsumedUncompressedBytes
Uncompressed bytes in input files fully consumed and dropped by Keeper LSMT mergesKeeperLSMTMergeExceptions
Number of exceptions in Keeper LSMT mergesKeeperLSMTMerges
Number of Keeper LSMT merges finished successfullyKeeperLSMTMergeWrittenCompressedBytes
Compressed bytes in output files published by Keeper LSMT mergesKeeperLSMTMergeWrittenFiles
Number of output files published by Keeper LSMT mergesKeeperLSMTMergeWrittenUncompressedBytes
Uncompressed bytes in output files published by Keeper LSMT mergesKeeperLSMTNodeCacheEntriesUpdated
Number of Keeper LSMT node cache entries re-pointed at a freshly loaded blockKeeperLSMTThrottledWrites
Number of writes delayed because Keeper LSMT background flushes or merges fell behindKeeperLSMTUncommittedCreateBytes
Serialized bytes of Create entries appended to Keeper LSMT uncommitted memtablesKeeperLSMTUncommittedCreates
Number of Create entries appended to Keeper LSMT uncommitted memtablesKeeperLSMTUncommittedMemtablesCreated
Number of uncommitted memtables created by Keeper LSMTKeeperLSMTUncommittedRemoveBytes
Serialized bytes of Remove entries appended to Keeper LSMT uncommitted memtablesKeeperLSMTUncommittedRemoves
Number of Remove entries appended to Keeper LSMT uncommitted memtablesKeeperLSMTUncommittedUpdateBytes
Serialized bytes of Update entries appended to Keeper LSMT uncommitted memtablesKeeperLSMTUncommittedUpdates
Number of Update entries appended to Keeper LSMT uncommitted memtablesKeeperMultiReadRequest
Number of multi read requestsKeeperMultiRequest
Number of multi requestsKeeperPacketsReceived
Packets received by keeper serverKeeperPacketsSent
Packets sent by keeper serverKeeperPreprocessElapsedMicroseconds
Keeper preprocessing latency for a single requestKeeperProcessAndResponsesLockWaitMicroseconds
Time spent waiting for acquiring Keeper process and responses lockKeeperProcessElapsedMicroseconds
Keeper commit latency for a single requestKeeperReadBatchCount
Number of read request batches processed by KeeperKeeperReadBatchTotalRequests
Total number of read requests processed in batches by KeeperKeeperReadRequestQueueLockWaitMicroseconds
Time spent waiting for acquiring Keeper read request queue lockKeeperReadSnapshot
Number of completed snapshot readsKeeperReadSnapshotDeferred
Number of snapshot reads deferred because the requested snapshot is no longer available locally; NuRaft will retry the latest snapshot.KeeperReadSnapshotFailed
Number of failed snapshot readsKeeperReadSnapshotObject
Number of snapshot objects sent to followersKeeperReconfigRequest
Number of reconfig requestsKeeperRemoveRequest
Number of remove requestsKeeperRemoveWatchRequest
Number of remove watches requestsKeeperRequestRejectedDueToSoftMemoryLimitCount
Number requests that have been rejected due to soft memory limit exceededKeeperRequestTotal
Total requests number on keeper serverKeeperRequestTotalWithSubrequests
Total requests number on keeper server, counting each subrequest within a multi requestKeeperSaveSnapshot
Number of snapshot saveKeeperSaveSnapshotFailed
Number of failed snapshot savesKeeperSaveSnapshotObject
Number of snapshot objects received from leaderKeeperServerWriteLockWaitMicroseconds
Time spent waiting for acquiring Keeper server write lockKeeperSessionCallbackLockWaitMicroseconds
Time spent waiting for acquiring Keeper session callback lockKeeperSetRequest
Number of set requestsKeeperSetWatchesRequest
Number of set watches requestsKeeperSnapshotApplys
Number of snapshot applyingKeeperSnapshotApplysFailed
Number of failed snapshot applyingKeeperSnapshotCreations
Number of snapshots creationsKeeperSnapshotCreationsFailed
Number of failed snapshot creationsKeeperSnapshotFileSyncMicroseconds
Time spent in fsync for Keeper snapshot filesKeeperSnapshotRemoteLoaderErrors
Number of remote read errors in RemoteSnapshotLoader while serving a snapshot to a followerKeeperSnapshotWrittenBytes
Number of bytes written to snapshot files in KeeperKeeperStaleRequestsSkipped
Number of Keeper requests skipped because the session is no longer liveKeeperStorageLockWaitMicroseconds
Time spent waiting for acquiring Keeper storage lockKeeperStorageSharedLockWaitMicroseconds
Time spent waiting for acquiring Keeper storage shared lockKeeperTotalElapsedMicroseconds
Keeper total latency for a single requestKeeperTTLRemoveRequestsDropped
Number of TTL remove requests dropped because the queue was fullKeeperTTLRemoveRequestsEnqueued
Number of TTL remove requests successfully enqueuedKeeperWatchesTriggered
Number of watch triggersKeeperWatchTriggeredNodeChildrenChanged
Number of watch triggers by children change operationsKeeperWatchTriggeredNodeCreated
Number of watch triggers byCREATE operation
KeeperWatchTriggeredNodeDataChanged
Number of watch triggers by change operationsKeeperWatchTriggeredNodeDeleted
Number of watch triggers byDELETE operation
KeeperWriteBatchCount
Number of write request batches processed by KeeperKeeperWriteBatchTotalRequests
Total number of write requests processed in batches by KeeperLoadedDataParts
Number of data parts loaded by MergeTree tables during initialization.LoadedDataPartsMicroseconds
Microseconds spent by MergeTree tables for loading data parts during initialization.LoadedMarksCount
Number of marks loaded (total across columns).LoadedMarksFiles
Number of mark files loaded.LoadedMarksMemoryBytes
Size of in-memory representations of loaded marks.LoadedPrimaryIndexBytes
Number of rows of primary key loaded.LoadedPrimaryIndexFiles
Number of primary index files loaded.LoadedPrimaryIndexRows
Number of rows of primary key loaded.LoadedStatisticsMicroseconds
Elapsed time of loading statistics from partsLoadingMarksTasksCanceled
Number of times background tasks for loading marks were canceledLocalReadThrottlerBytes
Bytes passed through ‘max_local_read_bandwidth_for_server’/‘max_local_read_bandwidth’ throttler.LocalReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_local_read_bandwidth_for_server’/‘max_local_read_bandwidth’ throttling.LocalThreadPoolBusyMicroseconds
Total time threads have spent executing the actual work.LocalThreadPoolExpansions
Counts the total number of times threads have been borrowed from the global thread pool to expand local thread pools.LocalThreadPoolJobs
Counts the number of jobs that have been pushed to the local thread pools.LocalThreadPoolJobWaitTimeMicroseconds
Measures the elapsed time from when a job is scheduled in the thread pool to when it is picked up for execution by a worker thread. This metric helps identify delays in job processing, indicating the responsiveness of the thread pool to new tasks.LocalThreadPoolLockWaitMicroseconds
Total time threads have spent waiting for locks in the local thread pools.LocalThreadPoolShrinks
Counts the total number of times threads have been returned to the global thread pool from local thread pools.LocalThreadPoolThreadCreationMicroseconds
Total time local thread pools have spent waiting to borrow a thread from the global pool.LocalWriteThrottlerBytes
Bytes passed through ‘max_local_write_bandwidth_for_server’/‘max_local_write_bandwidth’ throttler.LocalWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_local_write_bandwidth_for_server’/‘max_local_write_bandwidth’ throttling.LogDebug
Number of log messages with level DebugLogError
Number of log messages with level ErrorLogFatal
Number of log messages with level FatalLoggerElapsedNanoseconds
Cumulative time spend in loggingLogInfo
Number of log messages with level InfoLogTest
Number of log messages with level TestLogTrace
Number of log messages with level TraceLogWarning
Number of log messages with level WarningMainConfigLoads
Number of times the main configuration was reloaded.MarkCacheEvictedBytes
Number of bytes evicted from the mark cache.MarkCacheEvictedFiles
Number of mark files evicted from the mark cache.MarkCacheEvictedMarks
Number of marks evicted from the mark cache.MarkCacheHits
Number of times an entry has been found in the mark cache, so we didn’t have to load a mark file.MarkCacheMisses
Number of times an entry has not been found in the mark cache, so we had to load a mark file in memory, which is a costly operation, adding to query latency.MarksTasksFromCache
Number of times marks were loaded synchronously because they were already present in the cache.MemoryAllocatedWithoutCheck
Number of times memory has been allocated without checking for memory constraints.MemoryAllocatedWithoutCheckBytes
Amount of bytes that has been allocated without checking for memory constraints.MemoryAllocatorPurge
Total number of times memory allocator purge was requestedMemoryAllocatorPurgeTimeMicroseconds
Total time spent for memory allocator purgeMemoryOvercommitWaitTimeMicroseconds
Total time spent in waiting for memory to be freed in OvercommitTracker.MemoryReservationAdmitMicroseconds
Total time a query was waiting for initial memory reservation to be admittedMemoryReservationDecreases
Total number of approved memory reservation decrease requestsMemoryReservationFailed
Total number of failed memory reservationsMemoryReservationIncreaseMicroseconds
Total time a query was waiting for memory reservation increase to be approvedMemoryReservationIncreases
Total number of approved memory reservation increase requestsMemoryReservationKilled
Total number of memory reservation kill signalsMemoryWorkerRun
Number of runs done by MemoryWorker in backgroundMemoryWorkerRunElapsedMicroseconds
Total time spent by MemoryWorker for background workMerge
Number of launched background merges.MergeCommitMilliseconds
Total time spent for committing merge results (part renaming, checksum verification, ZooKeeper updates)MergedColumns
Number of columns merged during the horizontal stage of merges.MergedIntoCompactParts
Number of parts merged into Compact format.MergedIntoWideParts
Number of parts merged into Wide format.MergedProjections
Number of projections merged (not rebuilt) during MergeTree merges.MergedRows
Rows read for background merges. This is the number of rows before merge.MergedUncompressedBytes
Uncompressed bytes (for columns as they stored in memory) that was read for background merges. This is the number before merge.MergeExecuteMilliseconds
Total busy time spent for execution of background mergesMergeHorizontalStageExecuteMilliseconds
Total busy time spent for execution of horizontal stage of background mergesMergeHorizontalStageTotalMilliseconds
Total time spent for horizontal stage of background mergesMergeMutateBackgroundExecutorTaskCancelMicroseconds
Time spent in cancel() for MergeMutate executor tasks.MergeMutateBackgroundExecutorTaskExecuteStepMicroseconds
Time spent in executeStep() for MergeMutate executor tasks.MergeMutateBackgroundExecutorTaskResetMicroseconds
Time spent resetting task for MergeMutate executor.MergeMutateBackgroundExecutorWaitMicroseconds
Time spent waiting for completion in MergeMutate executor.MergePrewarmStageExecuteMilliseconds
Total busy time spent for execution of prewarm stage of background mergesMergePrewarmStageTotalMilliseconds
Total time spent for prewarm stage of background mergesMergeProjectionStageExecuteMilliseconds
Total busy time spent for execution of projection stage of background mergesMergeProjectionStageTotalMilliseconds
Total time spent for projection stage of background mergesMergerMutatorPartsInRangesForMergeCount
Amount of candidate parts for mergeMergerMutatorPrepareRangesForMergeElapsedMicroseconds
Time spent to prepare parts ranges which can be merged according to merge predicate.MergerMutatorRangesForMergeCount
Amount of candidate ranges for mergeMergerMutatorSelectPartsForMergeElapsedMicroseconds
Time spent to select parts from ranges which can be merged.MergerMutatorSelectRangePartsCount
Amount of parts in selected range for mergeMergerMutatorsGetPartsForMergeElapsedMicroseconds
Time spent to take data parts snapshot to build ranges from them.MergeSourceParts
Number of source parts scheduled for merges.MergesRejectedByMemoryLimit
Number of background merges rejected due to memory limitMergesThrottlerBytes
Bytes passed through ‘max_merges_bandwidth_for_server’ throttler.MergesThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_merges_bandwidth_for_server’ throttling.MergeTextIndexStageExecuteMilliseconds
Total busy time spent for execution of text index stage of background mergesMergeTextIndexStageTotalMilliseconds
Total time spent for text index stage of background mergesMergeTotalMilliseconds
Total time spent for background mergesMergeTreeAllRangesAnnouncementsSent
The number of announcements sent from the remote server to the initiator server about the set of data parts (for MergeTree tables). Measured on the remote server side.MergeTreeAllRangesAnnouncementsSentElapsedMicroseconds
Time spent in sending the announcement from the remote server to the initiator server about the set of data parts (for MergeTree tables). Measured on the remote server side.MergeTreeDataProjectionWriterBlocks
Number of blocks INSERTed to MergeTree tables projection. Each block forms a data part of level zero.MergeTreeDataProjectionWriterBlocksAlreadySorted
Number of blocks INSERTed to MergeTree tables projection that appeared to be already sorted.MergeTreeDataProjectionWriterCompressedBytes
Bytes written to filesystem for data INSERTed to MergeTree tables projection.MergeTreeDataProjectionWriterMergingBlocksMicroseconds
Time spent merging blocksMergeTreeDataProjectionWriterRows
Number of rows INSERTed to MergeTree tables projection.MergeTreeDataProjectionWriterSortingBlocksMicroseconds
Time spent sorting blocks (for projection it might be a key different from table’s sorting key)MergeTreeDataProjectionWriterUncompressedBytes
Uncompressed bytes (for columns as they stored in memory) INSERTed to MergeTree tables projection.MergeTreeDataWriterBlocks
Number of blocks INSERTed to MergeTree tables. Each block forms a data part of level zero.MergeTreeDataWriterBlocksAlreadySorted
Number of blocks INSERTed to MergeTree tables that appeared to be already sorted.MergeTreeDataWriterCompressedBytes
Bytes written to filesystem for data INSERTed to MergeTree tables.MergeTreeDataWriterMergingBlocksMicroseconds
Time spent merging input blocks (for special MergeTree engines)MergeTreeDataWriterProjectionsCalculationMicroseconds
Time spent calculating projectionsMergeTreeDataWriterRows
Number of rows INSERTed to MergeTree tables.MergeTreeDataWriterSkipIndicesCalculationMicroseconds
Time spent calculating skip indicesMergeTreeDataWriterSortingBlocksMicroseconds
Time spent sorting blocksMergeTreeDataWriterStatisticsCalculationMicroseconds
Time spent calculating statisticsMergeTreeDataWriterUncompressedBytes
Uncompressed bytes (for columns as they stored in memory) INSERTed to MergeTree tables.MergeTreePrefetchedReadPoolInit
Time spent preparing tasks in MergeTreePrefetchedReadPoolMergeTreeReadTaskRequestsReceived
The number of callbacks requested from the remote server back to the initiator server to choose the read task (for MergeTree tables). Measured on the initiator server side.MergeTreeReadTaskRequestsSent
The number of callbacks requested from the remote server back to the initiator server to choose the read task (for MergeTree tables). Measured on the remote server side.MergeTreeReadTaskRequestsSentElapsedMicroseconds
Time spent in callbacks requested from the remote server back to the initiator server to choose the read task (for MergeTree tables). Measured on the remote server side.MergeVerticalStageExecuteMilliseconds
Total busy time spent for execution of vertical stage of background mergesMergeVerticalStageTotalMilliseconds
Total time spent for vertical stage of background mergesMergeWrittenRows
Number of rows written during the merge.MergingSortedMilliseconds
Total time spent while merging sorted columnsMetadataFromKeeperBackgroundCleanupBlobs
Number of times an empty blob layout part was cleaned up by background taskMetadataFromKeeperBackgroundCleanupErrors
Number of times an error was encountered in background cleanup taskMetadataFromKeeperBackgroundCleanupObjects
Number of times a old deleted object clean up was performed by background taskMetadataFromKeeperBackgroundCleanupTransactions
Number of times old transaction idempotency token was cleaned up by background taskMetadataFromKeeperCacheHit
Number of times an object storage metadata request was answered from cache without making request to KeeperMetadataFromKeeperCacheMiss
Number of times an object storage metadata request had to be answered from KeeperMetadataFromKeeperCacheTooManyInvalidated
Number of times filesystem cache returned too many invalidated entriesMetadataFromKeeperCacheUpdateMicroseconds
Total time spent in updating the cache including waiting for responses from KeeperMetadataFromKeeperCleanupTransactionCommit
Number of times metadata transaction commit for deleted objects cleanup was attemptedMetadataFromKeeperCleanupTransactionCommitRetry
Number of times metadata transaction commit for deleted objects cleanup was retriedMetadataFromKeeperIndividualOperations
Number of paths read or written by single or multi requests to KeeperMetadataFromKeeperIndividualOperationsMicroseconds
Time spend during single or multi requests to KeeperMetadataFromKeeperOperations
Number of times a request was made to KeeperMetadataFromKeeperReconnects
Number of times a reconnect to Keeper was doneMetadataFromKeeperTransactionCommit
Number of times metadata transaction commit was attemptedMetadataFromKeeperTransactionCommitRetry
Number of times metadata transaction commit was retriedMetadataFromKeeperUpdateCacheOneLevel
Number of times a cache update for one level of directory tree was doneMetadataTransactionRollbacks
Number of metadata transactions that failed to commit and were rolled back.MetadataTransactionRollbacksFailed
Number of metadata transaction rollbacks that did not run to completion, so the metadata keeps a part of a transaction that was reported as failed.MMappedFileCacheHits
Number of times a file has been found in the MMap cache (for the ‘mmap’ read_method), so we didn’t have to mmap it again.MMappedFileCacheMisses
Number of times a file has not been found in the MMap cache (for the ‘mmap’ read_method), so we had to mmap it again.MoveBackgroundExecutorTaskCancelMicroseconds
Time spent in cancel() for Move executor tasks.MoveBackgroundExecutorTaskExecuteStepMicroseconds
Time spent in executeStep() for Move executor tasks.MoveBackgroundExecutorTaskResetMicroseconds
Time spent resetting task for Move executor.MoveBackgroundExecutorWaitMicroseconds
Time spent waiting for completion in Move executor.MutatedRows
Rows read for mutations. This is the number of rows before mutationMutatedUncompressedBytes
Uncompressed bytes (for columns as they stored in memory) that was read for mutations. This is the number before mutation.MutateTaskProjectionsCalculationMicroseconds
Time spent calculating projections in mutationsMutationAffectedRowsUpperBound
The upper bound of number of rows that were affected by mutation (e.g. number of rows that satisfy the predicate of UPDATE or DELETE mutation). The actual number may be slightly lessMutationAllPartColumns
Number of times when task to mutate all columns in part was createdMutationCommitMilliseconds
Total time spent for committing mutation results (part renaming, checksum verification, ZooKeeper updates)MutationCreatedEmptyParts
Number of total parts which were replaced to empty parts instead of running mutationMutationExecuteMilliseconds
Total busy time spent for execution of mutations.MutationsAppliedOnFlyInAllReadTasks
Total number of applied mutations on-fly among all read tasksMutationSomePartColumns
Number of times when task to mutate some columns in part was createdMutationsThrottlerBytes
Bytes passed through ‘max_mutations_bandwidth_for_server’ throttler.MutationsThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_mutations_bandwidth_for_server’ throttling.MutationTotalMilliseconds
Total time spent for mutations.MutationTotalParts
Number of total parts for which mutations tried to be appliedMutationUntouchedParts
Number of total parts for which mutations tried to be applied but which was completely skipped according to predicateNativeProtocolSend
Number of non-empty native protocol output buffer flushes.NetworkReceiveBytes
Total number of bytes received from network. Only ClickHouse-related network interaction is included, not by 3rd party libraries.NetworkReceiveElapsedMicroseconds
Total time spent waiting for data to receive or receiving data from network. Only ClickHouse-related network interaction is included, not by 3rd party libraries.NetworkSendBytes
Total number of bytes send to network. Only ClickHouse-related network interaction is included, not by 3rd party libraries.NetworkSendElapsedMicroseconds
Total time spent waiting for data to send to network or sending data to network. Only ClickHouse-related network interaction is included, not by 3rd party libraries.NotCreatedLogEntryForMerge
Log entry to merge parts in ReplicatedMergeTree is not created due to concurrent log update by another replica.NotCreatedLogEntryForMutation
Log entry to mutate parts in ReplicatedMergeTree is not created due to concurrent log update by another replica.ObjectStorageGlobFilteredObjects
Objects that did not match the glob or regex pattern and were skipped during listing.ObjectStorageListedObjects
Total objects returned by object storage listing API before any filtering.ObjectStoragePredicateFilteredObjects
Objects removed by virtual column predicate filtering on _path/_file.ObjectStorageQueueBucketLockLostOwnership
Number of times ownership of a bucket lock was detected as lost in S3(Azure)Queue. Non-zero value indicates too small persistent_processing_node_ttl_seconds or a bugObjectStorageQueueBucketLockRefreshes
Number of successful bucket lock refreshes in S3(Azure)QueueObjectStorageQueueCancelledFiles
Number cancelled files in StorageS3(Azure)QueueObjectStorageQueueCleanupMaxSetSizeOrTTLMicroseconds
Time spent to set file as failedObjectStorageQueueCommitRequests
Number of keeper requests to commit files as either failed or processedObjectStorageQueueExceptionsDuringInsert
Number of exceptions during insert in S3(Azure)QueueObjectStorageQueueExceptionsDuringRead
Number of exceptions during read in S3(Azure)QueueObjectStorageQueueExclusiveModeProcessingErrors
Count of times a file in S3(Azure)Queue didn’t finalize state due to a processing error. Only for ‘exclusive’ mode.ObjectStorageQueueFailedFiles
Number of files which failed to be processedObjectStorageQueueFailedToBatchSetProcessing
Number of times batched set processing request failedObjectStorageQueueFilteredFiles
Number of filtered files in StorageS3(Azure)QueueObjectStorageQueueInsertIterations
Number of insert iterationsObjectStorageQueueListedFiles
Number of listed files in StorageS3(Azure)QueueObjectStorageQueueMovedObjects
Number of objects moved as part of after_processing = moveObjectStorageQueueProcessedFiles
Number of files which were processedObjectStorageQueueProcessedRows
Number of processed rows in StorageS3(Azure)QueueObjectStorageQueuePullMicroseconds
Time spent to read file dataObjectStorageQueueReadBytes
Number of read bytes (not equal to the number of actually inserted bytes)ObjectStorageQueueReadFiles
Number of read files (not equal to the number of actually inserted files)ObjectStorageQueueReadRows
Number of read rows (not equal to the number of actually inserted rows)ObjectStorageQueueRemovedObjects
Number of objects removed as part of after_processing = deleteObjectStorageQueueRemoveObjectFailures
Number of objects that failed to be removed as part of after_processing = deleteObjectStorageQueueSuccessfulCommits
Number of successful keeper commitsObjectStorageQueueTaggedObjects
Number of objects tagged as part of after_processing = tagObjectStorageQueueTrySetProcessingFailed
The number of times we unsuccessfully set file as processingObjectStorageQueueTrySetProcessingRequests
The number of times we tried to make set processing requestObjectStorageQueueTrySetProcessingSucceeded
The number of times we successfully set file as processingObjectStorageQueueUnsuccessfulCommits
Number of unsuccessful keeper commitsObjectStorageReadObjects
Objects actually opened for reading by the object storage source.ObsoleteReplicatedParts
Number of times a data part was covered by another data part that has been fetched from a replica (so, we have marked a covered data part as obsolete and no longer needed).OneLakeAccessTokenExpirations
Number of times the cached OneLake access token was found expired and renewed transparently.OneLakeAccessTokenRequestFailures
Number of failed access token requests issued by the OneLake catalog via the Entra ID refresh token grant (including expired or revoked refresh tokens).OneLakeAccessTokenRequestMicroseconds
Total time spent requesting access tokens via the Entra ID refresh token grant in the OneLake catalog.OneLakeAccessTokenRequests
Number of access token requests issued by the OneLake catalog via the Entra ID refresh token grant.OpenedFileCacheHits
Number of times a file has been found in the opened file cache, so we didn’t have to open it again.OpenedFileCacheMicroseconds
Amount of time spent executing OpenedFileCache methods.OpenedFileCacheMisses
Number of times a file has been found in the opened file cache, so we had to open it again.OSCPUVirtualTimeMicroseconds
CPU time spent seen by OS. Does not include involuntary waits due to virtualization.OSCPUWaitMicroseconds
Total time a thread was ready for execution but waiting to be scheduled by OS, from the OS point of view.OSIOWaitMicroseconds
Total time a thread spent waiting for a result of IO operation, from the OS point of view. This is real IO that doesn’t include page cache.OSReadBytes
Number of bytes read from disks or block devices. Doesn’t include bytes read from page cache. May include excessive data due to block size, readahead, etc.OSReadChars
Number of bytes read from filesystem, including page cache, as well as network and other files.OSWriteBytes
Number of bytes written to disks or block devices. Doesn’t include bytes that are in page cache dirty pages. May not include data that was written by OS asynchronously.OSWriteChars
Number of bytes written to filesystem, including page cache, as well as network and other files.OtherQueryTimeMicroseconds
Total time of queries that are not SELECT or INSERT.OverflowAny
Number of times approximate GROUP BY was in effect: when aggregation was performed only on top of first ‘max_rows_to_group_by’ unique keys and other keys were ignored due to ‘group_by_overflow_mode’ = ‘any’.OverflowBreak
Number of times, data processing was cancelled by query complexity limitation with setting ’*_overflow_mode’ = ‘break’ and the result is incomplete.OverflowThrow
Number of times, data processing was cancelled by query complexity limitation with setting ’*_overflow_mode’ = ‘throw’ and exception was thrown.PageCacheHits
Number of times a block of data has been found in the userspace page cache.PageCacheMisses
Number of times a block of data has not been found in the userspace page cache.PageCacheOvercommitResize
Number of times the userspace page cache was auto-resized to free memory during a memory allocation.PageCacheReadBytes
Number of bytes read from userspace page cache.PageCacheResized
Number of times the userspace page cache was auto-resized (typically happens a few times per second, controlled by memory_worker_period_ms).PageCacheWeightLost
Number of bytes evicted from the userspace page cachePaimonMetadataFilesCacheHits
Number of times paimon metadata files have been found in the cache.PaimonMetadataFilesCacheMisses
Number of times paimon metadata files have not been found in the paimon metadata cache and had to be read from (remote) disk.PaimonMetadataFilesCacheWeightLost
Approximate number of bytes evicted from the paimon metadata cache.ParallelReplicasAnnouncementMicroseconds
Time spent to send an announcementParallelReplicasAvailableCount
Number of replicas available to execute a query with task-based parallel replicasParallelReplicasCollectingOwnedSegmentsMicroseconds
Time spent collecting segments meant by hashParallelReplicasDeniedRequests
Number of completely denied requests to the initiatorParallelReplicasHandleAnnouncementMicroseconds
Time spent processing replicas announcementsParallelReplicasHandleRequestMicroseconds
Time spent processing requests for marks from replicasParallelReplicasNumRequests
Number of requests to the initiator.ParallelReplicasProcessingPartsMicroseconds
Time spent processing data partsParallelReplicasQueryCount
Number of (sub)queries executed using parallel replicas during a query executionParallelReplicasReadAssignedForStealingMarks
Sum across all replicas of how many of scheduled marks were assigned for stealing by consistent hashParallelReplicasReadAssignedMarks
Sum across all replicas of how many of scheduled marks were assigned by consistent hashParallelReplicasReadMarks
How many marks were read by the given replicaParallelReplicasReadRequestMicroseconds
Time spent for read requestsParallelReplicasReadUnassignedMarks
Sum across all replicas of how many unassigned marks were scheduledParallelReplicasStealingByHashMicroseconds
Time spent collecting segments meant for stealing by hashParallelReplicasStealingLeftoversMicroseconds
Time spent collecting orphaned segmentsParallelReplicasUnavailableCount
Number of replicas which was chosen, but found to be unavailable during query execution with task-based parallel replicasParallelReplicasUsedCount
Number of replicas used to execute a query with task-based parallel replicasParquetColumnsFilterExpression
The total number of columns that were passed through filterParquetDecodingTaskBatches
Task groups sent to a thread pool by parquet readerParquetDecodingTasks
Tasks issued by parquet readerParquetFetchWaitTimeMicroseconds
Time of waiting for parquet file reads from decoding threads (not prefetching threads)ParquetMetadataCacheHits
Number of times parquet metadata has been found in the cache.ParquetMetadataCacheMisses
Number of times parquet metadata has not been found in the cache and had to be read from disk.ParquetMetadataCacheWeightLost
Approximate number of bytes evicted from the parquet metadata cache.ParquetPrefetcherReadEntireFile
The total number of read with ReadMode::EntireFileIsInMemory by DB::Parquet::PrefetcherParquetPrefetcherReadRandomRead
The total number of reads with ReadMode::RandomRead by DB::Parquet::PrefetcherParquetPrefetcherReadSeekAndRead
The total number of reads with ReadMode::SeekAndRead by DB::Parquet::PrefetcherParquetPrunedPages
The total number of pages pruned from parquet data via column indexParquetPrunedRowGroups
The total number of row groups pruned from parquet dataParquetReadPages
The total number of Parquet data pages readParquetReadRowGroups
The total number of row groups read from parquet dataParquetRowsFilterExpression
The total number of rows that were passed through filterPartsLockHoldMicroseconds
Total time spent holding data parts lock in MergeTree tablesPartsLocks
Number of times data parts lock has been acquired for MergeTree tablesPartsLockWaitMicroseconds
Total time spent waiting for data parts lock in MergeTree tablesPatchesAcquireLockBadVersionRetries
Total number of retries while acquiring the lock for lightweight updates caused by a concurrent update committing firstPatchesAcquireLockMicroseconds
Total number of microseconds spent to acquire lock for executing lightweight updatesPatchesAcquireLockTries
Total number of tries to acquire lock for executing lightweight updatesPatchesAppliedInAllReadTasks
Total number of applied patch parts among all read tasksPatchesJoinAppliedInAllReadTasks
Total number of applied patch parts with Join mode among all read tasksPatchesJoinRowsAddedToHashTable
Total number of rows added to hash tables when applying patch parts with Join modePatchesMergeAppliedInAllReadTasks
Total number of applied patch parts with Merge mode among all read tasksPatchesMergeOnKeyAppliedInAllReadTasks
Total number of applied patch parts with MergeOnKey mode (v2) among all read tasksPatchesReadRows
Total number of rows read from patch partsPatchesReadUncompressedBytes
Total number of uncompressed bytes read from patch partsPerfAlignmentFaults
Number of alignment faults. These happen when unaligned memory accesses happen; the kernel can handle these but it reduces performance. This happens only on some architectures (never on x86).PerfBranchInstructions
Retired branch instructions. Prior to Linux 2.6.35, this used the wrong event on AMD processors.PerfBranchMisses
Mispredicted branch instructions.PerfBusCycles
Bus cycles, which can be different from total cycles.PerfCacheMisses
Cache misses. Usually this indicates Last Level Cache misses; this is intended to be used in conjunction with the PERFCOUNTHWCACHEREFERENCES event to calculate cache miss rates.PerfCacheReferences
Cache accesses. Usually, this indicates Last Level Cache accesses, but this may vary depending on your CPU. This may include prefetches and coherency messages; again this depends on the design of your CPU.PerfContextSwitches
Number of context switchesPerfCPUClock
The CPU clock, a high-resolution per-CPU timerPerfCPUCycles
Total cycles. Be wary of what happens during CPU frequency scaling.PerfCPUMigrations
Number of times the process has migrated to a new CPUPerfDataTLBMisses
Data TLB missesPerfDataTLBReferences
Data TLB referencesPerfEmulationFaults
Number of emulation faults. The kernel sometimes traps on unimplemented instructions and emulates them for user space. This can negatively impact performance.PerfInstructions
Retired instructions. Be careful, these can be affected by various issues, most notably hardware interrupt counts.PerfInstructionTLBMisses
Instruction TLB missesPerfInstructionTLBReferences
Instruction TLB referencesPerfLocalMemoryMisses
Local NUMA node memory read missesPerfLocalMemoryReferences
Local NUMA node memory readsPerfMinEnabledRunningTime
Running time for event with minimum enabled time. Used to track the amount of event multiplexingPerfMinEnabledTime
For all events, minimum time that an event was enabled. Used to track event multiplexing influencePerfRefCPUCycles
Total cycles; not affected by CPU frequency scaling.PerfStalledCyclesBackend
Stalled cycles during retirement.PerfStalledCyclesFrontend
Stalled cycles during issue.PerfTaskClock
A clock count specific to the task that is runningPreferredWarmedUnmergedParts
See setting prefer_warmed_unmerged_parts_seconds. Number of times read queries used outdated pre-merge parts that are in cache instead of merged part that wasn’t pulled into cache by CacheWarmer yet.PrimaryIndexCacheHits
Number of times an entry has been found in the primary index cache, so we didn’t have to load a index file.PrimaryIndexCacheMisses
Number of times an entry has not been found in the primary index cache, so we had to load a index file in memory, which is a costly operation, adding to query latency.QueriesWithSubqueries
Count queries with all subqueriesQuery
Number of queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.QueryAnalysisMicroseconds
Total time spent building the query tree and running analyzer passes (resolves identifiers, types, expressions).QueryBackupThrottlerBytes
Bytes passed through ‘max_backup_bandwidth’ throttler.QueryBackupThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_backup_bandwidth’ throttling.QueryCacheAgeSeconds
The sum of ages of found query cache entries in seconds. The value is set both for hits and misses.QueryCacheHits
Number of times a query result has been found in the query cache (and query computation was avoided). Only updated for SELECT queries with SETTING use_query_cache = 1.QueryCacheMisses
Number of times a query result has not been found in the query cache (and required query computation). Only updated for SELECT queries with SETTING use_query_cache = 1.QueryCacheReadBytes
The number of (uncompressed) bytes read from the query cache.QueryCacheReadRows
The number of rows read from the query cache.QueryCacheWrittenBytes
The number of (uncompressed) bytes saved into the query cacheQueryCacheWrittenRows
The number of rows saved into the query cache.QueryConditionCacheHits
Number of times an entry has been found in the query condition cache (and reading of marks can be skipped). Only updated for SELECT queries with SETTING use_query_condition_cache = 1.QueryConditionCacheMisses
Number of times an entry has not been found in the query condition cache (and reading of mark cannot be skipped). Only updated for SELECT queries with SETTING use_query_condition_cache = 1.QueryLocalReadThrottlerBytes
Bytes passed through ‘max_local_read_bandwidth’ throttler.QueryLocalReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_local_read_bandwidth’ throttling.QueryLocalWriteThrottlerBytes
Bytes passed through ‘max_local_write_bandwidth’ throttler.QueryLocalWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_local_write_bandwidth’ throttling.QueryMaskingRulesMatch
Number of times query masking rules was successfully matched.QueryMemoryLimitExceeded
Number of times when memory limit exceeded for query.QueryParseMicroseconds
Total time spent parsing the query text into an AST.QueryPipelineBuildMicroseconds
Total time spent constructing the QueryPipeline from the optimized QueryPlan.QueryPlanBuildMicroseconds
Total time spent building the QueryPlan from the analyzed query tree.QueryPlanOptimizeMicroseconds
Total time spent executing query plan optimizations.QueryPreempted
How many times tasks are paused and waiting due to ‘priority’ settingQueryProfilerConcurrencyOverruns
Number of times we drop processing of a query profiler signal due to too many concurrent query profilers in other threads, which may indicate overload.QueryProfilerErrors
Invalid memory accesses during asynchronous stack unwinding.QueryProfilerRuns
Number of times QueryProfiler had been run.QueryProfilerSignalOverruns
Number of times we drop processing of a query profiler signal due to overrun plus the number of signals that OS has not delivered due to overrun.QueryRemoteReadThrottlerBytes
Bytes passed through ‘max_remote_read_network_bandwidth’ throttler.QueryRemoteReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_remote_read_network_bandwidth’ throttling.QueryRemoteWriteThrottlerBytes
Bytes passed through ‘max_remote_write_network_bandwidth’ throttler.QueryRemoteWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_remote_write_network_bandwidth’ throttling.QueryTimeMicroseconds
Total time of all queries.QuorumFailedInserts
Number of inserts failed due to quorum not reaching.QuorumParts
Number of data parts written with quorum. It counts as one part for sync insert and maybe up to async inserts count for insert which flushes async inserts.QuorumWaitMicroseconds
Total time spent waiting for quorum during inserts.QuotaCacheRecalculationMicroseconds
Total time spent re-choosing quotas for all live enabled sets (held under the cache mutex that the ContextAccess build path also takes).QuotaCacheRecalculations
Number of times the quota cache re-chose quotas for all live enabled sets. Coalesced to once per access entity notification batch.ReadBackoff
Number of times the number of query processing threads was lowered due to slow reads.ReadBufferFromAzureBytes
Bytes read from Azure.ReadBufferFromAzureInitMicroseconds
Time spent initializing connection to Azure.ReadBufferFromAzureMicroseconds
Time spent on reading from Azure.ReadBufferFromAzureRequestsErrors
Number of exceptions while reading from AzureReadBufferFromFileDescriptorRead
Number of reads (read/pread) from a file descriptor. Does not include sockets.ReadBufferFromFileDescriptorReadBytes
Number of bytes read from file descriptors. If the file is compressed, this will show the compressed data size.ReadBufferFromFileDescriptorReadFailed
Number of times the read (read/pread) from a file descriptor have failed.ReadBufferFromS3Bytes
Bytes read from S3.ReadBufferFromS3InitMicroseconds
Time spent initializing connection to S3.ReadBufferFromS3Microseconds
Time spent on reading from S3.ReadBufferFromS3RequestsErrors
Number of exceptions while reading from S3.ReadBufferSeekCancelConnection
Number of seeks which lead to new connection (s3, http)ReadCompressedBytes
Number of bytes (the number of bytes before decompression) read from compressed sources (files, network).ReaderExecutorBytesFromSource
Physical bytes ReaderExecutor issued to the source after missing all cache tiers (foreground plus background prefetch, including a prefetch’s bytes wasted by a later discard); not consumer-served bytes - see ReaderExecutorDeliveredBytes.ReaderExecutorCacheGetRequests
Number of ICacheHandle::get invocations in ReaderExecutor. Zero until the ReaderExecutor cache tiers are introduced.ReaderExecutorCachePopulateRequests
Number of ICacheHandle::put invocations in ReaderExecutor. Zero until the ReaderExecutor cache tiers are introduced.ReaderExecutorDecryptMicroseconds
Time ReaderExecutor spent decrypting served payload in place (CTR, position-addressable). Zero for unencrypted reads and in builds without SSL.ReaderExecutorDeliveredBytes
Useful bytes ReaderExecutor delivered to read requests (the requested window payload, excluding over-read and cache write-back). Denominator for the modeled cost-per-byte KPI (ReaderExecutorModeledCostMicroseconds / ReaderExecutorDeliveredBytes).ReaderExecutorIncompleteConnections
Number of source connections ReaderExecutor dropped before draining them to their right bound; not pool-reusable, forcing a re-establishment. Zero until ReaderExecutor live source-buffer reuse is introduced.ReaderExecutorLongConnectionBytes
Total bytes read through long source connections.ReaderExecutorLongConnectionFallbacks
Number of times ReaderExecutor wanted a long connection but fell back to a one-shot read because no slot was available.ReaderExecutorLongConnectionHits
Number of windows ReaderExecutor served by reading from an already-open long source connection.ReaderExecutorLongConnectionOpened
Number of long source connections opened by ReaderExecutor for sequential read optimization.ReaderExecutorModeledCostMicroseconds
Modeled I/O cost of ReaderExecutor reads: a synthetic proxy KPI for read-path optimality, NOT measured latency. Weighted sum of the counters above with heuristic S3 weights: 30ms per source request + 5ms per incomplete connection + 20ms per MiB transferred from source (useful payload plus over-read) + 0.1ms per cache put + 0.05ms per cache get. Divide by ReaderExecutorDeliveredBytes for a load-independent cost-per-byte. Experimental, tracks the experimental ReaderExecutor.ReaderExecutorSourceRequests
Number of source-side requests opened by ReaderExecutor (excludes live-buffer reuses).ReaderExecutorWorkMicroseconds
Total wall-clock time spent inside ReaderExecutor::readNextWindow (opening, seeking and reading the served window). Direct contributor to query read latency.ReadPatchesMicroseconds
Total time spent reading patch partsReadPoolRangeRefinerDroppedCuts
How many times ranges cut from a part by a MergeTree read pool were fully dropped by a ranges refiner, so no read task was created for themReadPoolRangeRefinerDroppedMarks
How many marks were dropped from read tasks by a ranges refiner in MergeTree read pools (e.g. marks fully filtered out by a projection index)ReadTaskRequestsReceived
The number of callbacks requested from the remote server back to the initiator server to choose the read task (for s3Cluster table function and similar). Measured on the initiator server side.ReadTaskRequestsSent
The number of callbacks requested from the remote server back to the initiator server to choose the read task (for s3Cluster table function and similar). Measured on the remote server side.ReadTaskRequestsSentElapsedMicroseconds
Time spent in callbacks requested from the remote server back to the initiator server to choose the read task (for s3Cluster table function and similar). Measured on the remote server side.ReadTasksWithAppliedMutationsOnFly
Total number of read tasks for which there was any mutation applied on flyReadTasksWithAppliedPatches
Total number of read tasks for which there was any patch part appliedReadWriteBufferFromHTTPBytes
Total size of payload bytes received and sent by ReadWriteBufferFromHTTP. Doesn’t include HTTP headers.ReadWriteBufferFromHTTPRequestsSent
Number of HTTP requests sent by ReadWriteBufferFromHTTPRealTimeMicroseconds
Total (wall clock) time spent in processing (queries and other tasks) threads (note that this is a sum).RebuiltProjections
Number of projections rebuilt from scratch during MergeTree merges.RefreshableViewLockTableRetry
How many times a SELECT from refreshable materialized view had to switch to a new table because the old table was droppedRefreshableViewRefreshFailed
How many times refreshable materialized views failed to refreshRefreshableViewRefreshSuccess
How many times refreshable materialized views refreshedRefreshableViewSyncReplicaRetry
How many times a SELECT from refreshable materialized view failed and retried an implicit SYNC REPLICARefreshableViewSyncReplicaSuccess
How many times a SELECT from refreshable materialized view did an implicit SYNC REPLICARegexpLocalCacheHit
Number of times we fetched compiled regular expression from a local cache.RegexpLocalCacheMiss
Number of times we failed to fetch compiled regular expression from a local cache.RegexpWithMultipleNeedlesCreated
Regular expressions with multiple needles (VectorScan library) compiled.RegexpWithMultipleNeedlesGlobalCacheHit
Number of times we fetched compiled regular expression with multiple needles (VectorScan library) from the global cache.RegexpWithMultipleNeedlesGlobalCacheMiss
Number of times we failed to fetch compiled regular expression with multiple needles (VectorScan library) from the global cache.RejectedInserts
Number of times the INSERT of a block to a MergeTree table was rejected with ‘Too many parts’ exception due to high number of active data parts for partition.RejectedLightweightUpdates
Number of time the lightweight update was rejected due to too many uncompressed bytes in patches.RejectedMutations
Number of times the mutation of a MergeTree table was rejected with ‘Too many mutations’ exception due to high number of unfinished mutations for table.RemoteFSBuffers
Number of buffers created for asynchronous reading from remote filesystemRemoteFSCancelledPrefetches
Number of cancelled prefetches (because of seek)RemoteFSLazySeeks
Number of lazy seeksRemoteFSPrefetchedBytes
Number of bytes from prefetched bufferRemoteFSPrefetchedReads
Number of reads from prefetched bufferRemoteFSPrefetches
Number of prefetches made with asynchronous reading from remote filesystemRemoteFSSeeks
Total number of seeks for async bufferRemoteFSSeeksWithReset
Number of seeks which lead to a new connectionRemoteFSUnprefetchedBytes
Number of bytes from unprefetched bufferRemoteFSUnprefetchedReads
Number of reads from unprefetched bufferRemoteFSUnusedPrefetches
Number of prefetches pending at buffer destructionRemoteReadThrottlerBytes
Bytes passed through ‘max_remote_read_network_bandwidth_for_server’/‘max_remote_read_network_bandwidth’ throttler.RemoteReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_remote_read_network_bandwidth_for_server’/‘max_remote_read_network_bandwidth’ throttling.RemoteWriteThrottlerBytes
Bytes passed through ‘max_remote_write_network_bandwidth_for_server’/‘max_remote_write_network_bandwidth’ throttler.RemoteWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_remote_write_network_bandwidth_for_server’/‘max_remote_write_network_bandwidth’ throttling.ReplacingSortedMilliseconds
Total time spent while replacing sorted columnsReplicaPartialShutdown
How many times Replicated table has to deinitialize its state due to session expiration in ZooKeeper. The state is reinitialized every time when ZooKeeper is available again.ReplicatedCoveredPartsInZooKeeperOnStart
For debugging purposes. Number of parts in ZooKeeper that have a covering part, but doesn’t exist on disk. Checked on server start.ReplicatedDataLoss
Number of times a data part that we wanted doesn’t exist on any replica (even on replicas that are offline right now). That data parts are definitely lost. This is normal due to asynchronous replication (if quorum inserts were not enabled), when the replica on which the data part was written was failed and when it became online after fail it doesn’t contain that data part.ReplicatedPartChecks
Number of times we had to perform advanced search for a data part on replicas or to clarify the need of an existing data part.ReplicatedPartChecksFailed
Number of times the advanced search for a data part on replicas did not give result or when unexpected part has been found and moved away.ReplicatedPartFailedFetches
Number of times a data part was failed to download from replica of a ReplicatedMergeTree table.ReplicatedPartFetches
Number of times a data part was downloaded from replica of a ReplicatedMergeTree table.ReplicatedPartFetchesOfMerged
Number of times we prefer to download already merged part from replica of ReplicatedMergeTree table instead of performing a merge ourself (usually we prefer doing a merge ourself to save network traffic). This happens when we have not all source parts to perform a merge or when the data part is old enough.ReplicatedPartMerges
Number of times data parts of ReplicatedMergeTree tables were successfully merged.ReplicatedPartMutations
Number of times data parts of ReplicatedMergeTree tables were successfully mutated.RestorePartsSkippedBytes
Total size of files skipped while restoring partsRestorePartsSkippedFiles
Number of files skipped while restoring partsRoleCacheRecalculationMicroseconds
Total time spent recalculating all live enabled role sets (held under the cache mutex that the ContextAccess build path also takes).RoleCacheRecalculations
Number of times the role cache recalculated all live enabled sets. Coalesced to once per access entity notification batch.RowPolicyCacheRecalculationMicroseconds
Total time spent re-mixing row policy filters for all live enabled sets, off the RowPolicyCache mutex (so it does not block the ContextAccess build path).RowPolicyCacheRecalculations
Number of times the row policy cache re-mixed filters for all live enabled sets. Coalesced to once per access entity notification batch.RowsReadByMainReader
Number of rows read from MergeTree tables by the main reader (after PREWHERE step).RowsReadByPrewhereReaders
Number of rows read from MergeTree tables (in total) by prewhere readers.RuntimeDataflowStatisticsInputBytes
Collected statistics on the number of bytes replicas would read if the query was executed with parallel replicasRuntimeDataflowStatisticsOutputBytes
Collected statistics on the number of bytes replicas would send to the initiator if the query was executed with parallel replicasRuntimeFilterBlocksProcessed
Number of blocks processed by JOIN Runtime FiltersRuntimeFilterBlocksSkipped
Number of blocks skipped by JOIN Runtime Filters without processing due to filter being dynamically disabled because of poor filtering ratioRuntimeFilterBloomFilterBuildsSkipped
Number of JOIN Runtime Filter Bloom filter builds skipped because the build-side key count from the hash table statistics predicted that the filter would exceed the maximal ratio of set bitsRuntimeFilterGranulesConsidered
Number of granules examined for read time pruning by JOIN Runtime FiltersRuntimeFilterGranulesDropped
Number of granules pruned at read time by JOIN Runtime FiltersRuntimeFilterRowsChecked
Number of rows checked by JOIN Runtime FiltersRuntimeFilterRowsPassed
Number of rows that passed (not filtered out by) JOIN Runtime FiltersRuntimeFilterRowsSkipped
Number of rows in blocks that were skipped by JOIN Runtime FiltersRuntimeFiltersCreated
Number of distinct JOIN Runtime Filters created within a queryRWLockAcquiredReadLocks
Number of times a read lock was acquired (in a heavy RWLock).RWLockAcquiredWriteLocks
Number of times a write lock was acquired (in a heavy RWLock).RWLockReadersWaitMilliseconds
Total time spent waiting for a read lock to be acquired (in a heavy RWLock).RWLockWritersWaitMilliseconds
Total time spent waiting for a write lock to be acquired (in a heavy RWLock).S3AbortMultipartUpload
Number of S3 API AbortMultipartUpload calls.S3CachedCredentialsProvidersAdded
Total number of newly added credentials providers to the cacheS3CachedCredentialsProvidersReused
Total number of reused credentials provider from the cacheS3Clients
Number of created S3 clients.S3CompleteMultipartUpload
Number of S3 API CompleteMultipartUpload calls.S3CopyObject
Number of S3 API CopyObject calls.S3CreateMultipartUpload
Number of S3 API CreateMultipartUpload calls.S3DeleteObjects
Number of S3 API DeleteObject(s) calls.S3GetObject
Number of S3 API GetObject calls.S3GetObjectTagging
Number of S3 API GetObjectTagging calls.S3GetRequestThrottlerBlocked
Number of S3 GET and SELECT requests blocked by throttler.S3GetRequestThrottlerCount
Number of S3 GET and SELECT requests passed through throttler: blocked and not blocked.S3GetRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform S3 GET and SELECT request throttling.S3HeadObject
Number of S3 API HeadObject calls.S3ListObjects
Number of S3 API ListObjects calls.S3PutObject
Number of S3 API PutObject calls.S3PutRequestThrottlerBlocked
Number of S3 PUT, COPY, POST and LIST requests blocked by throttler.S3PutRequestThrottlerCount
Number of S3 PUT, COPY, POST and LIST requests passed through throttler: blocked and not blocked.S3PutRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform S3 PUT, COPY, POST and LIST request throttling.S3QueueSetFileFailedMicroseconds
Time spent to set file as failedS3QueueSetFileProcessedMicroseconds
Time spent to set file as processedS3QueueSetFileProcessingMicroseconds
Time spent to set file as processingS3ReadMicroseconds
Time of GET and HEAD requests to S3 storage.S3ReadRequestAttempts
Number of attempts for GET and HEAD requests, including the initial try and any retries, but excluding retries performed internally by the S3 retry strategyS3ReadRequestRetryableErrors
Number of retryable errors for GET and HEAD requests, excluding retries performed internally by the S3 retry strategyS3ReadRequestsCount
Number of GET and HEAD requests to S3 storage.S3ReadRequestsErrors
Number of non-throttling errors in GET and HEAD requests to S3 storage.S3ReadRequestsRedirects
Number of redirects in GET and HEAD requests to S3 storage.S3ReadRequestsThrottling
Number of 429 and 503 errors in GET and HEAD requests to S3 storage.S3UploadPart
Number of S3 API UploadPart calls.S3UploadPartCopy
Number of S3 API UploadPartCopy calls.S3WriteMicroseconds
Time of POST, DELETE, PUT and PATCH requests to S3 storage.S3WriteRequestAttempts
Number of attempts for POST, DELETE, PUT and PATCH requests, including the initial try and any retries, but excluding retries performed internally by the retry strategyS3WriteRequestRetryableErrors
Number of retryable errors for POST, DELETE, PUT and PATCH requests, excluding retries performed internally by the retry strategyS3WriteRequestsCount
Number of POST, DELETE, PUT and PATCH requests to S3 storage.S3WriteRequestsErrors
Number of non-throttling errors in POST, DELETE, PUT and PATCH requests to S3 storage.S3WriteRequestsRedirects
Number of redirects in POST, DELETE, PUT and PATCH requests to S3 storage.S3WriteRequestsThrottling
Number of 429 and 503 errors in POST, DELETE, PUT and PATCH requests to S3 storage.ScalarSubqueriesCacheMiss
Number of times a read from a scalar subquery was not cached and had to be calculated completelyScalarSubqueriesGlobalCacheHit
Number of times a read from a scalar subquery was done using the global cacheScalarSubqueriesLocalCacheHit
Number of times a read from a scalar subquery was done using the local cacheSchedulerIOReadBytes
Bytes passed through scheduler for IO reads.SchedulerIOReadRequests
Resource requests passed through scheduler for IO reads.SchedulerIOReadWaitMicroseconds
Total time a query was waiting on resource requests for IO reads.SchedulerIOWriteBytes
Bytes passed through scheduler for IO writes.SchedulerIOWriteRequests
Resource requests passed through scheduler for IO writes.SchedulerIOWriteWaitMicroseconds
Total time a query was waiting on resource requests for IO writes.SchemaInferenceCacheEvictions
Number of times a schema from cache was evicted due to overflowSchemaInferenceCacheHits
Number of times the requested source is found in schema cacheSchemaInferenceCacheInvalidations
Number of times a schema in cache became invalid due to changes in dataSchemaInferenceCacheMisses
Number of times the requested source is not in schema cacheSchemaInferenceCacheNumRowsHits
Number of times the number of rows is found in schema cache during count from filesSchemaInferenceCacheNumRowsMisses
Number of times the requested source is in cache but the number of rows is not in cache while count from filesSchemaInferenceCacheSchemaHits
Number of times the schema is found in schema cache during schema inferenceSchemaInferenceCacheSchemaMisses
Number of times the requested source is in cache but the schema is not in cache during schema inferenceSeek
Number of times the ‘lseek’ function was called.SelectedBytes
Number of bytes (uncompressed; for columns as they stored in memory) SELECTed from all tables.SelectedMarks
Number of marks (index granules) selected to read from a MergeTree table.SelectedMarksTotal
Number of total marks (index granules) before selecting which ones to read from a MergeTree table.SelectedParts
Number of data parts selected to read from a MergeTree table.SelectedPartsTotal
Number of total data parts before selecting which ones to read from a MergeTree table.SelectedRanges
Number of (non-adjacent) ranges in all data parts selected to read from a MergeTree table.SelectedRows
Number of rows SELECTed from all tables.SelectivityEstimatorInSetEstimatedFromSize
Number ofIN conditions whose selectivity was estimated from the size and bounds of the set instead of its exact ranges, because the set exceeds statistics_max_set_size_for_exact_selectivity_estimation
SelectivityEstimatorInSetNotBuilt
Number ofIN conditions the selectivity estimator could not analyse because the set was not built yet, and it must not run the subquery to fill it
SelectQueriesWithPrimaryKeyUsage
Count SELECT queries which use the primary key to evaluate the WHERE conditionSelectQueriesWithSubqueries
Count SELECT queries with all subqueriesSelectQuery
Same as Query, but only for SELECT queries.SelectQueryTimeMicroseconds
Total time of SELECT queries.SelfDuplicatedAsyncInserts
Number of async inserts in the INSERTed block to a ReplicatedMergeTree table was self deduplicated.ServerStartupMilliseconds
Time elapsed from starting server to listening to sockets in millisecondsSetsBuiltFromSubquery
Number ofIN/JOIN sets filled by running their subquery. A set taken from the prepared sets cache, or already built and reused, is not counted.
SettingsProfileCacheRecalculationMicroseconds
Total time spent re-merging settings and constraints for all live enabled sets (held under the cache mutex that the ContextAccess build path also takes).SettingsProfileCacheRecalculations
Number of times the settings profile cache re-merged settings and constraints for all live enabled sets. Coalesced to once per access entity notification batch.Shards
The number of shards involved in a query, summed across all distributed tables and table functions. A single host is counted multiple times if it appears in multiple tables. The number counts the total expected number of shards, which includes skipped shards with theskip_unavailable_shards setting.
SharedCatalogStateFetchFullFallback
Number of times the incremental fetch of the Shared Catalog state failed too many times in a row and a full state fetch was performed insteadSharedDatabaseCatalogFailedToApplyState
Number of failures to apply new state in SharedDatabaseCatalogSharedDatabaseCatalogStateApplicationMicroseconds
Total time spend on application of new state in SharedDatabaseCatalogSharedMergeTreeBlobRefCounterSharedBlobs
How many data blobs were path-shared between SharedMergeTree blob-list parts (reference-count increments staged in commit transactions, before accumulation into one update per counter node)SharedMergeTreeBlobsListBlobsWritten
How many data blobs were recorded across SharedMergeTree blobs.list mapsSharedMergeTreeBlobsListInlineFilesWritten
How many small files were stored inline in SharedMergeTree blobs.list maps instead of separate blobsSharedMergeTreeBlobsListPartsWritten
How many SharedMergeTree blob-list parts were written (one consolidated blobs.list node each)SharedMergeTreeCondemnedPartsKillRequest
How many ZooKeeper requests were used to remove condemned partsSharedMergeTreeCondemnedPartsLockConflict
How many times we failed to acquire lock because of conflictSharedMergeTreeCondemnedPartsRemoved
How many condemned parts were removedSharedMergeTreeDataPartsFetchAttempt
How many times we tried to fetch data partsSharedMergeTreeDataPartsFetchFromPeer
How many times we fetch data parts from peerSharedMergeTreeDataPartsFetchFromPeerMicroseconds
Data parts fetch from peer microsecondsSharedMergeTreeDataPartsFetchFromS3
How many times we fetch data parts from S3SharedMergeTreeFetchDeduplicated
Number of part fetches deduplicated because the part was already being fetched or produced by another threadSharedMergeTreeHandleBlockingParts
How many blocking parts to handle in scheduleDataProcessingJobSharedMergeTreeHandleBlockingPartsMicroseconds
Time of handling blocking parts in scheduleDataProcessingJobSharedMergeTreeHandleFetchPartsMicroseconds
Time of handling fetched parts in scheduleDataProcessingJobSharedMergeTreeHandleOutdatedParts
How many outdated parts to handle in scheduleDataProcessingJobSharedMergeTreeHandleOutdatedPartsMicroseconds
Time of handling outdated parts in scheduleDataProcessingJobSharedMergeTreeLoadChecksumAndIndexesMicroseconds
Time of loadColumnsChecksumsIndexes only for SharedMergeTreeSharedMergeTreeMergeMutationAssignmentAttempt
How many times we tried to assign merge or mutationSharedMergeTreeMergeMutationAssignmentFailedWithConflict
How many times we tried to assign merge or mutation and failed because of conflict in KeeperSharedMergeTreeMergeMutationAssignmentFailedWithNothingToDo
How many times we tried to assign merge or mutation and failed because nothing to mergeSharedMergeTreeMergeMutationAssignmentSuccessful
How many times we tried to assign merge or mutationSharedMergeTreeMergePartsMovedToCondemned
How many parts moved to condemned directorySharedMergeTreeMergePartsMovedToOudated
How many parts moved to outdated directorySharedMergeTreeMergeSelectingTaskMicroseconds
Merge selecting task microseconds for SMTSharedMergeTreeMetadataCacheHintLoadedFromCache
Number of times metadata cache hint was found without going to KeeperSharedMergeTreeOptimizeAsync
Asynchronous OPTIMIZE queries executedSharedMergeTreeOptimizeSync
Synchronous OPTIMIZE queries executedSharedMergeTreeOutdatedPartsConfirmationInvocations
How many invocations were made to confirm outdated partsSharedMergeTreeOutdatedPartsConfirmationRequest
How many ZooKeeper requests were used to config outdated partsSharedMergeTreeOutdatedPartsHTTPRequest
How many HTTP requests were send to confirm outdated partsSharedMergeTreeOutdatedPartsHTTPResponse
How many HTTP responses were send to confirm outdated partsSharedMergeTreePartitionOperationRetries
How many times a partition-level operation (DROP/MOVE/REPLACE/CLONE/TRUNCATE PARTITION) had to retry the Keeper multi-write because of a concurrent changeSharedMergeTreePartsKillerMicroseconds
How much time does parts killer main thread takesSharedMergeTreePartsKillerParts
How many parts has been scheduled by the killerSharedMergeTreePartsKillerPartsMicroseconds
How many time does it take to remove parts (executed from multiple threads)SharedMergeTreePartsKillerRuns
How many times parts killer has been runningSharedMergeTreeReplicaSetUpdatesFromZooKeeper
How many times we have update replica set from ZooKeeperSharedMergeTreeReplicaSetUpdatesFromZooKeeperMicroseconds
How much time we spend to update replica setSharedMergeTreeReplicaSetUpdatesFromZooKeeperRequests
How many total ZooKeeper requests we made to update replica setSharedMergeTreeReplicaSetUpdateTaskRuns
Number of times updateReplicaSetTask has runSharedMergeTreeScheduleDataProcessingJob
How many times scheduleDataProcessingJob called/SharedMergeTreeScheduleDataProcessingJobMicroseconds
scheduleDataProcessingJob execute timeSharedMergeTreeScheduleDataProcessingJobNothingToScheduled
How many times scheduleDataProcessingJob called but nothing to doSharedMergeTreeSelectPartsForCoordinatedFetchMicroseconds
Time of selectPartsForCoordinatedFetchSharedMergeTreeSelectPartsForCoordinatedFetchParts
Number of parts selected by selectPartsForCoordinatedFetchSharedMergeTreeSelectPartsForFullFetchMicroseconds
Time of selectPartsForFullFetchSharedMergeTreeSelectPartsForFullFetchParts
Number of parts selected by selectPartsForFullFetchSharedMergeTreeSelectPartsForRendezvousFetchMicroseconds
Time of selectPartsForRendezvousFetchSharedMergeTreeSelectPartsForRendezvousFetchParts
Number of parts selected by selectPartsForRendezvousFetchSharedMergeTreeSnapshotDeadLocksReclaimed
How many Keeper nodes or subtrees of no longer live lightweight snapshots were reclaimedSharedMergeTreeSnapshotPartsCleanerMicroseconds
How long time SnapshotCleanerThread has runSharedMergeTreeSnapshotPartsCleanerParts
How long time SnapshotCleanerThread tries to clean a partSharedMergeTreeSnapshotPartsCleanerPartsMicroseconds
How long time SnapshotCleanerThread takes to clean partsSharedMergeTreeSnapshotPartsCleanerRuns
How many times SnapshotCleanerThread runsSharedMergeTreeSnapshotPartsCleanRequest
How many times SnapshotCleanerThread decides to clean a partSharedMergeTreeSnapshotPartsRemoved
How many times SnapshotCleanerThread successfully clean a partSharedMergeTreeTryUpdateDiskMetadataCacheForPartMicroseconds
Time of tryUpdateDiskMetadataCacheForPart in scheduleDataProcessingJobSharedMergeTreeVirtualPartsUpdateMicroseconds
Virtual parts update microsecondsSharedMergeTreeVirtualPartsUpdates
Virtual parts update countSharedMergeTreeVirtualPartsUpdatesByLeader
Virtual parts updates by leaderSharedMergeTreeVirtualPartsUpdatesForMergesOrStatus
Virtual parts updates from non-default background jobSharedMergeTreeVirtualPartsUpdatesFromPeer
Virtual parts updates count from peerSharedMergeTreeVirtualPartsUpdatesFromPeerMicroseconds
Virtual parts updates from peer microsecondsSharedMergeTreeVirtualPartsUpdatesFromZooKeeper
Virtual parts updates count from ZooKeeperSharedMergeTreeVirtualPartsUpdatesFromZooKeeperFullyCached
Number of virtual parts discoveries where all partitions were used from the discovery cache and nothing was fetched from ZooKeeperSharedMergeTreeVirtualPartsUpdatesFromZooKeeperMicroseconds
Virtual parts updates from ZooKeeper microsecondsSharedMergeTreeVirtualPartsUpdatesFromZooKeeperPartitions
Number of partitions discovered from ZooKeeper during virtual parts discovery because their stat was not present or stale in the discovery cacheSharedMergeTreeVirtualPartsUpdatesFromZooKeeperPartitionsFromCache
Number of partitions skipped during virtual parts discovery because their stat was up to date in the discovery cacheSharedMergeTreeVirtualPartsUpdatesLeaderFailedElection
Virtual parts updates leader election failedSharedMergeTreeVirtualPartsUpdatesLeaderSuccessfulElection
Virtual parts updates leader election successfulSharedMergeTreeVirtualPartsUpdatesPeerNotFound
Virtual updates from peer failed because no one foundSharedPartsLockHoldMicroseconds
Total time spent holding shared data parts lock in MergeTree tablesSharedPartsLocks
Number of times shared data parts lock has been acquired for MergeTree tablesSharedPartsLockWaitMicroseconds
Total time spent waiting for shared data parts lock in MergeTree tablesSleepFunctionCalls
Number of times a sleep function (sleep, sleepEachRow) has been called.SleepFunctionElapsedMicroseconds
Time spent sleeping in a sleep function (sleep, sleepEachRow).SleepFunctionMicroseconds
Time set to sleep in a sleep function (sleep, sleepEachRow).SlowRead
Number of reads from a file that were slow. This indicate system overload. Thresholds are controlled by read_backoff_* settings.SoftPageFaults
The number of soft page faults in query execution threads. Soft page fault usually means a miss in the memory allocator cache, which requires a new memory mapping from the OS and subsequent allocation of a page of physical memory.StatelessWorkerCreateLeaseErrors
Number of failed create_lease requests to the stateless worker discovery service (any error or timeout).StatelessWorkerCreateLeaseMicroseconds
Total time spent in create_lease requests to the stateless worker discovery service.StatelessWorkerCreateLeaseRequests
Number of create_lease requests sent to the stateless worker discovery service to acquire workers.StatelessWorkerDiscoveryCreateLeaseErrors
Number of create_lease requests the stateless worker discovery service answered with an error.StatelessWorkerDiscoveryCreateLeaseMicroseconds
Total time the stateless worker discovery service spent handling create_lease requests.StatelessWorkerDiscoveryCreateLeaseRequests
Number of create_lease requests handled by the stateless worker discovery service.StatelessWorkerDiscoveryCreateWorkerErrors
Number of create_worker requests the stateless worker discovery service answered with an error.StatelessWorkerDiscoveryCreateWorkerMicroseconds
Total time the stateless worker discovery service spent handling create_worker requests.StatelessWorkerDiscoveryCreateWorkerRequests
Number of create_worker requests handled by the stateless worker discovery service.StatelessWorkerDiscoveryDeleteWorkerErrors
Number of delete_worker requests the stateless worker discovery service answered with an error.StatelessWorkerDiscoveryDeleteWorkerMicroseconds
Total time the stateless worker discovery service spent handling delete_worker requests.StatelessWorkerDiscoveryDeleteWorkerRequests
Number of delete_worker requests handled by the stateless worker discovery service.StatelessWorkerDiscoveryGetLeaseErrors
Number of get_lease requests the stateless worker discovery service answered with an error.StatelessWorkerDiscoveryGetLeaseMicroseconds
Total time the stateless worker discovery service spent handling get_lease requests.StatelessWorkerDiscoveryGetLeaseRequests
Number of get_lease requests handled by the stateless worker discovery service.StatelessWorkerDiscoveryGetWorkerErrors
Number of get_worker requests the stateless worker discovery service answered with an error.StatelessWorkerDiscoveryGetWorkerMicroseconds
Total time the stateless worker discovery service spent handling get_worker requests.StatelessWorkerDiscoveryGetWorkerRequests
Number of get_worker requests handled by the stateless worker discovery service.StatelessWorkerDiscoveryHeartbeatErrors
Number of heartbeat requests the stateless worker discovery service answered with an error.StatelessWorkerDiscoveryHeartbeatMicroseconds
Total time the stateless worker discovery service spent handling heartbeat requests.StatelessWorkerDiscoveryHeartbeatRequests
Number of heartbeat requests handled by the stateless worker discovery service.StatelessWorkerDiscoveryHeartbeatsRejected
Number of heartbeats the stateless worker discovery service rejected because the worker had already been evicted.StatelessWorkerDiscoveryInternalErrors
Number of requests the stateless worker discovery service failed to handle because of an internal error (an exception or a coordination-store failure), rather than rejecting an invalid request.StatelessWorkerDiscoveryKeeperTransactionRetries
Number of write transactions the stateless worker discovery service retried because its coordination store (Keeper) state was modified concurrently.StatelessWorkerDiscoveryListWorkersErrors
Number of list_workers requests the stateless worker discovery service answered with an error.StatelessWorkerDiscoveryListWorkersMicroseconds
Total time the stateless worker discovery service spent handling list_workers requests.StatelessWorkerDiscoveryListWorkersRequests
Number of list_workers requests handled by the stateless worker discovery service.StatelessWorkerDiscoveryTenantErrors
Number of tenant management requests the stateless worker discovery service answered with an error.StatelessWorkerDiscoveryTenantMicroseconds
Total time the stateless worker discovery service spent handling tenant management requests.StatelessWorkerDiscoveryTenantRequests
Number of tenant management requests (create, update, delete, get, or list tenant) handled by the stateless worker discovery service.StatelessWorkerDiscoveryUpdateLeaseErrors
Number of update_lease requests the stateless worker discovery service answered with an error.StatelessWorkerDiscoveryUpdateLeaseMicroseconds
Total time the stateless worker discovery service spent handling update_lease requests.StatelessWorkerDiscoveryUpdateLeaseRequests
Number of update_lease requests handled by the stateless worker discovery service.StatelessWorkerDiscoveryWorkersHeartbeatExpired
Number of stateless workers reclaimed by the stateless worker discovery service because they stopped sending heartbeats.StatelessWorkerDiscoveryWorkersLeased
Number of stateless workers newly leased to tenants by the stateless worker discovery service.StatelessWorkerDiscoveryWorkersLeaseExpired
Number of stateless workers reclaimed by the stateless worker discovery service because their lease expired.StatelessWorkerDiscoveryWorkersUnavailable
Number of stateless workers that tenants requested but the stateless worker discovery service could not provide because no free workers were available or the tenant worker limit was reached.StatelessWorkerGetLeaseErrors
Number of failed get_lease requests to the stateless worker discovery service (any error or timeout).StatelessWorkerGetLeaseMicroseconds
Total time spent in get_lease requests to the stateless worker discovery service.StatelessWorkerGetLeaseRequests
Number of get_lease requests sent to the stateless worker discovery service to refresh the list of leased workers.StatelessWorkerNotAccessible
Number of times a leased stateless worker was found not yet reachable and excluded from query allocation until it passes the accessibility check.StatelessWorkerProvided
Number of stateless workers provided to queries for distributed query execution.StatelessWorkerProvisioningMicroseconds
Total time queries spent waiting for stateless workers to be provisioned.StatelessWorkerProvisioningWaits
Number of times a query had to wait for the stateless worker discovery service to provide workers.StatelessWorkerRequested
Number of stateless workers requested by queries for distributed query execution.StatelessWorkerServerHeartbeatErrors
Number of failed heartbeats from stateless workers to the stateless worker discovery service (any error or timeout).StatelessWorkerServerHeartbeatMicroseconds
Total time stateless workers spent sending heartbeats to the stateless worker discovery service.StatelessWorkerServerHeartbeatRequests
Number of heartbeats a stateless worker sent to the stateless worker discovery service.StatelessWorkerServerLeasesLost
Number of times a stateless worker’s lease ended and it began awaiting shutdown.StatelessWorkerServerRegisterErrors
Number of failed registration requests from stateless workers to the stateless worker discovery service (any error or timeout).StatelessWorkerServerRegisterMicroseconds
Total time stateless workers spent registering with the stateless worker discovery service.StatelessWorkerServerRegisterRequests
Number of registration requests a stateless worker sent to the stateless worker discovery service.StatelessWorkerServerTenantBindings
Number of times a stateless worker was assigned to a tenant to serve its queries.StatelessWorkerTimeToAvailableMicroseconds
Total time (summed across workers) between a leased stateless worker first appearing from the discovery service and it passing the accessibility check to become available for query execution.StatelessWorkerUpdateLeaseErrors
Number of failed update_lease requests to the stateless worker discovery service (any error or timeout).StatelessWorkerUpdateLeaseMicroseconds
Total time spent in update_lease requests to the stateless worker discovery service.StatelessWorkerUpdateLeaseRequests
Number of update_lease requests sent to the stateless worker discovery service to renew worker leases.StorageBufferErrorOnFlush
Number of times a buffer in the ‘Buffer’ table has not been able to flush due to error writing in the destination table.StorageBufferFlush
Number of times a buffer in a ‘Buffer’ table was flushed.StorageBufferLayerLockReadersWaitMilliseconds
Time for waiting for Buffer layer during reading.StorageBufferLayerLockWritersWaitMilliseconds
Time for waiting free Buffer layer to write to (can be used to tune Buffer layers).StorageBufferPassedAllMinThresholds
Number of times a criteria on min thresholds has been reached to flush a buffer in a ‘Buffer’ table.StorageBufferPassedBytesFlushThreshold
Number of times background-only flush threshold on bytes has been reached to flush a buffer in a ‘Buffer’ table. This is expert-only metric. If you read this and you are not an expert, stop reading.StorageBufferPassedBytesMaxThreshold
Number of times a criteria on max bytes threshold has been reached to flush a buffer in a ‘Buffer’ table.StorageBufferPassedRowsFlushThreshold
Number of times background-only flush threshold on rows has been reached to flush a buffer in a ‘Buffer’ table. This is expert-only metric. If you read this and you are not an expert, stop reading.StorageBufferPassedRowsMaxThreshold
Number of times a criteria on max rows threshold has been reached to flush a buffer in a ‘Buffer’ table.StorageBufferPassedTimeFlushThreshold
Number of times background-only flush threshold on time has been reached to flush a buffer in a ‘Buffer’ table. This is expert-only metric. If you read this and you are not an expert, stop reading.StorageBufferPassedTimeMaxThreshold
Number of times a criteria on max time threshold has been reached to flush a buffer in a ‘Buffer’ table.StorageConnectionsCreated
Number of created connections for storagesStorageConnectionsElapsedMicroseconds
Total time spend on creating connections for storagesStorageConnectionsErrors
Number of cases when creation of a connection for storage is failedStorageConnectionsExpired
Number of expired connections for storagesStorageConnectionsPreserved
Number of preserved connections for storagesStorageConnectionsReset
Number of reset connections for storagesStorageConnectionsReused
Number of reused connections for storagesStreamingExchangeConnectionWaitMicroseconds
Time a streaming exchange sink waited for the receiving task to connect.StreamingExchangeDeserializeMicroseconds
Time spent decompressing and deserializing streaming exchange packets into chunks.StreamingExchangeEarlyCloses
How many times a streaming exchange source told its sender to stop before the stream was read to the end, for example because a LIMIT was satisfied. A sender that was already gone is not counted.StreamingExchangePacketsReceived
Data packets read from streaming exchanges, one per chunk plus one end-of-stream packet per stream.StreamingExchangePacketsSent
Data packets written whole to the sockets of streaming exchanges, one per chunk plus one end-of-stream packet per stream. A packet is counted when it is written whole; a packet cut short because the receiver needed no more data is not counted.StreamingExchangeReceiveBytes
Bytes read from the sockets of the streaming exchanges of a distributed query plan.NetworkReceiveBytes does not count them.
StreamingExchangeReceiveWaitMicroseconds
Time a streaming exchange source waited for the next bytes from its sender.StreamingExchangeSendBytes
Bytes written to the sockets of the streaming exchanges of a distributed query plan.NetworkSendBytes does not count them.
StreamingExchangeSendQueueFullMicroseconds
Time a streaming exchange sink did not take chunks because its queue of packets waiting for the socket was full.StreamingExchangeSerializedBytes
Bytes of the Native blocks in streaming exchange packets before compression, counted once per packet where it is serialized. AgainstStreamingExchangeSendBytes this gives the compression ratio of a shuffle or a gather; a broadcast sends every packet to each destination, so there StreamingExchangeSendBytes grows with the number of destinations.
StreamingExchangeSerializeMicroseconds
Time spent serializing and compressing chunks into streaming exchange packets.SummingSortedMilliseconds
Total time spent while summing sorted columnsSuspendSendingQueryToShard
Total count when sending query to shard was suspended when async_query_sending_for_remote is enabled.SynchronousReadWaitMicroseconds
Time spent in waiting for synchronous reads in asynchronous local read.SynchronousRemoteReadWaitMicroseconds
Time spent in waiting for synchronous remote reads.SystemLogErrorOnFlush
Number of times any of the system logs have failed to flush to the corresponding system table. Attempts to flush are repeated.SystemTimeMicroseconds
Total time spent in processing (queries and other tasks) threads executing CPU instructions in OS kernel mode. This is time spent in syscalls, excluding waiting time during blocking syscalls.TableFunctionExecute
Number of table function calls.TextIndexDiscardHint
Number of index granules where a direct reading from the text index was added as hint and was discarded due to low selectivity.TextIndexDiscardPatternScan
Number of times pattern-based dictionary scan in a text index was discarded because the number of posting lists to read exceeded the threshold.TextIndexGenericExclusionSearchAlgorithm
Number of times the generic exclusion search algorithm is used over the text index.TextIndexGenericExclusionSearchStepLimitReached
Number of times the generic exclusion search over the text index reached merge_tree_generic_exclusion_search_max_steps and accepted the remaining mark ranges without further splitting.TextIndexHeaderCacheHits
Number of times a header has been found in the cache.TextIndexHeaderCacheMisses
Number of times a header has not been found in the cache.TextIndexLazyAdvanceCount
Number of advance operations performed in lazy posting list mode.TextIndexLazyBlocksSkippedResolved
Number of packed blocks skipped because the output region was already resolved (all-ones for OR, all-zeros for AND) in lazy posting list mode.TextIndexLazyBruteForceIntersections
Number of brute-force intersections performed in lazy posting list mode.TextIndexLazyLeapfrogIntersections
Number of leapfrog intersections performed in lazy posting list mode.TextIndexLazyPackedBlocksDecoded
Number of packed blocks decoded in lazy posting list mode.TextIndexLazySegmentsBuilt
Number of segments actually read and decoded (cache misses) in lazy posting list mode.TextIndexLazySegmentsPrepared
Number of segments prepared (read from disk or cached) in lazy posting list mode.TextIndexLazySegmentsSkippedDense
Number of fully-dense segments padded as a whole (memset for OR, increment for AND) instead of decoding blocks, in lazy posting list mode.TextIndexLazySegmentsSkippedResolved
Number of segments skipped because the output region was already resolved (all-ones for OR, all-zeros for AND) in lazy posting list mode.TextIndexPhraseCandidates
Candidate rows (postings intersection) examined by candidate-driven phrase search.TextIndexPhraseFallbacks
Number of times a phrase query skipped the text index because the phrase was estimated to match more rows than ‘text_index_hint_max_selectivity’, and was evaluated on the column instead.TextIndexPhraseMatchMicroseconds
Total time spent matching phrase positions over candidate rows.TextIndexPhraseSearches
Number of phrase searches executed over a text index. Repeated searches of the same phrase in the same part are served from the cache and are not counted.TextIndexPositionsBlocksRead
Number of position blocks read to match phrase queries over a text index. A block spanning two candidate windows is counted once per read.TextIndexPositionsBlocksTotal
Number of total position blocks of the phrase tokens before selecting which ones to read.TextIndexPositionsBytesRead
Bytes of position blocks read to match phrase queries over a text index.TextIndexPositionsDecodeMicroseconds
Total time spent decoding text index position lists (.pos) for phrase search.TextIndexPostingsCacheHits
Number of times a text index posting list has been found in the cache.TextIndexPostingsCacheMisses
Number of times a text index posting list has not been found in the cache.TextIndexReadDictionaryBlocks
Number of times a text index dictionary block has been read from disk.TextIndexReaderTotalMicroseconds
Total time spent reading the text index.TextIndexReadGranulesMicroseconds
Total time spent reading and analyzing granules of the text index.TextIndexReadPostings
Number of times a posting list has been read from the text index.TextIndexReadSparseIndexBlocks
Number of times a sparse index block has been read from the text index.TextIndexTemporarySegmentsWritten
Number of temporary segments written while building text indexes.TextIndexTokensCacheHits
Number of times a text index token info has been found in the cache.TextIndexTokensCacheMisses
Number of times a text index token info has not been found in the cache.TextIndexTokensCacheNegativeHits
Number of times an absent text index token has been found in the cache.TextIndexTokensCacheNegativeMisses
Number of absent text index tokens added to the cache after a dictionary lookup.TextIndexUsedEmbeddedPostings
Number of times a posting list embedded in the dictionary has been used.TextIndexUseHint
Number of index granules where a direct reading from the text index was added as hint and was used.ThreadPoolReaderPageCacheHit
Number of times the read inside ThreadPoolReader was done from the page cache.ThreadPoolReaderPageCacheHitBytes
Number of bytes read inside ThreadPoolReader when it was done from the page cache.ThreadPoolReaderPageCacheHitElapsedMicroseconds
Time spent reading data from page cache in ThreadPoolReader.ThreadPoolReaderPageCacheMiss
Number of times the read inside ThreadPoolReader was not done from page cache and was hand off to thread pool.ThreadPoolReaderPageCacheMissBytes
Number of bytes read inside ThreadPoolReader when read was not done from page cache and was hand off to thread pool.ThreadPoolReaderPageCacheMissElapsedMicroseconds
Time spent reading data inside the asynchronous job in ThreadPoolReader - when read was not done from the page cache.ThreadpoolReaderPrepareMicroseconds
Time spent on preparation (e.g. call to reader seek() method)ThreadpoolReaderReadBytes
Bytes read from a threadpool task in asynchronous readingThreadpoolReaderSubmit
Bytes read from a threadpool task in asynchronous readingThreadpoolReaderSubmitLookupInCacheMicroseconds
How much time we spent checking if content is cachedThreadpoolReaderSubmitReadSynchronously
How many times we haven’t scheduled a task on the thread pool and read synchronously insteadThreadpoolReaderSubmitReadSynchronouslyBytes
How many bytes were read synchronouslyThreadpoolReaderSubmitReadSynchronouslyMicroseconds
How much time we spent reading synchronouslyThreadpoolReaderTaskMicroseconds
Time spent getting the data in asynchronous readingThrottlerSleepMicroseconds
Total time a query was sleeping to conform all throttling settings.TinyS3Clients
Number of S3 clients copies which reuse an existing auth provider from another client.UncompressedCacheHits
Number of times a block of data has been found in the uncompressed cache (and decompression was avoided).UncompressedCacheMisses
Number of times a block of data has not been found in the uncompressed cache (and required decompression).UncompressedCacheWeightLost
Number of bytes evicted from the uncompressed cache.UniqueKeyBitmapLoadMicroseconds
Time spent reading and deserializing UNIQUE KEY delete-bitmap sidecar files on a DeleteBitmapCache miss.UniqueKeyBitmapUpdates
Number of UNIQUE KEY delete-bitmap sidecar files written (one per supersession / DELETE commit).UniqueKeyIndexCacheHits
Number of times an entry has been found in the UNIQUE KEY index cache, so we didn’t have to load an SST block.UniqueKeyIndexCacheLookupMicroseconds
Wall-clock time insideUniqueKeyIndexCache::Lookup + UniqueKeyIndexCache::Insert (ClickHouse-side CacheBase adapter for the RocksDB block cache).
UniqueKeyIndexCacheMisses
Number of times an entry has not been found in the UNIQUE KEY index cache, so we had to load an SST block from disk.UniqueKeyLoadTimeSSTRebuildCount
Number of UNIQUE KEY parts whoseunique_key_index.sst was rebuilt at load time after the crash-before-flush window.
UniqueKeyLoadTimeSSTRebuildMicroseconds
Total time spent rebuildingunique_key_index.sst at load time (sequential read of UK columns + SST write).
UniqueKeySSTWriteMicroseconds
Total wall-clock time spent inside anSSTIndexWriter lifetime — covers SST Open, every addEncoded Put, and Finish + WriteBuffer finalize (+ fsync) in finish (the SST bytes are streamed straight into the part storage’s WriteBuffer). Excludes work the static helpers do before constructing the writer (encode + non-prefix-path sort). Emitted once per writer.
USearchAddComputedDistances
Number of times distance was computed when adding vectors to usearch indexes.USearchAddCount
Number of vectors added to usearch indexes.USearchAddVisitedMembers
Number of nodes visited when adding vectors to usearch indexes.USearchSearchComputedDistances
Number of times distance was computed when searching usearch indexes.USearchSearchCount
Number of search operations performed in usearch indexes.USearchSearchVisitedMembers
Number of nodes visited when searching in usearch indexes.UserThrottlerBytes
Bytes passed through ‘max_network_bandwidth_for_user’ throttler.UserThrottlerSleepMicroseconds
Total time a query was sleeping to conform ‘max_network_bandwidth_for_user’ throttling.UserTimeMicroseconds
Total time spent in processing (queries and other tasks) threads executing CPU instructions in user mode. This includes time CPU pipeline was stalled due to main memory access, cache misses, branch mispredictions, hyper-threading, etc.VectorSimilarityIndexCacheHits
Number of times an index granule has been found in the vector index cache.VectorSimilarityIndexCacheMisses
Number of times an index granule has not been found in the vector index cache and had to be read from disk.VectorSimilarityIndexCacheWeightLost
Approximate number of bytes evicted from the vector index cache.VersionedCollapsingSortedMilliseconds
Total time spent while version collapsing sorted columnsWaitMarksLoadMicroseconds
Time spent loading marksWaitPrefetchTaskMicroseconds
Time spend waiting for prefetched readerWasmDeserializationMicroseconds
Time spent executing WebAssembly codeWasmGuestExecuteMicroseconds
Time spent executing WebAssembly codeWasmMemoryAllocated
Total memory allocated for WebAssembly compartmentsWasmModuleInstatiate
Number of WebAssembly compartments createdWasmSerializationMicroseconds
Time spent executing WebAssembly codeWasmTotalExecuteMicroseconds
Time spent executing WebAssembly codeWriteBufferFromFileDescriptorWrite
Number of writes (write/pwrite) to a file descriptor. Does not include sockets.WriteBufferFromFileDescriptorWriteBytes
Number of bytes written to file descriptors. If the file is compressed, this will show compressed data size.WriteBufferFromFileDescriptorWriteFailed
Number of times the write (write/pwrite) to a file descriptor have failed.WriteBufferFromHTTPBytes
Total size of payload bytes received and sent by WriteBufferFromHTTP. Doesn’t include HTTP headers.WriteBufferFromHTTPRequestsSent
Number of HTTP requests sent by WriteBufferFromHTTPWriteBufferFromS3Bytes
Bytes written to S3.WriteBufferFromS3Microseconds
Time spent on writing to S3.WriteBufferFromS3RequestsErrors
Number of exceptions while writing to S3.WriteBufferFromS3WaitInflightLimitMicroseconds
Time spent on waiting while some of the current requests are done when its number reached the limit defined by s3_max_inflight_parts_for_one_file.ZooKeeperBytesReceived
Number of bytes received over network while communicating with ZooKeeper.ZooKeeperBytesSent
Number of bytes send over network while communicating with ZooKeeper.ZooKeeperCheck
Number of ‘check’ requests to ZooKeeper. Usually they don’t make sense in isolation, only as part of a complex transaction.ZooKeeperClose
Number of times connection with ZooKeeper has been closed voluntary.ZooKeeperCreate
Number of ‘create’ requests to ZooKeeper.ZooKeeperExists
Number of ‘exists’ requests to ZooKeeper.ZooKeeperGet
Number of ‘get’ requests to ZooKeeper.ZooKeeperGetACL
Number of ‘getACL’ requests to ZooKeeper.ZooKeeperHardwareExceptions
Number of exceptions while working with ZooKeeper related to network (connection loss or similar).ZooKeeperInit
Number of times connection with ZooKeeper has been established.ZooKeeperList
Number of ‘list’ (getChildren) requests to ZooKeeper.ZooKeeperListRecursive
Number of ‘listRecursive’ requests to ZooKeeper.ZooKeeperMulti
Number of ‘multi’ requests to ZooKeeper (compound transactions).ZooKeeperMultiRead
Number of read ‘multi’ requests to ZooKeeper (compound transactions).ZooKeeperMultiWrite
Number of write ‘multi’ requests to ZooKeeper (compound transactions).ZooKeeperOtherExceptions
Number of exceptions while working with ZooKeeper other than ZooKeeperUserExceptions and ZooKeeperHardwareExceptions.ZooKeeperReconfig
Number of ‘reconfig’ requests to ZooKeeper.ZooKeeperRemove
Number of ‘remove’ requests to ZooKeeper.ZooKeeperSet
Number of ‘set’ requests to ZooKeeper.ZooKeeperSync
Number of ‘sync’ requests to ZooKeeper. These requests are rarely needed or usable.ZooKeeperTransactions
Number of ZooKeeper operations, which include both read and write operations as well as multi-transactions.ZooKeeperUserExceptions
Number of exceptions while working with ZooKeeper related to the data (no node, bad version or similar).ZooKeeperWaitMicroseconds
Number of microseconds spent waiting for responses from ZooKeeper after creating a request, summed across all the requesting threads.ZooKeeperWatchCallbackDurationMicroseconds
Total time spent inside ZooKeeper watch callbacks (network thread).ZooKeeperWatchCallbackErrors
Number of exceptions thrown from ZooKeeper watch callbacks.ZooKeeperWatchResponse
Number of times watch notification has been received from ZooKeeper.ZooKeeperWatchTriggeredBackupCoordination
Number of watch notifications dispatched to backup / restore coordination.ZooKeeperWatchTriggeredClusterDiscovery
Number of watch notifications dispatched to ClusterDiscovery.ZooKeeperWatchTriggeredDistributedDDL
Number of watch notifications dispatched to DDLWorker queue.ZooKeeperWatchTriggeredMaterializedViewRefresh
Number of watch notifications dispatched to MaterializedView refresh coordination.ZooKeeperWatchTriggeredObjectStorageQueue
Number of watch notifications dispatched to ObjectStorageQueue / S3Queue.ZooKeeperWatchTriggeredOther
Number of watch notifications dispatched to uncategorized callbacks.ZooKeeperWatchTriggeredReplicatedAccessControl
Number of watch notifications dispatched to ReplicatedAccessStorage.ZooKeeperWatchTriggeredReplicatedMergeTreeLeaderElection
Number of watch notifications dispatched to ReplicatedMergeTree leader election / replica locks.ZooKeeperWatchTriggeredReplicatedMergeTreeLog
Number of watch notifications dispatched to ReplicatedMergeTree log entries.ZooKeeperWatchTriggeredReplicatedMergeTreeMutations
Number of watch notifications dispatched to ReplicatedMergeTree mutations.ZooKeeperWatchTriggeredReplicatedMergeTreeReplicaSync
Number of watch notifications dispatched while waiting for replicas to process log entries (SYNC REPLICA, ALTER, mutations).ZooKeeperWatchTriggeredUserDefinedSQLObjects
Number of watch notifications dispatched to UserDefinedSQLObjectsZooKeeperStorage.ZooKeeperWatchTriggeredWorkloadEntity
Number of watch notifications dispatched to WorkloadEntityKeeperStorage.KeeperLogsEntryReadFromCommitCache
Retired event that counted log entries read from the Keeper commit-log cache.KeeperLogsPrefetchedEntries
Retired event that counted Keeper log entries prefetched from the changelog file.NaiveBayesClassifierModelsAllocatedBytes
Retired event that counted bytes allocated for Naive Bayes Classifier models.NaiveBayesClassifierModelsLoaded
Retired event that counted loaded Naive Bayes Classifier models.ReaderExecutorRequestedBytes
Retired event that counted useful bytes delivered byReaderExecutor read requests.
Columns
event(String) — Event name.value(UInt64) — Number of events occurred.description(String) — Event description.
name— Alias forevent.
Example
See Also
- system.asynchronous_metrics — Contains periodically calculated metrics.
- system.metrics — Contains instantly calculated metrics.
- system.metric_log — Contains a history of metrics values from tables
system.metricsandsystem.events. - Monitoring — Base concepts of ClickHouse monitoring.