min_chunk_bytes_for_parallel_parsing
The minimum chunk size in bytes, which each thread will parse in parallel.min_compress_block_size
For MergeTree tables. In order to reduce latency when processing queries, a block is compressed when writing the next mark if its size is at leastmin_compress_block_size. By default, 65,536.
The actual size of the block, if the uncompressed data is less than max_compress_block_size, is no less than this value and no less than the volume of data for one mark.
Let’s look at an example. Assume that index_granularity was set to 8192 during table creation.
We are writing a UInt32-type column (4 bytes per value). When writing 8192 rows, the total will be 32 KB of data. Since min_compress_block_size = 65,536, a compressed block will be formed for every two marks.
We are writing a URL column with the String type (average size of 60 bytes per value). When writing 8192 rows, the average will be slightly less than 500 KB of data. Since this is more than 65,536, a compressed block will be formed for each mark. In this case, when reading data from the disk in the range of a single mark, extra data won’t be decompressed.
This is an expert-level setting, and you shouldn’t change it if you’re just getting started with ClickHouse.
min_filtered_ratio_for_lazy_final
Minimum ratio of marks filtered by index analysis for lazy FINAL optimization. If less than this fraction of marks is filtered, falls back to normal FINAL. Value 0 disables this check.min_hit_rate_to_use_consecutive_keys_optimization
Minimal hit rate of a cache which is used for consecutive keys optimization in aggregation to keep it enabledmin_marks_per_part_to_load_statistics
The minimum number of marks left to read after the primary key analysis, per data part whose statistics are not in memory yet, for column statistics to be loaded for pruning parts and for ordering the conditions ofPREWHERE.
Loading statistics costs a few reads per data part, and on a cold start it can take longer than reading the data itself when the primary key already narrows the read to a small number of granules. If fewer marks than this setting multiplied by the number of parts whose statistics would be loaded are left to read, statistics are not loaded for these optimizations: parts are not pruned by statistics, and PREWHERE conditions are ordered by the sizes of their columns, as for tables without statistics. Statistics that are already in memory are used regardless of the size of the read.
Answering min, max and count from statistics (see use_statistics_for_min_max_aggregation) replaces the read and is not affected by this setting.
Possible values:
- 0 — Load statistics regardless of the size of the read.
- Positive integer — The number of marks per data part.
min_os_cpu_wait_time_ratio_to_throw
Min ratio between OS CPU wait (OSCPUWaitMicroseconds metric) and busy (OSCPUVirtualTimeMicroseconds metric) times to consider rejecting queries. Linear interpolation between min and max ratio is used to calculate the probability, the probability is 0 at this point.min_outstreams_per_resize_after_split
Specifies the minimum number of output streams of aResize or StrictResize processor after the split is performed during pipeline generation. If the resulting number of streams is less than this value, the split operation will not occur.
What is a Resize Node
AResize node is a processor in the query pipeline that adjusts the number of data streams flowing through the pipeline. It can either increase or decrease the number of streams to balance the workload across multiple threads or processors. For example, if a query requires more parallelism, the Resize node can split a single stream into multiple streams. Conversely, it can merge multiple streams into fewer streams to consolidate data processing.
The Resize node ensures that data is evenly distributed across streams, maintaining the structure of the data blocks. This helps optimize resource utilization and improve query performance.
Why the Resize Node Needs to Be Split
During pipeline execution, ExecutingGraph::Node::status_mutex of the centrally-hubbedResize node is heavily contended especially in high-core-count environments, and this contention leads to:
- Increased latency for ExecutingGraph::updateNode, directly impacting query performance.
- Excessive CPU cycles are wasted in spin-lock contention (native_queued_spin_lock_slowpath), degrading efficiency.
- Reduced CPU utilization, limiting parallelism and throughput.
How the Resize Node Gets Split
- The number of output streams is checked to ensure the split could be performed: the output streams of each split processor meet or exceed the
min_outstreams_per_resize_after_splitthreshold. - The
Resizenode is divided into smallerResizenodes with equal count of ports, each handling a subset of input and output streams. - Each group is processed independently, reducing the lock contention.
Splitting Resize Node with Arbitrary Inputs/Outputs
In some cases, where the inputs/outputs are indivisible by the number of splitResize nodes, some inputs are connected to NullSources and some outputs are connected to NullSinks. This allows the split to occur without affecting the overall data flow.
Purpose of the Setting
Themin_outstreams_per_resize_after_split setting ensures that the splitting of Resize nodes is meaningful and avoids creating too few streams, which could lead to inefficient parallel processing. By enforcing a minimum number of output streams, this setting helps maintain a balance between parallelism and overhead, optimizing query execution in scenarios involving stream splitting and merging.
Disabling the Setting
To disable the split ofResize nodes, set this setting to 0. This will prevent the splitting of Resize nodes during pipeline generation, allowing them to retain their original structure without division into smaller nodes.
min_rows_for_packed_string_keys_in_hash_join
The minimum estimated number of rows in the right side of a join forenable_packed_string_keys_in_hash_join to apply. Smaller joins gain little from it, and can be slower with long keys.
The estimate is the number of right-side rows in an earlier run of the same join, if collect_hash_table_stats_during_joins is enabled. Otherwise it is the query optimizer’s estimate. parallel_hash_join_threshold uses the same estimate. A join without an estimate always uses enable_packed_string_keys_in_hash_join.
0 means no minimum.
min_rows_ratio_for_hash_join_row_store
Minimum estimated ratio of join output rows to build-side rows to enable transforming hash join payload to row-major. 0 means the transformation is always allowed.min_string_column_bytes_for_hash_join_row_store
Minimum estimated size of a build-sideString column, the estimated number of rows multiplied by the average value size from statistics, to store it in the hash join row store. 0 means String columns are not stored in the row store.