2020 Changelog
ClickHouse release 20.12
ClickHouse release v20.12.5.14-stable, 2020-12-28
Bug Fix
- Disable write with AIO during merges because it can lead to extremely rare data corruption of primary key columns during merge. #18481 (alesapin).
- Fixed
value is too short
error when executingtoType(...)
functions (toDate
,toUInt32
, etc) with argument of typeNullable(String)
. Now such functions returnNULL
on parsing errors instead of throwing exception. Fixes #7673. #18445 (tavplubix). - Restrict merges from wide to compact parts. In case of vertical merge it led to broken result part. #18381 (Anton Popov).
- Fix filling table
system.settings_profile_elements
. This PR fixes #18231. #18379 (Vitaly Baranov). - Fix possible crashes in aggregate functions with combinator
Distinct
, while using two-level aggregation. Fixes #17682. #18365 (Anton Popov). - Fix error when query
MODIFY COLUMN ... REMOVE TTL
does not actually remove column TTL. #18130 (alesapin).
Build/Testing/Packaging Improvement
ClickHouse release v20.12.4.5-stable, 2020-12-24
Bug Fix
- Fixed issue when
clickhouse-odbc-bridge
process is unreachable by server on machines with dual IPv4/IPv6 stack; - Fixed issue when ODBC dictionary updates are performed using malformed queries and/or cause crashes; Possibly closes #14489. #18278 (Denis Glazachev). - Fixed key comparison between Enum and Int types. This fixes #17989. #18214 (Amos Bird).
- Fixed unique key convert crash in
MaterializeMySQL
database engine. This fixes #18186 and fixes #16372 #18211 (Winter Zhang). - Fixed
std::out_of_range: basic_string
in S3 URL parsing. #18059 (Vladimir Chebotarev). - Fixed the issue when some tables not synchronized to ClickHouse from MySQL caused by the fact that convertion MySQL prefix index wasn't supported for MaterializeMySQL. This fixes #15187 and fixes #17912 #17944 (Winter Zhang).
- Fixed the issue when query optimization was producing wrong result if query contains
ARRAY JOIN
. #17887 (sundyli). - Fixed possible segfault in
topK
aggregate function. This closes #17404. #17845 (Maksim Kita). - Fixed empty
system.stack_trace
table when server is running in daemon mode. #17630 (Amos Bird).
ClickHouse release v20.12.3.3-stable, 2020-12-13
Backward Incompatible Change
- Enable
use_compact_format_in_distributed_parts_names
by default (see the documentation for the reference). #16728 (Azat Khuzhin). - Accept user settings related to file formats (e.g.
format_csv_delimiter
) in theSETTINGS
clause when creating a table that usesFile
engine, and use these settings in allINSERT
s andSELECT
s. The file format settings changed in the current user session, or in theSETTINGS
clause of a DML query itself, no longer affect the query. #16591 (Alexander Kuzmenkov).
New Feature
- add
*.xz
compression/decompression support.It enables using*.xz
infile()
function. This closes #8828. #16578 (Abi Palagashvili). - Introduce the query
ALTER TABLE ... DROP|DETACH PART 'part_name'
. #15511 (nvartolomei). - Added new ALTER UPDATE/DELETE IN PARTITION syntax. #13403 (Vladimir Chebotarev).
- Allow formatting named tuples as JSON objects when using JSON input/output formats, controlled by the
output_format_json_named_tuples_as_objects
setting, disabled by default. #17175 (Alexander Kuzmenkov). - Add a possibility to input enum value as it's id in TSV and CSV formats by default. #16834 (Kruglov Pavel).
- Add COLLATE support for Nullable, LowCardinality, Array and Tuple, where nested type is String. Also refactor the code associated with collations in ColumnString.cpp. #16273 (Kruglov Pavel).
- New
tcpPort
function returns TCP port listened by this server. #17134 (Ivan). - Add new math functions:
acosh
,asinh
,atan2
,atanh
,cosh
,hypot
,log1p
,sinh
. #16636 (Konstantin Malanchev). - Possibility to distribute the merges between different replicas. Introduces the
execute_merges_on_single_replica_time_threshold
mergetree setting. #16424 (filimonov). - Add setting
aggregate_functions_null_for_empty
for SQL standard compatibility. This option will rewrite all aggregate functions in a query, adding -OrNull suffix to them. Implements 10273. #16123 (flynn). - Updated DateTime, DateTime64 parsing to accept string Date literal format. #16040 (Maksim Kita).
- Make it possible to change the path to history file in
clickhouse-client
using the--history_file
parameter. #15960 (Maksim Kita).
Bug Fix
- Fix the issue when server can stop accepting connections in very rare cases. #17542 (Amos Bird).
- Fixed
Function not implemented
error when executingRENAME
query inAtomic
database with ClickHouse running on Windows Subsystem for Linux. Fixes #17661. #17664 (tavplubix). - Do not restore parts from WAL if
in_memory_parts_enable_wal
is disabled. #17802 (detailyang). - fix incorrect initialization of
max_compress_block_size
of MergeTreeWriterSettings withmin_compress_block_size
. #17833 (flynn). - Exception message about max table size to drop was displayed incorrectly. #17764 (alexey-milovidov).
- Fixed possible segfault when there is not enough space when inserting into
Distributed
table. #17737 (tavplubix). - Fixed problem when ClickHouse fails to resume connection to MySQL servers. #17681 (Alexander Kazakov).
- In might be determined incorrectly if cluster is circular- (cross-) replicated or not when executing
ON CLUSTER
query due to race condition whenpool_size
> 1. It's fixed. #17640 (tavplubix). - Exception
fmt::v7::format_error
can be logged in background for MergeTree tables. This fixes #17613. #17615 (alexey-milovidov). - When clickhouse-client is used in interactive mode with multiline queries, single line comment was erronously extended till the end of query. This fixes #13654. #17565 (alexey-milovidov).
- Fix alter query hang when the corresponding mutation was killed on the different replica. Fixes #16953. #17499 (alesapin).
- Fix issue when mark cache size was underestimated by clickhouse. It may happen when there are a lot of tiny files with marks. #17496 (alesapin).
- Fix
ORDER BY
with enabled settingoptimize_redundant_functions_in_order_by
. #17471 (Anton Popov). - Fix duplicates after
DISTINCT
which were possible because of incorrect optimization. Fixes #17294. #17296 (li chengxiang). #17439 (Nikolai Kochetov). - Fix crash while reading from
JOIN
table withLowCardinality
types. Fixes #17228. #17397 (Nikolai Kochetov). - fix
toInt256(inf)
stack overflow. Int256 is an experimental feature. Closed #17235. #17257 (flynn). - Fix possible
Unexpected packet Data received from client
error logged for Distributed queries withLIMIT
. #17254 (Azat Khuzhin). - Fix set index invalidation when there are const columns in the subquery. This fixes #17246. #17249 (Amos Bird).
- Fix possible wrong index analysis when the types of the index comparison are different. This fixes #17122. #17145 (Amos Bird).
- Fix ColumnConst comparison which leads to crash. This fixed #17088 . #17135 (Amos Bird).
- Multiple fixed for MaterializeMySQL (experimental feature). Fixes #16923 Fixes #15883 Fix MaterializeMySQL SYNC failure when the modify MySQL binlog_checksum. #17091 (Winter Zhang).
- Fix bug when
ON CLUSTER
queries may hang forever for non-leader ReplicatedMergeTreeTables. #17089 (alesapin). - Fixed crash on
CREATE TABLE ... AS some_table
query whensome_table
was createdAS table_function()
Fixes #16944. #17072 (tavplubix). - Bug unfinished implementation for funciton fuzzBits, related issue: #16980. #17051 (hexiaoting).
- Fix LLVM's libunwind in the case when CFA register is RAX. This is the bug in LLVM's libunwind. We already have workarounds for this bug. #17046 (alexey-milovidov).
- Avoid unnecessary network errors for remote queries which may be cancelled while execution, like queries with
LIMIT
. #17006 (Azat Khuzhin). - Fix
optimize_distributed_group_by_sharding_key
setting (that is disabled by default) for query with OFFSET only. #16996 (Azat Khuzhin). - Fix for Merge tables over Distributed tables with JOIN. #16993 (Azat Khuzhin).
- Fixed wrong result in big integers (128, 256 bit) when casting from double. Big integers support is experimental. #16986 (Mike).
- Fix possible server crash after
ALTER TABLE ... MODIFY COLUMN ... NewType
whenSELECT
haveWHERE
expression on altering column and alter does not finished yet. #16968 (Amos Bird). - Blame info was not calculated correctly in
clickhouse-git-import
. #16959 (alexey-milovidov). - Fix order by optimization with monotonous functions. Fixes #16107. #16956 (Anton Popov).
- Fix optimization of group by with enabled setting
optimize_aggregators_of_group_by_keys
and joins. Fixes #12604. #16951 (Anton Popov). - Fix possible error
Illegal type of argument
for queries withORDER BY
. Fixes #16580. #16928 (Nikolai Kochetov). - Fix strange code in InterpreterShowAccessQuery. #16866 (tavplubix).
- Prevent clickhouse server crashes when using the function
timeSeriesGroupSum
. The function is removed from newer ClickHouse releases. #16865 (filimonov). - Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes #15301. This fixes #13098. #16846 (alexey-milovidov).
- Fix crash when using
any
without any arguments. This is for #16803 . cc @azat. #16826 (Amos Bird). - If no memory can be allocated while writing table metadata on disk, broken metadata file can be written. #16772 (alexey-milovidov).
- Fix trivial query optimization with partition predicate. #16767 (Azat Khuzhin).
- Fix
IN
operator over several columns and tuples with enabledtransform_null_in
setting. Fixes #15310. #16722 (Anton Popov). - Return number of affected rows for INSERT queries via MySQL protocol. Previously ClickHouse used to always return 0, it's fixed. Fixes #16605. #16715 (Winter Zhang).
- Fix remote query failure when using 'if' suffix aggregate function. Fixes #16574 Fixes #16231 #16610 (Winter Zhang).
- Fix inconsistent behavior caused by
select_sequential_consistency
for optimized trivial count query and system.tables. #16309 (Hao Chen).
Improvement
- Remove empty parts after they were pruned by TTL, mutation, or collapsing merge algorithm. #16895 (Anton Popov).
- Enable compact format of directories for asynchronous sends in Distributed tables:
use_compact_format_in_distributed_parts_names
is set to 1 by default. #16788 (Azat Khuzhin). - Abort multipart upload if no data was written to S3. #16840 (Pavel Kovalenko).
- Reresolve the IP of the
format_avro_schema_registry_url
in case of errors. #16985 (filimonov). - Mask password in data_path in the system.distribution_queue. #16727 (Azat Khuzhin).
- Throw error when use column transformer replaces non existing column. #16183 (hexiaoting).
- Turn off parallel parsing when there is no enough memory for all threads to work simultaneously. Also there could be exceptions like "Memory limit exceeded" when somebody will try to insert extremely huge rows (> min_chunk_bytes_for_parallel_parsing), because each piece to parse has to be independent set of strings (one or more). #16721 (Nikita Mikhaylov).
- Install script should always create subdirs in config folders. This is only relevant for Docker build with custom config. #16936 (filimonov).
- Correct grammar in error message in JSONEachRow, JSONCompactEachRow, and RegexpRow input formats. #17205 (nico piderman).
- Set default
host
andport
parameters forSOURCE(CLICKHOUSE(...))
to current instance and set defaultuser
value to'default'
. #16997 (vdimir). - Throw an informative error message when doing
ATTACH/DETACH TABLE <DICTIONARY>
. Before this PR,detach table <dict>
works but leads to an ill-formed in-memory metadata. #16885 (Amos Bird). - Add cutToFirstSignificantSubdomainWithWWW(). #16845 (Azat Khuzhin).
- Server refused to startup with exception message if wrong config is given (
metric_log
.collect_interval_milliseconds
is missing). #16815 (Ivan). - Better exception message when configuration for distributed DDL is absent. This fixes #5075. #16769 (Nikita Mikhaylov).
- Usability improvement: better suggestions in syntax error message when
CODEC
expression is misplaced inCREATE TABLE
query. This fixes #12493. #16768 (alexey-milovidov). - Remove empty directories for async INSERT at start of Distributed engine. #16729 (Azat Khuzhin).
- Workaround for use S3 with nginx server as proxy. Nginx currenty does not accept urls with empty path like
http://domain.com?delete
, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, likehttp://domain.com/?delete
. #16709 (ianton-ru). - Allow
reinterpretAs*
functions to work for integers and floats of the same size. Implements 16640. #16657 (flynn). - Now,
<auxiliary_zookeepers>
configuration can be changed inconfig.xml
and reloaded without server startup. #16627 (Amos Bird). - Support SNI in https connections to remote resources. This will allow to connect to Cloudflare servers that require SNI. This fixes #10055. #16252 (alexey-milovidov).
- Make it possible to connect to
clickhouse-server
secure endpoint which requires SNI. This is possible whenclickhouse-server
is hosted behind TLS proxy. #16938 (filimonov). - Fix possible stack overflow if a loop of materialized views is created. This closes #15732. #16048 (alexey-milovidov).
- Simplify the implementation of background tasks processing for the MergeTree table engines family. There should be no visible changes for user. #15983 (alesapin).
- Improvement for MaterializeMySQL (experimental feature). Throw exception about right sync privileges when MySQL sync user has error privileges. #15977 (TCeason).
- Made
indexOf()
use BloomFilter. #14977 (achimbab).
Performance Improvement
- Use Floyd-Rivest algorithm, it is the best for the ClickHouse use case of partial sorting. Bechmarks are in https://github.com/danlark1/miniselect and here. #16825 (Danila Kutenin).
- Now
ReplicatedMergeTree
tree engines family uses a separate thread pool for replicated fetches. Size of the pool limited by settingbackground_fetches_pool_size
which can be tuned with a server restart. The default value of the setting is 3 and it means that the maximum amount of parallel fetches is equal to 3 (and it allows to utilize 10G network). Fixes #520. #16390 (alesapin). - Fixed uncontrolled growth of the state of
quantileTDigest
. #16680 (hrissan). - Add
VIEW
subquery description toEXPLAIN
. Limit push down optimisation forVIEW
. Add local replicas ofDistributed
to query plan. #14936 (Nikolai Kochetov). - Fix optimize_read_in_order/optimize_aggregation_in_order with max_threads > 0 and expression in ORDER BY. #16637 (Azat Khuzhin).
- Fix performance of reading from
Merge
tables over huge number ofMergeTree
tables. Fixes #7748. #16988 (Anton Popov). - Now we can safely prune partitions with exact match. Useful case: Suppose table is partitioned by
intHash64(x) % 100
and the query has condition onintHash64(x) % 100
verbatim, not on x. #16253 (Amos Bird).
Experimental Feature
Build/Testing/Packaging Improvement
- Improvements in test coverage building images. #17233 (alesapin).
- Update embedded timezone data to version 2020d (also update cctz to the latest master). #17204 (filimonov).
- Fix UBSan report in Poco. This closes #12719. #16765 (alexey-milovidov).
- Do not instrument 3rd-party libraries with UBSan. #16764 (alexey-milovidov).
- Fix UBSan report in cache dictionaries. This closes #12641. #16763 (alexey-milovidov).
- Fix UBSan report when trying to convert infinite floating point number to integer. This closes #14190. #16677 (alexey-milovidov).
ClickHouse release 20.11
ClickHouse release v20.11.7.16-stable, 2021-03-02
Improvement
- Explicitly set uid / gid of clickhouse user & group to the fixed values (101) in clickhouse-server images. #19096 (filimonov).
Bug Fix
- BloomFilter index crash fix. Fixes #19757. #19884 (Maksim Kita).
- Deadlock was possible if system.text_log is enabled. This fixes #19874. #19875 (alexey-milovidov).
- In previous versions, unusual arguments for function arrayEnumerateUniq may cause crash or infinite loop. This closes #19787. #19788 (alexey-milovidov).
- Fixed stack overflow when using accurate comparison of arithmetic type with string type. #19773 (tavplubix).
- Fix a segmentation fault in
bitmapAndnot
function. Fixes #19668. #19713 (Maksim Kita). - Some functions with big integers may cause segfault. Big integers is experimental feature. This closes #19667. #19672 (alexey-milovidov).
- Fix wrong result of function
neighbor
forLowCardinality
argument. Fixes #10333. #19617 (Nikolai Kochetov). - Fix use-after-free of the CompressedWriteBuffer in Connection after disconnect. #19599 (Azat Khuzhin).
DROP/DETACH TABLE table ON CLUSTER cluster SYNC
query might hang, it's fixed. Fixes #19568. #19572 (tavplubix).- Query CREATE DICTIONARY id expression fix. #19571 (Maksim Kita).
- Fix SIGSEGV with merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read=0/UINT64_MAX. #19528 (Azat Khuzhin).
- Buffer overflow (on memory read) was possible if
addMonth
function was called with specifically crafted arguments. This fixes #19441. This fixes #19413. #19472 (alexey-milovidov). - Mark distributed batch as broken in case of empty data block in one of files. #19449 (Azat Khuzhin).
- Fix possible buffer overflow in Uber H3 library. See https://github.com/uber/h3/issues/392. This closes #19219. #19383 (alexey-milovidov).
- Fix system.parts _state column (LOGICAL_ERROR when querying this column, due to incorrect order). #19346 (Azat Khuzhin).
- Fix error
Cannot convert column now64() because it is constant but values of constants are different in source and result
. Continuation of #7156. #19316 (Nikolai Kochetov). - Fix bug when concurrent
ALTER
andDROP
queries may hang while processing ReplicatedMergeTree table. #19237 (alesapin). - Fix infinite reading from file in
ORC
format (was introduced in #10580). Fixes #19095. #19134 (Nikolai Kochetov). - Fix startup bug when clickhouse was not able to read compression codec from
LowCardinality(Nullable(...))
and throws exceptionAttempt to read after EOF
. Fixes #18340. #19101 (alesapin). - Fixed
There is no checkpoint
error when inserting data through http interface usingTemplate
orCustomSeparated
format. Fixes #19021. #19072 (tavplubix). - Restrict
MODIFY TTL
queries forMergeTree
tables created in old syntax. Previously the query succeeded, but actually it had no effect. #19064 (Anton Popov). - Make sure
groupUniqArray
returns correct type for argument of Enum type. This closes #17875. #19019 (alexey-milovidov). - Fix possible error
Expected single dictionary argument for function
if use functionignore
withLowCardinality
argument. Fixes #14275. #19016 (Nikolai Kochetov). - Fix inserting of
LowCardinality
column to table withTinyLog
engine. Fixes #18629. #19010 (Nikolai Kochetov). - Disable
optimize_move_functions_out_of_any
because optimization is not always correct. This closes #18051. This closes #18973. #18981 (alexey-milovidov). - Fixed very rare deadlock at shutdown. #18977 (tavplubix).
- Fix bug when mutation with some escaped text (like
ALTER ... UPDATE e = CAST('foo', 'Enum8(\'foo\' = 1')
serialized incorrectly. Fixes #18878. #18944 (alesapin). - Attach partition should reset the mutation. #18804. #18935 (fastio).
- Fix possible hang at shutdown in clickhouse-local. This fixes #18891. #18893 (alexey-milovidov).
- Fix *If combinator with unary function and Nullable types. #18806 (Azat Khuzhin).
- Asynchronous distributed INSERTs can be rejected by the server if the setting
network_compression_method
is globally set to non-default value. This fixes #18741. #18776 (alexey-milovidov). - Fixed
Attempt to read after eof
error when trying toCAST
NULL
fromNullable(String)
toNullable(Decimal(P, S))
. Now functionCAST
returnsNULL
when it cannot parse decimal from nullable string. Fixes #7690. #18718 (Winter Zhang). - Fix Logger with unmatched arg size. #18717 (sundyli).
- Add FixedString Data type support. I'll get this exception "Code: 50, e.displayText() = DB::Exception: Unsupported type FixedString(1)" when replicating data from MySQL to ClickHouse. This patch fixes bug #18450 Also fixes #6556. #18553 (awesomeleo).
- Fix possible
Pipeline stuck
error while usingORDER BY
after subquery withRIGHT
orFULL
join. #18550 (Nikolai Kochetov). - Fix bug which may lead to
ALTER
queries hung after corresponding mutation kill. Found by thread fuzzer. #18518 (alesapin). - Disable write with AIO during merges because it can lead to extremely rare data corruption of primary key columns during merge. #18481 (alesapin).
- Disable constant folding for subqueries on the analysis stage, when the result cannot be calculated. #18446 (Azat Khuzhin).
- Fixed
value is too short
error when executingtoType(...)
functions (toDate
,toUInt32
, etc) with argument of typeNullable(String)
. Now such functions returnNULL
on parsing errors instead of throwing exception. Fixes #7673. #18445 (tavplubix). - Restrict merges from wide to compact parts. In case of vertical merge it led to broken result part. #18381 (Anton Popov).
- Fix filling table
system.settings_profile_elements
. This PR fixes #18231. #18379 (Vitaly Baranov). - Fix index analysis of binary functions with constant argument which leads to wrong query results. This fixes #18364. #18373 (Amos Bird).
- Fix possible crashes in aggregate functions with combinator
Distinct
, while using two-level aggregation. Fixes #17682. #18365 (Anton Popov). SELECT count() FROM table
now can be executed if only one any column can be selected from thetable
. This PR fixes #10639. #18233 (Vitaly Baranov).SELECT JOIN
now requires theSELECT
privilege on each of the joined tables. This PR fixes #17654. #18232 (Vitaly Baranov).- Fix possible incomplete query result while reading from
MergeTree*
in case of read backoff (message<Debug> MergeTreeReadPool: Will lower number of threads
in logs). Was introduced in #16423. Fixes #18137. #18216 (Nikolai Kochetov). - Fix error when query
MODIFY COLUMN ... REMOVE TTL
does not actually remove column TTL. #18130 (alesapin). - Fix indeterministic functions with predicate optimizer. This fixes #17244. #17273 (Winter Zhang).
- Mutation might hang waiting for some non-existent part after
MOVE
orREPLACE PARTITION
or, in rare cases, afterDETACH
orDROP PARTITION
. It's fixed. #15537 (tavplubix).
Build/Testing/Packaging Improvement
ClickHouse release v20.11.6.6-stable, 2020-12-24
Bug Fix
- Fixed issue when
clickhouse-odbc-bridge
process is unreachable by server on machines with dualIPv4/IPv6 stack
and fixed issue when ODBC dictionary updates are performed using malformed queries and/or cause crashes. This possibly closes #14489. #18278 (Denis Glazachev). - Fixed key comparison between Enum and Int types. This fixes #17989. #18214 (Amos Bird).
- Fixed unique key convert crash in
MaterializeMySQL
database engine. This fixes #18186 and fixes #16372 #18211 (Winter Zhang). - Fixed
std::out_of_range: basic_string
in S3 URL parsing. #18059 (Vladimir Chebotarev). - Fixed the issue when some tables not synchronized to ClickHouse from MySQL caused by the fact that convertion MySQL prefix index wasn't supported for MaterializeMySQL. This fixes #15187 and fixes #17912 #17944 (Winter Zhang).
- Fixed the issue when query optimization was producing wrong result if query contains
ARRAY JOIN
. #17887 (sundyli). - Fix possible segfault in
topK
aggregate function. This closes #17404. #17845 (Maksim Kita). - Do not restore parts from WAL if
in_memory_parts_enable_wal
is disabled. #17802 (detailyang). - Fixed problem when ClickHouse fails to resume connection to MySQL servers. #17681 (Alexander Kazakov).
- Fixed inconsistent behaviour of
optimize_trivial_count_query
with partition predicate. #17644 (Azat Khuzhin). - Fixed empty
system.stack_trace
table when server is running in daemon mode. #17630 (Amos Bird). - Fixed the behaviour when xxception
fmt::v7::format_error
can be logged in background for MergeTree tables. This fixes #17613. #17615 (alexey-milovidov). - Fixed the behaviour when clickhouse-client is used in interactive mode with multiline queries and single line comment was erronously extended till the end of query. This fixes #13654. #17565 (alexey-milovidov).
- Fixed the issue when server can stop accepting connections in very rare cases. #17542 (alexey-milovidov).
- Fixed alter query hang when the corresponding mutation was killed on the different replica. This fixes #16953. #17499 (alesapin).
- Fixed bug when mark cache size was underestimated by clickhouse. It may happen when there are a lot of tiny files with marks. #17496 (alesapin).
- Fixed
ORDER BY
with enabled settingoptimize_redundant_functions_in_order_by
. #17471 (Anton Popov). - Fixed duplicates after
DISTINCT
which were possible because of incorrect optimization. This fixes #17294. #17296 (li chengxiang). #17439 (Nikolai Kochetov). - Fixed crash while reading from
JOIN
table withLowCardinality
types. This fixes #17228. #17397 (Nikolai Kochetov). - Fixed set index invalidation when there are const columns in the subquery. This fixes #17246 . #17249 (Amos Bird).
- Fixed possible wrong index analysis when the types of the index comparison are different. This fixes #17122. #17145 (Amos Bird).
- Fixed
ColumnConst
comparison which leads to crash. This fixes #17088 . #17135 (Amos Bird). - Fixed bug when
ON CLUSTER
queries may hang forever for non-leaderReplicatedMergeTreeTables
. #17089 (alesapin). - Fixed fuzzer-found bug in funciton
fuzzBits
. This fixes #16980. #17051 (hexiaoting). - Avoid unnecessary network errors for remote queries which may be cancelled while execution, like queries with
LIMIT
. #17006 (Azat Khuzhin). - Fixed wrong result in big integers (128, 256 bit) when casting from double. #16986 (Mike).
- Reresolve the IP of the
format_avro_schema_registry_url
in case of errors. #16985 (filimonov). - Fixed possible server crash after
ALTER TABLE ... MODIFY COLUMN ... NewType
whenSELECT
haveWHERE
expression on altering column and alter does not finished yet. #16968 (Amos Bird). - Blame info was not calculated correctly in
clickhouse-git-import
. #16959 (alexey-milovidov). - Fixed order by optimization with monotonous functions. Fixes #16107. #16956 (Anton Popov).
- Fixed optimization of group by with enabled setting
optimize_aggregators_of_group_by_keys
and joins. This fixes #12604. #16951 (Anton Popov). - Install script should always create subdirs in config folders. This is only relevant for Docker build with custom config. #16936 (filimonov).
- Fixed possible error
Illegal type of argument
for queries withORDER BY
. This fixes #16580. #16928 (Nikolai Kochetov). - Abort multipart upload if no data was written to WriteBufferFromS3. #16840 (Pavel Kovalenko).
- Fixed crash when using
any
without any arguments. This fixes #16803. #16826 (Amos Bird). - Fixed the behaviour when ClickHouse used to always return 0 insted of a number of affected rows for
INSERT
queries via MySQL protocol. This fixes #16605. #16715 (Winter Zhang). - Fixed uncontrolled growth of TDigest. #16680 (hrissan).
- Fixed remote query failure when using suffix
if
in Aggregate function. This fixes #16574 fixes #16231 #16610 (Winter Zhang). - Fixed inconsistent behavior caused by
select_sequential_consistency
for optimized trivial count query and system.tables. #16309 (Hao Chen). - Throw error when use ColumnTransformer replace non exist column. #16183 (hexiaoting).
ClickHouse release v20.11.3.3-stable, 2020-11-13
Bug Fix
- Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes #15301. This fixes #13098. #16846 (alexey-milovidov).
ClickHouse release v20.11.2.1, 2020-11-11
Backward Incompatible Change
- If some
profile
was specified indistributed_ddl
config section, then this profile could overwrite settings ofdefault
profile on server startup. It's fixed, now settings of distributed DDL queries should not affect global server settings. #16635 (tavplubix). - Restrict to use of non-comparable data types (like
AggregateFunction
) in keys (Sorting key, Primary key, Partition key, and so on). #16601 (alesapin). - Remove
ANALYZE
andAST
queries, and make the settingenable_debug_queries
obsolete since now it is the part of full featuredEXPLAIN
query. #16536 (Ivan). - Aggregate functions
boundingRatio
,rankCorr
,retention
,timeSeriesGroupSum
,timeSeriesGroupRateSum
,windowFunnel
were erroneously made case-insensitive. Now their names are made case sensitive as designed. Only functions that are specified in SQL standard or made for compatibility with other DBMS or functions similar to those should be case-insensitive. #16407 (alexey-milovidov). - Make
rankCorr
function return nan on insufficient data #16124. #16135 (hexiaoting). - When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to
Part ... intersects previous part
errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).
New Feature
- Added support of LDAP as a user directory for locally non-existent users. #12736 (Denis Glazachev).
- Add
system.replicated_fetches
table which shows currently running background fetches. #16428 (alesapin). - Added setting
date_time_output_format
. #15845 (Maksim Kita). - Added minimal web UI to ClickHouse. #16158 (alexey-milovidov).
- Allows to read/write Single protobuf message at once (w/o length-delimiters). #15199 (filimonov).
- Added initial OpenTelemetry support. ClickHouse now accepts OpenTelemetry traceparent headers over Native and HTTP protocols, and passes them downstream in some cases. The trace spans for executed queries are saved into the
system.opentelemetry_span_log
table. #14195 (Alexander Kuzmenkov). - Allow specify primary key in column list of
CREATE TABLE
query. This is needed for compatibility with other SQL dialects. #15823 (Maksim Kita). - Implement
OFFSET offset_row_count {ROW | ROWS} FETCH {FIRST | NEXT} fetch_row_count {ROW | ROWS} {ONLY | WITH TIES}
in SELECT query with ORDER BY. This is the SQL-standard way to specifyLIMIT
. #15855 (hexiaoting). errorCodeToName
function - return variable name of the error (useful for analyzing query_log and similar).system.errors
table - shows how many times errors has been happened (respectssystem_events_show_zero_values
). #16438 (Azat Khuzhin).- Added function
untuple
which is a special function which can introduce new columns to the SELECT list by expanding a named tuple. #16242 (Nikolai Kochetov, Amos Bird). - Now we can provide identifiers via query parameters. And these parameters can be used as table objects or columns. #16594 (Amos Bird).
- Added big integers (UInt256, Int128, Int256) and UUID data types support for MergeTree BloomFilter index. Big integers is an experimental feature. #16642 (Maksim Kita).
- Add
farmFingerprint64
function (non-cryptographic string hashing). #16570 (Jacob Hayes). - Add
log_queries_min_query_duration_ms
, only queries slower than the value of this setting will go toquery_log
/query_thread_log
(i.e. something likeslow_query_log
in mysql). #16529 (Azat Khuzhin). - Ability to create a docker image on the top of
Alpine
. Uses precompiled binary and glibc components from ubuntu 20.04. #16479 (filimonov). - Added
toUUIDOrNull
,toUUIDOrZero
cast functions. #16337 (Maksim Kita). - Add
max_concurrent_queries_for_all_users
setting, see #6636 for use cases. #16154 (nvartolomei). - Add a new option
print_query_id
to clickhouse-client. It helps generate arbitrary strings with the current query id generated by the client. Also print query id in clickhouse-client by default. #15809 (Amos Bird). - Add
tid
andlogTrace
functions. This closes #9434. #15803 (flynn). - Add function
formatReadableTimeDelta
that format time delta to human readable string ... #15497 (Filipe Caixeta). - Added
disable_merges
option for volumes in multi-disk configuration. #13956 (Vladimir Chebotarev).
Experimental Feature
- New functions
encrypt
,aes_encrypt_mysql
,decrypt
,aes_decrypt_mysql
. These functions are working slowly, so we consider it as an experimental feature. #11844 (Vasily Nemkov).
Bug Fix
- Mask password in data_path in the
system.distribution_queue
. #16727 (Azat Khuzhin). - Fix
IN
operator over several columns and tuples with enabledtransform_null_in
setting. Fixes #15310. #16722 (Anton Popov). - The setting
max_parallel_replicas
worked incorrectly if the queried table has no sampling. This fixes #5733. #16675 (alexey-milovidov). - Fix optimize_read_in_order/optimize_aggregation_in_order with max_threads > 0 and expression in ORDER BY. #16637 (Azat Khuzhin).
- Calculation of
DEFAULT
expressions was involving possible name collisions (that was very unlikely to encounter). This fixes #9359. #16612 (alexey-milovidov). - Fix
query_thread_log.query_duration_ms
unit. #16563 (Azat Khuzhin). - Fix a bug when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine.
MaterializeMySQL
is an experimental feature. #16504 (TCeason). - Specifically crafted argument of
round
function withDecimal
was leading to integer division by zero. This fixes #13338. #16451 (alexey-milovidov). - Fix DROP TABLE for Distributed (racy with INSERT). #16409 (Azat Khuzhin).
- Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes #16307. #16332 (alexey-milovidov).
- Fixed the inconsistent behaviour when a part of return data could be dropped because the set for its filtration wasn't created. #16308 (Nikita Mikhaylov).
- Fix dictGet in sharding_key (and similar places, i.e. when the function context is stored permanently). #16205 (Azat Khuzhin).
- Fix the exception thrown in
clickhouse-local
when trying to executeOPTIMIZE
command. Fixes #16076. #16192 (filimonov). - Fixes #15780 regression, e.g.
indexOf([1, 2, 3], toLowCardinality(1))
now is prohibited but it should not be. #16038 (Mike). - Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query
SELECT ... FROM system.parts
should work only with MergeTree tables and don't touch MySQL database at all. #16032 (Kruglov Pavel). - Now exception will be thrown when
ALTER MODIFY COLUMN ... DEFAULT ...
has incompatible default with column type. Fixes #15854. #15858 (alesapin). - Fixed IPv4CIDRToRange/IPv6CIDRToRange functions to accept const IP-column values. #15856 (vladimir-golovchenko).
Improvement
- Treat
INTERVAL '1 hour'
as equivalent toINTERVAL 1 HOUR
, to be compatible with Postgres and similar. This fixes #15637. #15978 (flynn). - Enable parsing enum values by their numeric ids for CSV, TSV and JSON input formats. #15685 (vivarum).
- Better read task scheduling for JBOD architecture and
MergeTree
storage. New settingread_backoff_min_concurrency
which serves as the lower limit to the number of reading threads. #16423 (Amos Bird). - Add missing support for
LowCardinality
inAvro
format. #16521 (Mike). - Workaround for use
S3
with nginx server as proxy. Nginx currenty does not accept urls with empty path likehttp://domain.com?delete
, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, likehttp://domain.com/?delete
. #16814 (ianton-ru). - Better diagnostics on parse errors in input data. Provide row number on
Cannot read all data
errors. #16644 (alexey-milovidov). - Make the behaviour of
minMap
andmaxMap
more desireable. It will not skip zero values in the result. Fixes #16087. #16631 (Ildus Kurbangaliev). - Better update of ZooKeeper configuration in runtime. #16630 (sundyli).
- Apply SETTINGS clause as early as possible. It allows to modify more settings in the query. This closes #3178. #16619 (alexey-milovidov).
- Now
event_time_microseconds
field stores in Decimal64, not UInt64. #16617 (Nikita Mikhaylov). - Now paratmeterized functions can be used in
APPLY
column transformer. #16589 (Amos Bird). - Improve scheduling of background task which removes data of dropped tables in
Atomic
databases.Atomic
databases do not create broken symlink to table data directory if table actually has no data directory. #16584 (tavplubix). - Subqueries in
WITH
section (CTE) can reference previous subqueries inWITH
section by their name. #16575 (Amos Bird). - Add current_database into
system.query_thread_log
. #16558 (Azat Khuzhin). - Allow to fetch parts that are already committed or outdated in the current instance into the detached directory. It's useful when migrating tables from another cluster and having N to 1 shards mapping. It's also consistent with the current fetchPartition implementation. #16538 (Amos Bird).
- Multiple improvements for
RabbitMQ
: Fixed bug for #16263. Also minimized event loop lifetime. Added more efficient queues setup. #16426 (Kseniia Sumarokova). - Fix debug assertion in
quantileDeterministic
function. In previous version it may also transfer up to two times more data over the network. Although no bug existed. This fixes #15683. #16410 (alexey-milovidov). - Add
TablesToDropQueueSize
metric. It's equal to number of dropped tables, that are waiting for background data removal. #16364 (tavplubix). - Better diagnostics when client has dropped connection. In previous versions,
Attempt to read after EOF
andBroken pipe
exceptions were logged in server. In new version, it's information messageClient has dropped the connection, cancel the query.
. #16329 (alexey-milovidov). - Add total_rows/total_bytes (from system.tables) support for Set/Join table engines. #16306 (Azat Khuzhin).
- Now it's possible to specify
PRIMARY KEY
withoutORDER BY
for MergeTree table engines family. Closes #15591. #16284 (alesapin). - If there is no tmp folder in the system (chroot, misconfigutation etc)
clickhouse-local
will create temporary subfolder in the current directory. #16280 (filimonov). - Add support for nested data types (like named tuple) as sub-types. Fixes #15587. #16262 (Ivan).
- Support for
database_atomic_wait_for_drop_and_detach_synchronously
/NO DELAY
/SYNC
forDROP DATABASE
. #16127 (Azat Khuzhin). - Add
allow_nondeterministic_optimize_skip_unused_shards
(to allow non deterministic likerand()
ordictGet()
in sharding key). #16105 (Azat Khuzhin). - Fix
memory_profiler_step
/max_untracked_memory
for queries via HTTP (test included). Fix the issue that adjusting this value globally in xml config does not help either, since those settings are not applied anyway, only default (4MB) value is used. Fixquery_id
for the most root ThreadStatus of the http query (by initializing QueryScope after reading query_id). #16101 (Azat Khuzhin). - Now it's allowed to execute
ALTER ... ON CLUSTER
queries regardless of the<internal_replication>
setting in cluster config. #16075 (alesapin). - Fix rare issue when
clickhouse-client
may abort on exit due to loading of suggestions. This fixes #16035. #16047 (alexey-milovidov). - Add support of
cache
layout forRedis
dictionaries with complex key. #15985 (Anton Popov). - Fix query hang (endless loop) in case of misconfiguration (
connections_with_failover_max_tries
set to 0). #15876 (Azat Khuzhin). - Change level of some log messages from information to debug, so information messages will not appear for every query. This closes #5293. #15816 (alexey-milovidov).
- Remove
MemoryTrackingInBackground*
metrics to avoid potentially misleading results. This fixes #15684. #15813 (alexey-milovidov). - Add reconnects to
zookeeper-dump-tree
tool. #15711 (alexey-milovidov). - Allow explicitly specify columns list in
CREATE TABLE table AS table_function(...)
query. Fixes #9249 Fixes #14214. #14295 (tavplubix).
Performance Improvement
- Do not merge parts across partitions in SELECT FINAL. #15938 (Kruglov Pavel).
- Improve performance of
-OrNull
and-OrDefault
aggregate functions. #16661 (alexey-milovidov). - Improve performance of
quantileMerge
. In previous versions it was obnoxiously slow. This closes #1463. #16643 (alexey-milovidov). - Improve performance of logical functions a little. #16347 (alexey-milovidov).
- Improved performance of merges assignment in MergeTree table engines. Shouldn't be visible for the user. #16191 (alesapin).
- Speedup hashed/sparse_hashed dictionary loading by preallocating the hash table. #15454 (Azat Khuzhin).
- Now trivial count optimization becomes slightly non-trivial. Predicates that contain exact partition expr can be optimized too. This also fixes #11092 which returns wrong count when
max_parallel_replicas > 1
. #15074 (Amos Bird).
Build/Testing/Packaging Improvement
- Add flaky check for stateless tests. It will detect potentially flaky functional tests in advance, before they are merged. #16238 (alesapin).
- Use proper version for
croaring
instead of amalgamation. #16285 (sundyli). - Improve generation of build files for
ya.make
build system (Arcadia). #16700 (alexey-milovidov). - Add MySQL BinLog file check tool for
MaterializeMySQL
database engine.MaterializeMySQL
is an experimental feature. #16223 (Winter Zhang). - Check for executable bit on non-executable files. People often accidentially commit executable files from Windows. #15843 (alexey-milovidov).
- Check for
#pragma once
in headers. #15818 (alexey-milovidov). - Fix illegal code style
&vector[idx]
in libhdfs3. This fixes libcxx debug build. See also https://github.com/ClickHouse-Extras/libhdfs3/pull/8 . #15815 (Amos Bird). - Fix build of one miscellaneous example tool on Mac OS. Note that we don't build examples on Mac OS in our CI (we build only ClickHouse binary), so there is zero chance it will not break again. This fixes #15804. #15808 (alexey-milovidov).
- Simplify Sys/V init script. #14135 (alexey-milovidov).
- Added
boost::program_options
todb_generator
in order to increase its usability. This closes #15940. #15973 (Nikita Mikhaylov).
ClickHouse release 20.10
ClickHouse release v20.10.7.4-stable, 2020-12-24
Bug Fix
- Fixed issue when
clickhouse-odbc-bridge
process is unreachable by server on machines with dualIPv4/IPv6
stack and fixed issue when ODBC dictionary updates are performed using malformed queries and/or cause crashes. This possibly closes #14489. #18278 (Denis Glazachev). - Fix key comparison between Enum and Int types. This fixes #17989. #18214 (Amos Bird).
- Fixed unique key convert crash in
MaterializeMySQL
database engine. This fixes #18186 and fixes #16372 #18211 (Winter Zhang). - Fixed
std::out_of_range: basic_string
in S3 URL parsing. #18059 (Vladimir Chebotarev). - Fixed the issue when some tables not synchronized to ClickHouse from MySQL caused by the fact that convertion MySQL prefix index wasn't supported for MaterializeMySQL. This fixes #15187 and fixes #17912 #17944 (Winter Zhang).
- Fix possible segfault in
topK
aggregate function. This closes #17404. #17845 (Maksim Kita). - Do not restore parts from
WAL
ifin_memory_parts_enable_wal
is disabled. #17802 (detailyang). - Fixed problem when ClickHouse fails to resume connection to MySQL servers. #17681 (Alexander Kazakov).
- Fixed empty
system.stack_trace
table when server is running in daemon mode. #17630 (Amos Bird). - Fixed the behaviour when
clickhouse-client
is used in interactive mode with multiline queries and single line comment was erronously extended till the end of query. This fixes #13654. #17565 (alexey-milovidov). - Fixed the issue when server can stop accepting connections in very rare cases. #17542 (alexey-milovidov).
- Fixed
ALTER
query hang when the corresponding mutation was killed on the different replica. This fixes #16953. #17499 (alesapin). - Fixed bug when mark cache size was underestimated by clickhouse. It may happen when there are a lot of tiny files with marks. #17496 (alesapin).
- Fixed
ORDER BY
with enabled settingoptimize_redundant_functions_in_order_by
. #17471 (Anton Popov). - Fixed duplicates after
DISTINCT
which were possible because of incorrect optimization. Fixes #17294. #17296 (li chengxiang). #17439 (Nikolai Kochetov). - Fixed crash while reading from
JOIN
table withLowCardinality
types. This fixes #17228. #17397 (Nikolai Kochetov). - Fixed set index invalidation when there are const columns in the subquery. This fixes #17246 . #17249 (Amos Bird).
- Fixed
ColumnConst
comparison which leads to crash. This fixed #17088 . #17135 (Amos Bird). - Fixed bug when
ON CLUSTER
queries may hang forever for non-leaderReplicatedMergeTreeTables
. #17089 (alesapin). - Fixed fuzzer-found bug in function
fuzzBits
. This fixes #16980. #17051 (hexiaoting). - Avoid unnecessary network errors for remote queries which may be cancelled while execution, like queries with
LIMIT
. #17006 (Azat Khuzhin). - Fixed wrong result in big integers (128, 256 bit) when casting from double. #16986 (Mike).
- Reresolve the IP of the
format_avro_schema_registry_url
in case of errors. #16985 (filimonov). - Fixed possible server crash after
ALTER TABLE ... MODIFY COLUMN ... NewType
whenSELECT
haveWHERE
expression on altering column and alter does not finished yet. #16968 (Amos Bird). - Blame info was not calculated correctly in
clickhouse-git-import
. #16959 (alexey-milovidov). - Fixed order by optimization with monotonous functions. This fixes #16107. #16956 (Anton Popov).
- Fixrf optimization of group by with enabled setting
optimize_aggregators_of_group_by_keys
and joins. This fixes #12604. #16951 (Anton Popov). - Install script should always create subdirs in config folders. This is only relevant for Docker build with custom config. #16936 (filimonov).
- Fixrf possible error
Illegal type of argument
for queries withORDER BY
. This fixes #16580. #16928 (Nikolai Kochetov). - Abort multipart upload if no data was written to
WriteBufferFromS3
. #16840 (Pavel Kovalenko). - Fixed crash when using
any
without any arguments. This fixes #16803. #16826 (Amos Bird). - Fixed the behaviour when ClickHouse used to always return 0 insted of a number of affected rows for
INSERT
queries via MySQL protocol. This fixes #16605. #16715 (Winter Zhang). - Fixed uncontrolled growth of
TDigest
. #16680 (hrissan). - Fixed remote query failure when using suffix
if
in Aggregate function. This fixes #16574 fixes #16231 #16610 (Winter Zhang).
ClickHouse release v20.10.4.1-stable, 2020-11-13
Bug Fix
- Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes #15301. This fixes #13098. #16846 (alexey-milovidov).
- Fix
IN
operator over several columns and tuples with enabledtransform_null_in
setting. Fixes #15310. #16722 (Anton Popov). - This will fix optimize_read_in_order/optimize_aggregation_in_order with max_threads>0 and expression in ORDER BY. #16637 (Azat Khuzhin).
- Now when parsing AVRO from input the LowCardinality is removed from type. Fixes #16188. #16521 (Mike).
- Fix rapid growth of metadata when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine, and
slave_parallel_worker
enabled on MySQL Slave, by properly shrinking GTID sets. This fixes #15951. #16504 (TCeason). - Fix DROP TABLE for Distributed (racy with INSERT). #16409 (Azat Khuzhin).
- Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes #16307. #16332 (alexey-milovidov).
- Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query
SELECT ... FROM system.parts
should work only with MergeTree tables and don't touch MySQL database at all. #16032 (Kruglov Pavel).
Improvement
- Workaround for use S3 with nginx server as proxy. Nginx currenty does not accept urls with empty path like http://domain.com?delete, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, like http://domain.com/?delete. #16813 (ianton-ru).
ClickHouse release v20.10.3.30, 2020-10-28
Backward Incompatible Change
- Make
multiple_joins_rewriter_version
obsolete. Remove first version of joins rewriter. #15472 (Artem Zuikov). - Change default value of
format_regexp_escaping_rule
setting (it's related toRegexp
format) toRaw
(it means - read whole subpattern as a value) to make the behaviour more like to what users expect. #15426 (alexey-milovidov). - Add support for nested multiline comments
/* comment /* comment */ */
in SQL. This conforms to the SQL standard. #14655 (alexey-milovidov). - Added MergeTree settings (
max_replicated_merges_with_ttl_in_queue
andmax_number_of_merges_with_ttl_in_pool
) to control the number of merges with TTL in the background pool and replicated queue. This change breaks compatibility with older versions only if you use delete TTL. Otherwise, replication will stay compatible. You can avoid incompatibility issues if you update all shard replicas at once or executeSYSTEM STOP TTL MERGES
until you finish the update of all replicas. If you'll get an incompatible entry in the replication queue, first of all, executeSYSTEM STOP TTL MERGES
and afterALTER TABLE ... DETACH PARTITION ...
the partition where incompatible TTL merge was assigned. Attach it back on a single replica. #14490 (alesapin). - When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to
Part ... intersects previous part
errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).
New Feature
- Background data recompression. Add the ability to specify
TTL ... RECOMPRESS codec_name
for MergeTree table engines family. #14494 (alesapin). - Add parallel quorum inserts. This closes #15601. #15601 (Latysheva Alexandra).
- Settings for additional enforcement of data durability. Useful for non-replicated setups. #11948 (Anton Popov).
- When duplicate block is written to replica where it does not exist locally (has not been fetched from replicas), don't ignore it and write locally to achieve the same effect as if it was successfully replicated. #11684 (alexey-milovidov).
- Now we support
WITH <identifier> AS (subquery) ...
to introduce named subqueries in the query context. This closes #2416. This closes #4967. #14771 (Amos Bird). - Introduce
enable_global_with_statement
setting which propagates the first select'sWITH
statements to other select queries at the same level, and makes aliases inWITH
statements visible to subqueries. #15451 (Amos Bird). - Secure inter-cluster query execution (with initial_user as current query user). #13156 (Azat Khuzhin). #15551 (Azat Khuzhin).
- Add the ability to remove column properties and table TTLs. Introduced queries
ALTER TABLE MODIFY COLUMN col_name REMOVE what_to_remove
andALTER TABLE REMOVE TTL
. Both operations are lightweight and executed at the metadata level. #14742 (alesapin). - Added format
RawBLOB
. It is intended for input or output a single value without any escaping and delimiters. This closes #15349. #15364 (alexey-milovidov). - Add the
reinterpretAsUUID
function that allows to convert a big-endian byte string to UUID. #15480 (Alexander Kuzmenkov). - Implement
force_data_skipping_indices
setting. #15642 (Azat Khuzhin). - Add a setting
output_format_pretty_row_numbers
to numerate the result in Pretty formats. This closes #15350. #15443 (flynn). - Added query obfuscation tool. It allows to share more queries for better testing. This closes #15268. #15321 (alexey-milovidov).
- Add table function
null('structure')
. #14797 (vxider). - Added
formatReadableQuantity
function. It is useful for reading big numbers by human. #14725 (Artem Hnilov). - Add format
LineAsString
that accepts a sequence of lines separated by newlines, every line is parsed as a whole as a single String field. #14703 (Nikita Mikhaylov), #13846 (hexiaoting). - Add
JSONStrings
format which output data in arrays of strings. #14333 (hcz). - Add support for "Raw" column format for
Regexp
format. It allows to simply extract subpatterns as a whole without any escaping rules. #15363 (alexey-milovidov). - Allow configurable
NULL
representation forTSV
output format. It is controlled by the settingoutput_format_tsv_null_representation
which is\N
by default. This closes #9375. Note that the setting only controls output format and\N
is the only supportedNULL
representation forTSV
input format. #14586 (Kruglov Pavel). - Support Decimal data type for
MaterializeMySQL
.MaterializeMySQL
is an experimental feature. #14535 (Winter Zhang). - Add new feature:
SHOW DATABASES LIKE 'xxx'
. #14521 (hexiaoting). - Added a script to import (arbitrary) git repository to ClickHouse as a sample dataset. #14471 (alexey-milovidov).
- Now insert statements can have asterisk (or variants) with column transformers in the column list. #14453 (Amos Bird).
- New query complexity limit settings
max_rows_to_read_leaf
,max_bytes_to_read_leaf
for distributed queries to limit max rows/bytes read on the leaf nodes. Limit is applied for local reads only, excluding the final merge stage on the root node. #14221 (Roman Khavronenko). - Allow user to specify settings for
ReplicatedMergeTree*
storage in<replicated_merge_tree>
section of config file. It works similarly to<merge_tree>
section. ForReplicatedMergeTree*
storages settings from<merge_tree>
and<replicated_merge_tree>
are applied together, but settings from<replicated_merge_tree>
has higher priority. Addedsystem.replicated_merge_tree_settings
table. #13573 (Amos Bird). - Add
mapPopulateSeries
function. #13166 (Ildus Kurbangaliev). - Supporting MySQL types:
decimal
(as ClickHouseDecimal
) anddatetime
with sub-second precision (asDateTime64
). #11512 (Vasily Nemkov). - Introduce
event_time_microseconds
field tosystem.text_log
,system.trace_log
,system.query_log
andsystem.query_thread_log
tables. #14760 (Bharat Nallan). - Add
event_time_microseconds
tosystem.asynchronous_metric_log
&system.metric_log
tables. #14514 (Bharat Nallan). - Add
query_start_time_microseconds
field tosystem.query_log
&system.query_thread_log
tables. #14252 (Bharat Nallan).
Bug Fix
- Fix the case when memory can be overallocated regardless to the limit. This closes #14560. #16206 (alexey-milovidov).
- Fix
executable
dictionary source hang. In previous versions, when using some formats (e.g.JSONEachRow
) data was not feed to a child process before it outputs at least something. This closes #1697. This closes #2455. #14525 (alexey-milovidov). - Fix double free in case of exception in function
dictGet
. It could have happened if dictionary was loaded with error. #16429 (Nikolai Kochetov). - Fix group by with totals/rollup/cube modifers and min/max functions over group by keys. Fixes #16393. #16397 (Anton Popov).
- Fix async Distributed INSERT with prefer_localhost_replica=0 and internal_replication. #16358 (Azat Khuzhin).
- Fix a very wrong code in TwoLevelStringHashTable implementation, which might lead to memory leak. #16264 (Amos Bird).
- Fix segfault in some cases of wrong aggregation in lambdas. #16082 (Anton Popov).
- Fix
ALTER MODIFY ... ORDER BY
query hang forReplicatedVersionedCollapsingMergeTree
. This fixes #15980. #16011 (alesapin). MaterializeMySQL
(experimental feature): Fix collate name & charset name parser and supportlength = 0
for string type. #16008 (Winter Zhang).- Allow to use
direct
layout for dictionaries with complex keys. #16007 (Anton Popov). - Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. #15987 (filimonov).
- Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). #15984 (tavplubix).
- Fix ambiguity in parsing of settings profiles:
CREATE USER ... SETTINGS profile readonly
is now considered as using a profile namedreadonly
, not a setting namedprofile
with the readonly constraint. This fixes #15628. #15982 (Vitaly Baranov). MaterializeMySQL
(experimental feature): Fix crash on create database failure. #15954 (Winter Zhang).- Fixed
DROP TABLE IF EXISTS
failure withTable ... does not exist
error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (likeDROP DATABASE
andRENAME TABLE
) - FixedDROP/DETACH DATABASE
failure withTable ... does not exist
when concurrently executingDROP/DETACH TABLE
. #15934 (tavplubix). - Fix incorrect empty result for query from
Distributed
table if query hasWHERE
,PREWHERE
andGLOBAL IN
. Fixes #15792. #15933 (Nikolai Kochetov). - Fixes #12513: difference expressions with same alias when query is reanalyzed. #15886 (Winter Zhang).
- Fix possible very rare deadlocks in RBAC implementation. #15875 (Vitaly Baranov).
- Fix exception
Block structure mismatch
inSELECT ... ORDER BY DESC
queries which were executed afterALTER MODIFY COLUMN
query. Fixes #15800. #15852 (alesapin). MaterializeMySQL
(experimental feature): Fixselect count()
inaccuracy. #15767 (tavplubix).- Fix some cases of queries, in which only virtual columns are selected. Previously
Not found column _nothing in block
exception may be thrown. Fixes #12298. #15756 (Anton Popov). - Fix drop of materialized view with inner table in Atomic database (hangs all subsequent DROP TABLE due to hang of the worker thread, due to recursive DROP TABLE for inner table of MV). #15743 (Azat Khuzhin).
- Possibility to move part to another disk/volume if the first attempt was failed. #15723 (Pavel Kovalenko).
- Fix error
Cannot find column
which may happen at insertion intoMATERIALIZED VIEW
in case if query forMV
containesARRAY JOIN
. #15717 (Nikolai Kochetov). - Fixed too low default value of
max_replicated_logs_to_keep
setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. #15701 (tavplubix). - Fix rare race condition in dictionaries and tables from MySQL. #15686 (alesapin).
- Fix (benign) race condition in AMQP-CPP. #15667 (alesapin).
- Fix error
Cannot add simple transform to empty Pipe
which happened while reading fromBuffer
table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes #15529. #15662 (Nikolai Kochetov). - Proper error handling during insert into MergeTree with S3. MergeTree over S3 is an experimental feature. #15657 (Pavel Kovalenko).
- Fixed bug with S3 table function: region from URL was not applied to S3 client configuration. #15646 (Vladimir Chebotarev).
- Fix the order of destruction for resources in
ReadFromStorage
step of query plan. It might cause crashes in rare cases. Possibly connected with #15610. #15645 (Nikolai Kochetov). - Subtract
ReadonlyReplica
metric when detach readonly tables. #15592 (sundyli). - Fixed
Element ... is not a constant expression
error when usingJSON*
function result inVALUES
,LIMIT
or right side ofIN
operator. #15589 (tavplubix). - Query will finish faster in case of exception. Cancel execution on remote replicas if exception happens. #15578 (Azat Khuzhin).
- Prevent the possibility of error message
Could not calculate available disk space (statvfs), errno: 4, strerror: Interrupted system call
. This fixes #15541. #15557 (alexey-milovidov). - Fix
Database <db> does not exist.
in queries with IN and Distributed table when there's no database on initiator. #15538 (Artem Zuikov). - Mutation might hang waiting for some non-existent part after
MOVE
orREPLACE PARTITION
or, in rare cases, afterDETACH
orDROP PARTITION
. It's fixed. #15537 (tavplubix). - Fix bug when
ILIKE
operator stops being case insensitive ifLIKE
with the same pattern was executed. #15536 (alesapin). - Fix
Missing columns
errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes #15530. #15532 (alesapin). - Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. #15516 (nvartolomei).
- Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in
ON CLUSTER
. Introduced in #13450. #15477 (alesapin). - Report proper error when the second argument of
boundingRatio
aggregate function has a wrong type. #15407 (detailyang). - Fixes #15365: attach a database with MySQL engine throws exception (no query context). #15384 (Winter Zhang).
- Fix the case of multiple occurrences of column transformers in a select query. #15378 (Amos Bird).
- Fixed compression in
S3
storage. #15376 (Vladimir Chebotarev). - Fix bug where queries like
SELECT toStartOfDay(today())
fail complaining about empty time_zone argument. #15319 (Bharat Nallan). - Fix race condition during MergeTree table rename and background cleanup. #15304 (alesapin).
- Fix rare race condition on server startup when system logs are enabled. #15300 (alesapin).
- Fix hang of queries with a lot of subqueries to same table of
MySQL
engine. Previously, if there were more than 16 subqueries to sameMySQL
table in query, it hang forever. #15299 (Anton Popov). - Fix MSan report in QueryLog. Uninitialized memory can be used for the field
memory_usage
. #15258 (alexey-milovidov). - Fix 'Unknown identifier' in GROUP BY when query has JOIN over Merge table. #15242 (Artem Zuikov).
- Fix instance crash when using
joinGet
withLowCardinality
types. This fixes #15214. #15220 (Amos Bird). - Fix bug in table engine
Buffer
which does not allow to insert data of new structure intoBuffer
afterALTER
query. Fixes #15117. #15192 (alesapin). - Adjust Decimal field size in MySQL column definition packet. #15152 (maqroll).
- Fixes
Data compressed with different methods
injoin_algorithm='auto'
. Keep LowCardinality as type for left table join key injoin_algorithm='partial_merge'
. #15088 (Artem Zuikov). - Update
jemalloc
to fixpercpu_arena
with affinity mask. #15035 (Azat Khuzhin). #14957 (Azat Khuzhin). - We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes #14908. #15033 (Amos Bird).
- If function
bar
was called with specifically crafted arguments, buffer overflow was possible. This closes #13926. #15028 (alexey-milovidov). - Fixed
Cannot rename ... errno: 22, strerror: Invalid argument
error on DDL query execution in Atomic database when running clickhouse-server in Docker on Mac OS. #15024 (tavplubix). - Fix crash in RIGHT or FULL JOIN with join_algorith='auto' when memory limit exceeded and we should change HashJoin with MergeJoin. #15002 (Artem Zuikov).
- Now settings
number_of_free_entries_in_pool_to_execute_mutation
andnumber_of_free_entries_in_pool_to_lower_max_size_of_merge
can be equal tobackground_pool_size
. #14975 (alesapin). - Fix to make predicate push down work when subquery contains
finalizeAggregation
function. Fixes #14847. #14937 (filimonov). - Publish CPU frequencies per logical core in
system.asynchronous_metrics
. This fixes #14923. #14924 (Alexander Kuzmenkov). MaterializeMySQL
(experimental feature): Fixed.metadata.tmp File exists
error. #14898 (Winter Zhang).- Fix the issue when some invocations of
extractAllGroups
function may trigger "Memory limit exceeded" error. This fixes #13383. #14889 (alexey-milovidov). - Fix SIGSEGV for an attempt to INSERT into StorageFile with file descriptor. #14887 (Azat Khuzhin).
- Fixed segfault in
cache
dictionary #14837. #14879 (Nikita Mikhaylov). MaterializeMySQL
(experimental feature): Fixed bug in parsing MySQL binlog events, which causesAttempt to read after eof
andPacket payload is not fully read
inMaterializeMySQL
database engine. #14852 (Winter Zhang).- Fix rare error in
SELECT
queries when the queried column hasDEFAULT
expression which depends on the other column which also hasDEFAULT
and not present in select query and not exists on disk. Partially fixes #14531. #14845 (alesapin). - Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the
from_zk
include option). This fixes #14814. #14843 (Alexander Kuzmenkov). - Fix wrong monotonicity detection for shrunk
Int -> Int
cast of signed types. It might lead to incorrect query result. This bug is unveiled in #14513. #14783 (Amos Bird). Replace
column transformer should replace identifiers with cloned ASTs. This fixes #14695 . #14734 (Amos Bird).- Fixed missed default database name in metadata of materialized view when executing
ALTER ... MODIFY QUERY
. #14664 (tavplubix). - Fix bug when
ALTER UPDATE
mutation withNullable
column in assignment expression and constant value (likeUPDATE x = 42
) leads to incorrect value in column or segfault. Fixes #13634, #14045. #14646 (alesapin). - Fix wrong Decimal multiplication result caused wrong decimal scale of result column. #14603 (Artem Zuikov).
- Fix function
has
withLowCardinality
ofNullable
. #14591 (Mike). - Cleanup data directory after Zookeeper exceptions during CreateQuery for StorageReplicatedMergeTree Engine. #14563 (Bharat Nallan).
- Fix rare segfaults in functions with combinator
-Resample
, which could appear in result of overflow with very large parameters. #14562 (Anton Popov). - Fix a bug when converting
Nullable(String)
to Enum. Introduced by #12745. This fixes #14435. #14530 (Amos Bird). - Fixed the incorrect sorting order of
Nullable
column. This fixes #14344. #14495 (Nikita Mikhaylov). - Fix
currentDatabase()
function cannot be used inON CLUSTER
ddl query. #14211 (Winter Zhang). MaterializeMySQL
(experimental feature): FixedPacket payload is not fully read
error inMaterializeMySQL
database engine. #14696 (BohuTANG).
Improvement
- Enable
Atomic
database engine by default for newly created databases. #15003 (tavplubix). - Add the ability to specify specialized codecs like
Delta
,T64
, etc. for columns with subtypes. Implements #12551, fixes #11397, fixes #4609. #15089 (alesapin). - Dynamic reload of zookeeper config. #14678 (sundyli).
- Now it's allowed to execute
ALTER ... ON CLUSTER
queries regardless of the<internal_replication>
setting in cluster config. #16075 (alesapin). - Now
joinGet
supports multi-key lookup. Continuation of #12418. #13015 (Amos Bird). - Wait for
DROP/DETACH TABLE
to actually finish ifNO DELAY
orSYNC
is specified forAtomic
database. #15448 (tavplubix). - Now it's possible to change the type of version column for
VersionedCollapsingMergeTree
withALTER
query. #15442 (alesapin). - Unfold
{database}
,{table}
and{uuid}
macros inzookeeper_path
on replicated table creation. Do not allowRENAME TABLE
if it may breakzookeeper_path
after server restart. Fixes #6917. #15348 (tavplubix). - The function
now
allows an argument with timezone. This closes 15264. #15285 (flynn). - Do not allow connections to ClickHouse server until all scripts in
/docker-entrypoint-initdb.d/
are executed. #15244 (Aleksei Kozharin). - Added
optimize
setting toEXPLAIN PLAN
query. If enabled, query plan level optimisations are applied. Enabled by default. #15201 (Nikolai Kochetov). - Proper exception message for wrong number of arguments of CAST. This closes #13992. #15029 (alexey-milovidov).
- Add option to disable TTL move on data part insert. #15000 (Pavel Kovalenko).
- Ignore key constraints when doing mutations. Without this pull request, it's not possible to do mutations when
force_index_by_date = 1
orforce_primary_key = 1
. #14973 (Amos Bird). - Allow to drop Replicated table if previous drop attempt was failed due to ZooKeeper session expiration. This fixes #11891. #14926 (alexey-milovidov).
- Fixed excessive settings constraint violation when running SELECT with SETTINGS from a distributed table. #14876 (Amos Bird).
- Provide a
load_balancing_first_offset
query setting to explicitly state what the first replica is. It's used together withFIRST_OR_RANDOM
load balancing strategy, which allows to control replicas workload. #14867 (Amos Bird). - Show subqueries for
SET
andJOIN
inEXPLAIN
result. #14856 (Nikolai Kochetov). - Allow using multi-volume storage configuration in storage
Distributed
. #14839 (Pavel Kovalenko). - Construct
query_start_time
andquery_start_time_microseconds
from the same timespec. #14831 (Bharat Nallan). - Support for disabling persistency for
StorageJoin
andStorageSet
, this feature is controlled by settingdisable_set_and_join_persistency
. And this PR solved issue #6318. #14776 (vxider). - Now
COLUMNS
can be used to wrap over a list of columns and apply column transformers afterwards. #14775 (Amos Bird). - Add
merge_algorithm
tosystem.merges
table to improve merging inspections. #14705 (Amos Bird). - Fix potential memory leak caused by zookeeper exists watch. #14693 (hustnn).
- Allow parallel execution of distributed DDL. #14684 (Azat Khuzhin).
- Add
QueryMemoryLimitExceeded
event counter. This closes #14589. #14647 (fastio). - Fix some trailing whitespaces in query formatting. #14595 (Azat Khuzhin).
- ClickHouse treats partition expr and key expr differently. Partition expr is used to construct an minmax index containing related columns, while primary key expr is stored as an expr. Sometimes user might partition a table at coarser levels, such as
partition by i / 1000
. However, binary operators are not monotonic and this PR tries to fix that. It might also benifit other use cases. #14513 (Amos Bird). - Add an option to skip access checks for
DiskS3
.s3
disk is an experimental feature. #14497 (Pavel Kovalenko). - Speed up server shutdown process if there are ongoing S3 requests. #14496 (Pavel Kovalenko).
SYSTEM RELOAD CONFIG
now throws an exception if failed to reload and continues using the previous users.xml. The background periodic reloading also continues using the previous users.xml if failed to reload. #14492 (Vitaly Baranov).- For INSERTs with inline data in VALUES format in the script mode of
clickhouse-client
, support semicolon as the data terminator, in addition to the new line. Closes #12288. #13192 (Alexander Kuzmenkov). - Support custom codecs in compact parts. #12183 (Anton Popov).
Performance Improvement
- Enable compact parts by default for small parts. This will allow to process frequent inserts slightly more efficiently (4..100 times). #11913 (alexey-milovidov).
- Improve
quantileTDigest
performance. This fixes #2668. #15542 (Kruglov Pavel). - Significantly reduce memory usage in AggregatingInOrderTransform/optimize_aggregation_in_order. #15543 (Azat Khuzhin).
- Faster 256-bit multiplication. #15418 (Artem Zuikov).
- Improve performance of 256-bit types using (u)int64_t as base type for wide integers. Original wide integers use 8-bit types as base. #14859 (Artem Zuikov).
- Explicitly use a temporary disk to store vertical merge temporary data. #15639 (Grigory Pervakov).
- Use one S3 DeleteObjects request instead of multiple DeleteObject in a loop. No any functionality changes, so covered by existing tests like integration/test_log_family_s3. #15238 (ianton-ru).
- Fix
DateTime <op> DateTime
mistakenly choosing the slow generic implementation. This fixes #15153. #15178 (Amos Bird). - Improve performance of GROUP BY key of type
FixedString
. #15034 (Amos Bird). - Only
mlock
code segment when starting clickhouse-server. In previous versions, all mapped regions were locked in memory, including debug info. Debug info is usually splitted to a separate file but if it isn't, it led to +2..3 GiB memory usage. #14929 (alexey-milovidov). - ClickHouse binary become smaller due to link time optimization.
Build/Testing/Packaging Improvement
- Now we use clang-11 for production ClickHouse build. #15239 (alesapin).
- Now we use clang-11 to build ClickHouse in CI. #14846 (alesapin).
- Switch binary builds (Linux, Darwin, AArch64, FreeDSD) to clang-11. #15622 (Ilya Yatsishin).
- Now all test images use
llvm-symbolizer-11
. #15069 (alesapin). - Allow to build with llvm-11. #15366 (alexey-milovidov).
- Switch from
clang-tidy-10
toclang-tidy-11
. #14922 (alexey-milovidov). - Use LLVM's experimental pass manager by default. #15608 (Danila Kutenin).
- Don't allow any C++ translation unit to build more than 10 minutes or to use more than 10 GB or memory. This fixes #14925. #15060 (alexey-milovidov).
- Make performance test more stable and representative by splitting test runs and profile runs. #15027 (alexey-milovidov).
- Attempt to make performance test more reliable. It is done by remapping the executable memory of the process on the fly with
madvise
to use transparent huge pages - it can lower the number of iTLB misses which is the main source of instabilities in performance tests. #14685 (alexey-milovidov). - Convert to python3. This closes #14886. #15007 (Azat Khuzhin).
- Fail early in functional tests if server failed to respond. This closes #15262. #15267 (alexey-milovidov).
- Allow to run AArch64 version of clickhouse-server without configs. This facilitates #15174. #15266 (alexey-milovidov).
- Improvements in CI docker images: get rid of ZooKeeper and single script for test configs installation. #15215 (alesapin).
- Fix CMake options forwarding in fast test script. Fixes error in #14711. #15155 (alesapin).
- Added a script to perform hardware benchmark in a single command. #15115 (alexey-milovidov).
- Splitted huge test
test_dictionaries_all_layouts_and_sources
into smaller ones. #15110 (Nikita Mikhaylov). - Maybe fix MSan report in base64 (on servers with AVX-512). This fixes #14006. #15030 (alexey-milovidov).
- Reformat and cleanup code in all integration test *.py files. #14864 (Bharat Nallan).
- Fix MaterializeMySQL empty transaction unstable test case found in CI. #14854 (Winter Zhang).
- Attempt to speed up build a little. #14808 (alexey-milovidov).
- Speed up build a little by removing unused headers. #14714 (alexey-milovidov).
- Fix build failure in OSX. #14761 (Winter Zhang).
- Enable ccache by default in cmake if it's found in OS. #14575 (alesapin).
- Control CI builds configuration from the ClickHouse repository. #14547 (alesapin).
- In CMake files: - Moved some options' descriptions' parts to comments above. - Replace 0 ->
OFF
, 1 ->ON
inoption
s default values. - Added some descriptions and links to docs to the options. - ReplacedFUZZER
option (there is another optionENABLE_FUZZING
which also enables same functionality). - RemovedENABLE_GTEST_LIBRARY
option as there isENABLE_TESTS
. See the full description in PR: #14711 (Mike). - Make binary a bit smaller (~50 Mb for debug version). #14555 (Artem Zuikov).
- Use std::filesystem::path in ConfigProcessor for concatenating file paths. #14558 (Bharat Nallan).
- Fix debug assertion in
bitShiftLeft()
when called with negative big integer. #14697 (Artem Zuikov).
ClickHouse release 20.9
ClickHouse release v20.9.7.11-stable, 2020-12-07
Performance Improvement
- Fix performance of reading from
Merge
tables over huge number ofMergeTree
tables. Fixes #7748. #16988 (Anton Popov).
Bug Fix
- Do not restore parts from WAL if
in_memory_parts_enable_wal
is disabled. #17802 (detailyang). - Fixed segfault when there is not enough space when inserting into
Distributed
table. #17737 (tavplubix). - Fixed problem when ClickHouse fails to resume connection to MySQL servers. #17681 (Alexander Kazakov).
- Fixed
Function not implemented
error when executingRENAME
query inAtomic
database with ClickHouse running on Windows Subsystem for Linux. Fixes #17661. #17664 (tavplubix). - When clickhouse-client is used in interactive mode with multiline queries, single line comment was erronously extended till the end of query. This fixes #13654. #17565 (alexey-milovidov).
- Fix the issue when server can stop accepting connections in very rare cases. #17542 (alexey-milovidov).
- Fix alter query hang when the corresponding mutation was killed on the different replica. Fixes #16953. #17499 (alesapin).
- Fix bug when mark cache size was underestimated by clickhouse. It may happen when there are a lot of tiny files with marks. #17496 (alesapin).
- Fix
ORDER BY
with enabled settingoptimize_redundant_functions_in_order_by
. #17471 (Anton Popov). - Fix duplicates after
DISTINCT
which were possible because of incorrect optimization. Fixes #17294. #17296 (li chengxiang). #17439 (Nikolai Kochetov). - Fix crash while reading from
JOIN
table withLowCardinality
types. Fixes #17228. #17397 (Nikolai Kochetov). - Fix set index invalidation when there are const columns in the subquery. This fixes #17246 . #17249 (Amos Bird).
- Fix ColumnConst comparison which leads to crash. This fixed #17088 . #17135 (Amos Bird).
- Fixed crash on
CREATE TABLE ... AS some_table
query whensome_table
was createdAS table_function()
Fixes #16944. #17072 (tavplubix). - Bug fix for funciton fuzzBits, related issue: #16980. #17051 (hexiaoting).
- Avoid unnecessary network errors for remote queries which may be cancelled while execution, like queries with
LIMIT
. #17006 (Azat Khuzhin). - TODO. #16866 (tavplubix).
- Return number of affected rows for INSERT queries via MySQL protocol. Previously ClickHouse used to always return 0, it's fixed. Fixes #16605. #16715 (Winter Zhang).
Build/Testing/Packaging Improvement
- Update embedded timezone data to version 2020d (also update cctz to the latest master). #17204 (filimonov).
ClickHouse release v20.9.6.14-stable, 2020-11-20
Improvement
- Make it possible to connect to
clickhouse-server
secure endpoint which requires SNI. This is possible whenclickhouse-server
is hosted behind TLS proxy. #16938 (filimonov). - Conditional aggregate functions (for example:
avgIf
,sumIf
,maxIf
) should returnNULL
when miss rows and use nullable arguments. #13964 (Winter Zhang).
Bug Fix
- Fix bug when
ON CLUSTER
queries may hang forever for non-leader ReplicatedMergeTreeTables. #17089 (alesapin). - Reresolve the IP of the
format_avro_schema_registry_url
in case of errors. #16985 (filimonov). - Fix possible server crash after
ALTER TABLE ... MODIFY COLUMN ... NewType
whenSELECT
haveWHERE
expression on altering column and alter does not finished yet. #16968 (Amos Bird). - Install script should always create subdirs in config folders. This is only relevant for Docker build with custom config. #16936 (filimonov).
- Fix possible error
Illegal type of argument
for queries withORDER BY
. Fixes #16580. #16928 (Nikolai Kochetov). - Abort multipart upload if no data was written to WriteBufferFromS3. #16840 (Pavel Kovalenko).
- Fix crash when using
any
without any arguments. This is for #16803 . cc @azat. #16826 (Amos Bird). - Fix
IN
operator over several columns and tuples with enabledtransform_null_in
setting. Fixes #15310. #16722 (Anton Popov). - This will fix optimize_read_in_order/optimize_aggregation_in_order with max_threads>0 and expression in ORDER BY. #16637 (Azat Khuzhin).
- fixes #16574 fixes #16231 fix remote query failure when using 'if' suffix aggregate function. #16610 (Winter Zhang).
- Query is finished faster in case of exception. Cancel execution on remote replicas if exception happens. #15578 (Azat Khuzhin).
ClickHouse release v20.9.5.5-stable, 2020-11-13
Bug Fix
- Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes #15301. This fixes #13098. #16846 (alexey-milovidov).
- Now when parsing AVRO from input the LowCardinality is removed from type. Fixes #16188. #16521 (Mike).
- Fix rapid growth of metadata when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine, and
slave_parallel_worker
enabled on MySQL Slave, by properly shrinking GTID sets. This fixes #15951. #16504 (TCeason). - Fix DROP TABLE for Distributed (racy with INSERT). #16409 (Azat Khuzhin).
- Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes #16307. #16332 (alexey-milovidov).
- Fixed the inconsistent behaviour when a part of return data could be dropped because the set for its filtration wasn't created. #16308 (Nikita Mikhaylov).
- Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query
SELECT ... FROM system.parts
should work only with MergeTree tables and don't touch MySQL database at all. #16032 (Kruglov Pavel).
ClickHouse release v20.9.4.76-stable (2020-10-29)
Bug Fix
- Fix double free in case of exception in function
dictGet
. It could have happened if dictionary was loaded with error. #16429 (Nikolai Kochetov). - Fix group by with totals/rollup/cube modifers and min/max functions over group by keys. Fixes #16393. #16397 (Anton Popov).
- Fix async Distributed INSERT w/ prefer_localhost_replica=0 and internal_replication. #16358 (Azat Khuzhin).
- Fix a very wrong code in TwoLevelStringHashTable implementation, which might lead to memory leak. I'm suprised how this bug can lurk for so long.... #16264 (Amos Bird).
- Fix the case when memory can be overallocated regardless to the limit. This closes #14560. #16206 (alexey-milovidov).
- Fix
ALTER MODIFY ... ORDER BY
query hang forReplicatedVersionedCollapsingMergeTree
. This fixes #15980. #16011 (alesapin). - Fix collate name & charset name parser and support
length = 0
for string type. #16008 (Winter Zhang). - Allow to use direct layout for dictionaries with complex keys. #16007 (Anton Popov).
- Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. #15987 (filimonov).
- Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). #15984 (tavplubix).
- Fix ambiguity in parsing of settings profiles:
CREATE USER ... SETTINGS profile readonly
is now considered as using a profile namedreadonly
, not a setting namedprofile
with the readonly constraint. This fixes #15628. #15982 (Vitaly Baranov). - Fix a crash when database creation fails. #15954 (Winter Zhang).
- Fixed
DROP TABLE IF EXISTS
failure withTable ... does not exist
error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (likeDROP DATABASE
andRENAME TABLE
) FixedDROP/DETACH DATABASE
failure withTable ... does not exist
when concurrently executingDROP/DETACH TABLE
. #15934 (tavplubix). - Fix incorrect empty result for query from
Distributed
table if query hasWHERE
,PREWHERE
andGLOBAL IN
. Fixes #15792. #15933 (Nikolai Kochetov). - Fix possible deadlocks in RBAC. #15875 (Vitaly Baranov).
- Fix exception
Block structure mismatch
inSELECT ... ORDER BY DESC
queries which were executed afterALTER MODIFY COLUMN
query. Fixes #15800. #15852 (alesapin). - Fix
select count()
inaccuracy for MaterializeMySQL. #15767 (tavplubix). - Fix some cases of queries, in which only virtual columns are selected. Previously
Not found column _nothing in block
exception may be thrown. Fixes #12298. #15756 (Anton Popov). - Fixed too low default value of
max_replicated_logs_to_keep
setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. #15701 (tavplubix). - Fix error
Cannot add simple transform to empty Pipe
which happened while reading fromBuffer
table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes #15529. #15662 (Nikolai Kochetov). - Fixed bug with globs in S3 table function, region from URL was not applied to S3 client configuration. #15646 (Vladimir Chebotarev).
- Decrement the
ReadonlyReplica
metric when detaching read-only tables. This fixes #15598. #15592 (sundyli). - Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. #15516 (nvartolomei).
Improvement
- Now it's allowed to execute
ALTER ... ON CLUSTER
queries regardless of the<internal_replication>
setting in cluster config. #16075 (alesapin). - Unfold
{database}
,{table}
and{uuid}
macros inReplicatedMergeTree
arguments on table creation. #16160 (tavplubix).
ClickHouse release v20.9.3.45-stable (2020-10-09)
Bug Fix
- Fix error
Cannot find column
which may happen at insertion intoMATERIALIZED VIEW
in case if query forMV
containesARRAY JOIN
. #15717 (Nikolai Kochetov). - Fix race condition in AMQP-CPP. #15667 (alesapin).
- Fix the order of destruction for resources in
ReadFromStorage
step of query plan. It might cause crashes in rare cases. Possibly connected with #15610. #15645 (Nikolai Kochetov). - Fixed
Element ... is not a constant expression
error when usingJSON*
function result inVALUES
,LIMIT
or right side ofIN
operator. #15589 (tavplubix). - Prevent the possibility of error message
Could not calculate available disk space (statvfs), errno: 4, strerror: Interrupted system call
. This fixes #15541. #15557 (alexey-milovidov). - Significantly reduce memory usage in AggregatingInOrderTransform/optimize_aggregation_in_order. #15543 (Azat Khuzhin).
- Mutation might hang waiting for some non-existent part after
MOVE
orREPLACE PARTITION
or, in rare cases, afterDETACH
orDROP PARTITION
. It's fixed. #15537 (tavplubix). - Fix bug when
ILIKE
operator stops being case insensitive ifLIKE
with the same pattern was executed. #15536 (alesapin). - Fix
Missing columns
errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes #15530. #15532 (alesapin). - Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in
ON CLUSTER
. Introduced in #13450. #15477 (alesapin). - Report proper error when the second argument of
boundingRatio
aggregate function has a wrong type. #15407 (detailyang). - Fix bug where queries like
SELECT toStartOfDay(today())
fail complaining about empty time_zone argument. #15319 (Bharat Nallan). - Fix race condition during MergeTree table rename and background cleanup. #15304 (alesapin).
- Fix rare race condition on server startup when system.logs are enabled. #15300 (alesapin).
- Fix MSan report in QueryLog. Uninitialized memory can be used for the field
memory_usage
. #15258 (alexey-milovidov). - Fix instance crash when using joinGet with LowCardinality types. This fixes #15214. #15220 (Amos Bird).
- Fix bug in table engine
Buffer
which does not allow to insert data of new structure intoBuffer
afterALTER
query. Fixes #15117. #15192 (alesapin). - Adjust decimals field size in mysql column definition packet. #15152 (maqroll).
- Fixed
Cannot rename ... errno: 22, strerror: Invalid argument
error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. #15024 (tavplubix). - Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes #14847. #14937 (filimonov).
- Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the
from_zk
include option). This fixes #14814. #14843 (Alexander Kuzmenkov).
Improvement
- Now it's possible to change the type of version column for
VersionedCollapsingMergeTree
withALTER
query. #15442 (alesapin).
ClickHouse release v20.9.2.20, 2020-09-22
Backward Incompatible Change
- When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to
Part ... intersects previous part
errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).
New Feature
- Added column transformers
EXCEPT
,REPLACE
,APPLY
, which can be applied to the list of selected columns (after*
orCOLUMNS(...)
). For example, you can writeSELECT * EXCEPT(URL) REPLACE(number + 1 AS number)
. Another example:select * apply(length) apply(max) from wide_string_table
to find out the maxium length of all string columns. #14233 (Amos Bird). - Added an aggregate function
rankCorr
which computes a rank correlation coefficient. #11769 (antikvist) #14411 (Nikita Mikhaylov). - Added table function
view
which turns a subquery into a table object. This helps passing queries around. For instance, it can be used in remote/cluster table functions. #12567 (Amos Bird).
Bug Fix
- Fix bug when
ALTER UPDATE
mutation with Nullable column in assignment expression and constant value (likeUPDATE x = 42
) leads to incorrect value in column or segfault. Fixes #13634, #14045. #14646 (alesapin). - Fix wrong Decimal multiplication result caused wrong decimal scale of result column. #14603 (Artem Zuikov).
- Fixed the incorrect sorting order of
Nullable
column. This fixes #14344. #14495 (Nikita Mikhaylov). - Fixed inconsistent comparison with primary key of type
FixedString
on index analysis if they're compered with a string of less size. This fixes #14908. #15033 (Amos Bird). - Fix bug which leads to wrong merges assignment if table has partitions with a single part. #14444 (alesapin).
- If function
bar
was called with specifically crafted arguments, buffer overflow was possible. This closes #13926. #15028 (alexey-milovidov). - Publish CPU frequencies per logical core in
system.asynchronous_metrics
. This fixes #14923. #14924 (Alexander Kuzmenkov). - Fixed
.metadata.tmp File exists
error when usingMaterializeMySQL
database engine. #14898 (Winter Zhang). - Fix the issue when some invocations of
extractAllGroups
function may trigger "Memory limit exceeded" error. This fixes #13383. #14889 (alexey-milovidov). - Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). #14887 (Azat Khuzhin).
- Fix rare error in
SELECT
queries when the queried column hasDEFAULT
expression which depends on the other column which also hasDEFAULT
and not present in select query and not exists on disk. Partially fixes #14531. #14845 (alesapin). - Fix wrong monotonicity detection for shrunk
Int -> Int
cast of signed types. It might lead to incorrect query result. This bug is unveiled in #14513. #14783 (Amos Bird). - Fixed missed default database name in metadata of materialized view when executing
ALTER ... MODIFY QUERY
. #14664 (tavplubix). - Fix possibly incorrect result of function
has
when LowCardinality and Nullable types are involved. #14591 (Mike). - Cleanup data directory after Zookeeper exceptions during CREATE query for tables with ReplicatedMergeTree Engine. #14563 (Bharat Nallan).
- Fix rare segfaults in functions with combinator
-Resample
, which could appear in result of overflow with very large parameters. #14562 (Anton Popov). - Check for array size overflow in
topK
aggregate function. Without this check the user may send a query with carefully crafted parameters that will lead to server crash. This closes #14452. #14467 (alexey-milovidov). - Proxy restart/start/stop/reload of SysVinit to systemd (if it is used). #14460 (Azat Khuzhin).
- Stop query execution if exception happened in
PipelineExecutor
itself. This could prevent rare possible query hung. #14334 #14402 (Nikolai Kochetov). - Fix crash during
ALTER
query for table which was createdAS table_function
. Fixes #14212. #14326 (alesapin). - Fix exception during ALTER LIVE VIEW query with REFRESH command. LIVE VIEW is an experimental feature. #14320 (Bharat Nallan).
- Fix QueryPlan lifetime (for EXPLAIN PIPELINE graph=1) for queries with nested interpreter. #14315 (Azat Khuzhin).
- Better check for tuple size in SSD cache complex key external dictionaries. This fixes #13981. #14313 (alexey-milovidov).
- Disallows
CODEC
onALIAS
column type. Fixes #13911. #14263 (Bharat Nallan). - Fix GRANT ALL statement when executed on a non-global level. #13987 (Vitaly Baranov).
- Fix arrayJoin() capturing in lambda (exception with logical error message was thrown). #13792 (Azat Khuzhin).
Experimental Feature
- Added
db-generator
tool for random database generation by given SELECT queries. It may faciliate reproducing issues when there is only incomplete bug report from the user. #14442 (Nikita Mikhaylov) #10973 (ZeDRoman).
Improvement
- Allow using multi-volume storage configuration in storage Distributed. #14839 (Pavel Kovalenko).
- Disallow empty time_zone argument in
toStartOf*
type of functions. #14509 (Bharat Nallan). - MySQL handler returns
OK
for queries likeSET @@var = value
. Such statement is ignored. It is needed because some MySQL drivers sendSET @@
query for setup after handshake https://github.com/ClickHouse/ClickHouse/issues/9336#issuecomment-686222422 . #14469 (BohuTANG). - Now TTLs will be applied during merge if they were not previously materialized. #14438 (alesapin).
- Now
clickhouse-obfuscator
supports UUID type as proposed in #13163. #14409 (dimarub2000). - Added new setting
system_events_show_zero_values
as proposed in #11384. #14404 (dimarub2000). - Implicitly convert primary key to not null in
MaterializeMySQL
(Same asMySQL
). Fixes #14114. #14397 (Winter Zhang). - Replace wide integers (256 bit) from boost multiprecision with implementation from https://github.com/cerevra/int. 256bit integers are experimental. #14229 (Artem Zuikov).
- Add default compression codec for parts in
system.part_log
with the namedefault_compression_codec
. #14116 (alesapin). - Add precision argument for
DateTime
type. It allows to useDateTime
name instead ofDateTime64
. #13761 (Winter Zhang). - Added requirepass authorization for
Redis
external dictionary. #13688 (Ivan Torgashov). - Improvements in
RabbitMQ
engine: added connection and channels failure handling, proper commits, insert failures handling, better exchanges, queue durability and queue resume opportunity, new queue settings. Fixed tests. #12761 (Kseniia Sumarokova). - Support custom codecs in compact parts. #12183 (Anton Popov).
Performance Improvement
- Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key (under
optimize_skip_unused_shards
andoptimize_distributed_group_by_sharding_key
). #10373 (Azat Khuzhin). - Creating sets for multiple
JOIN
andIN
in parallel. It may slightly improve performance for queries with several differentIN subquery
expressions. #14412 (Nikolai Kochetov). - Improve Kafka engine performance by providing independent thread for each consumer. Separate thread pool for streaming engines (like Kafka). #13939 (fastio).
Build/Testing/Packaging Improvement
- Lower binary size in debug build by removing debug info from
Functions
. This is needed only for one internal project in Yandex who is using very old linker. #14549 (alexey-milovidov). - Prepare for build with clang 11. #14455 (alexey-milovidov).
- Fix the logic in backport script. In previous versions it was triggered for any labels of 100% red color. It was strange. #14433 (alexey-milovidov).
- Integration tests use default base config. All config changes are explicit with main_configs, user_configs and dictionaries parameters for instance. #13647 (Ilya Yatsishin).
ClickHouse release 20.8
ClickHouse release v20.8.12.2-lts, 2021-01-16
Bug Fix
- Fix *If combinator with unary function and Nullable types. #18806 (Azat Khuzhin).
- Restrict merges from wide to compact parts. In case of vertical merge it led to broken result part. #18381 (Anton Popov).
ClickHouse release v20.8.11.17-lts, 2020-12-25
Bug Fix
- Disable write with AIO during merges because it can lead to extremely rare data corruption of primary key columns during merge. #18481 (alesapin).
- Fixed
value is too short
error when executingtoType(...)
functions (toDate
,toUInt32
, etc) with argument of typeNullable(String)
. Now such functions returnNULL
on parsing errors instead of throwing exception. Fixes #7673. #18445 (tavplubix). - Fix possible crashes in aggregate functions with combinator
Distinct
, while using two-level aggregation. Fixes #17682. #18365 (Anton Popov).
ClickHouse release v20.8.10.13-lts, 2020-12-24
Bug Fix
- When server log rotation was configured using
logger.size
parameter with numeric value larger than 2^32, the logs were not rotated properly. #17905 (Alexander Kuzmenkov). - Fixed incorrect initialization of
max_compress_block_size
in MergeTreeWriterSettings withmin_compress_block_size
. #17833 (flynn). - Fixed problem when ClickHouse fails to resume connection to MySQL servers. #17681 (Alexander Kazakov).
- Fixed
ALTER
query hang when the corresponding mutation was killed on the different replica. This fixes #16953. #17499 (alesapin). - Fixed a bug when mark cache size was underestimated by ClickHouse. It may happen when there are a lot of tiny files with marks. #17496 (alesapin).
- Fixed
ORDER BY
with enabled settingoptimize_redundant_functions_in_order_by
. #17471 (Anton Popov). - Fixed
ColumnConst
comparison which leads to crash. This fixed #17088 . #17135 (Amos Bird). - Fixed bug when
ON CLUSTER
queries may hang forever for non-leader ReplicatedMergeTreeTables. #17089 (alesapin). - Avoid unnecessary network errors for remote queries which may be cancelled while execution, like queries with
LIMIT
. #17006 (Azat Khuzhin). - Reresolve the IP of the
format_avro_schema_registry_url
in case of errors. #16985 (filimonov). - Fixed possible server crash after
ALTER TABLE ... MODIFY COLUMN ... NewType
whenSELECT
haveWHERE
expression on altering column and alter does not finished yet. #16968 (Amos Bird). - Install script should always create subdirs in config folders. This is only relevant for Docker build with custom config. #16936 (filimonov).
- Fixed possible error
Illegal type of argument
for queries withORDER BY
. Fixes #16580. #16928 (Nikolai Kochetov). - Abort multipart upload if no data was written to WriteBufferFromS3. #16840 (Pavel Kovalenko).
- Fixed crash when using
any
without any arguments. This fixes #16803. #16826 (Amos Bird). - Fixed
IN
operator over several columns and tuples with enabledtransform_null_in
setting. Fixes #15310. #16722 (Anton Popov). - Fixed inconsistent behaviour of
optimize_read_in_order/optimize_aggregation_in_order
with max_threads > 0 and expression in ORDER BY. #16637 (Azat Khuzhin). - Fixed the issue when query optimization was producing wrong result if query contains
ARRAY JOIN
. #17887 (sundyli). - Query is finished faster in case of exception. Cancel execution on remote replicas if exception happens. #15578 (Azat Khuzhin).
ClickHouse release v20.8.6.6-lts, 2020-11-13
Bug Fix
- Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes #15301. This fixes #13098. #16846 (alexey-milovidov).
- Now when parsing AVRO from input the LowCardinality is removed from type. Fixes #16188. #16521 (Mike).
- Fix rapid growth of metadata when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine, and
slave_parallel_worker
enabled on MySQL Slave, by properly shrinking GTID sets. This fixes #15951. #16504 (TCeason). - Fix DROP TABLE for Distributed (racy with INSERT). #16409 (Azat Khuzhin).
- Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes #16307. #16332 (alexey-milovidov).
- Fixed the inconsistent behaviour when a part of return data could be dropped because the set for its filtration wasn't created. #16308 (Nikita Mikhaylov).
- Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query
SELECT ... FROM system.parts
should work only with MergeTree tables and don't touch MySQL database at all. #16032 (Kruglov Pavel).
ClickHouse release v20.8.5.45-lts, 2020-10-29
Bug Fix
- Fix double free in case of exception in function
dictGet
. It could have happened if dictionary was loaded with error. #16429 (Nikolai Kochetov). - Fix group by with totals/rollup/cube modifers and min/max functions over group by keys. Fixes #16393. #16397 (Anton Popov).
- Fix async Distributed INSERT w/ prefer_localhost_replica=0 and internal_replication. #16358 (Azat Khuzhin).
- Fix a possible memory leak during
GROUP BY
with string keys, caused by an error inTwoLevelStringHashTable
implementation. #16264 (Amos Bird). - Fix the case when memory can be overallocated regardless to the limit. This closes #14560. #16206 (alexey-milovidov).
- Fix
ALTER MODIFY ... ORDER BY
query hang forReplicatedVersionedCollapsingMergeTree
. This fixes #15980. #16011 (alesapin). - Fix collate name & charset name parser and support
length = 0
for string type. #16008 (Winter Zhang). - Allow to use direct layout for dictionaries with complex keys. #16007 (Anton Popov).
- Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. #15987 (filimonov).
- Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). #15984 (tavplubix).
- Fix ambiguity in parsing of settings profiles:
CREATE USER ... SETTINGS profile readonly
is now considered as using a profile namedreadonly
, not a setting namedprofile
with the readonly constraint. This fixes #15628. #15982 (Vitaly Baranov). - Fix a crash when database creation fails. #15954 (Winter Zhang).
- Fixed
DROP TABLE IF EXISTS
failure withTable ... does not exist
error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (likeDROP DATABASE
andRENAME TABLE
) FixedDROP/DETACH DATABASE
failure withTable ... does not exist
when concurrently executingDROP/DETACH TABLE
. #15934 (tavplubix). - Fix incorrect empty result for query from
Distributed
table if query hasWHERE
,PREWHERE
andGLOBAL IN
. Fixes #15792. #15933 (Nikolai Kochetov). - Fix possible deadlocks in RBAC. #15875 (Vitaly Baranov).
- Fix exception
Block structure mismatch
inSELECT ... ORDER BY DESC
queries which were executed afterALTER MODIFY COLUMN
query. Fixes #15800. #15852 (alesapin). - Fix some cases of queries, in which only virtual columns are selected. Previously
Not found column _nothing in block
exception may be thrown. Fixes #12298. #15756 (Anton Popov). - Fix error
Cannot find column
which may happen at insertion intoMATERIALIZED VIEW
in case if query forMV
containesARRAY JOIN
. #15717 (Nikolai Kochetov). - Fixed too low default value of
max_replicated_logs_to_keep
setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. #15701 (tavplubix). - Fix error
Cannot add simple transform to empty Pipe
which happened while reading fromBuffer
table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes #15529. #15662 (Nikolai Kochetov). - Fixed bug with globs in S3 table function, region from URL was not applied to S3 client configuration. #15646 (Vladimir Chebotarev).
- Decrement the
ReadonlyReplica
metric when detaching read-only tables. This fixes #15598. #15592 (sundyli). - Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. #15516 (nvartolomei).
Improvement
- Now it's allowed to execute
ALTER ... ON CLUSTER
queries regardless of the<internal_replication>
setting in cluster config. #16075 (alesapin). - Unfold
{database}
,{table}
and{uuid}
macros inReplicatedMergeTree
arguments on table creation. #16159 (tavplubix).
ClickHouse release v20.8.4.11-lts, 2020-10-09
Bug Fix
- Fix the order of destruction for resources in
ReadFromStorage
step of query plan. It might cause crashes in rare cases. Possibly connected with #15610. #15645 (Nikolai Kochetov). - Fixed
Element ... is not a constant expression
error when usingJSON*
function result inVALUES
,LIMIT
or right side ofIN
operator. #15589 (tavplubix). - Prevent the possibility of error message
Could not calculate available disk space (statvfs), errno: 4, strerror: Interrupted system call
. This fixes #15541. #15557 (alexey-milovidov). - Significantly reduce memory usage in AggregatingInOrderTransform/optimize_aggregation_in_order. #15543 (Azat Khuzhin).
- Mutation might hang waiting for some non-existent part after
MOVE
orREPLACE PARTITION
or, in rare cases, afterDETACH
orDROP PARTITION
. It's fixed. #15537 (tavplubix). - Fix bug when
ILIKE
operator stops being case insensitive ifLIKE
with the same pattern was executed. #15536 (alesapin). - Fix
Missing columns
errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes #15530. #15532 (alesapin). - Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in
ON CLUSTER
. Introduced in #13450. #15477 (alesapin). - Report proper error when the second argument of
boundingRatio
aggregate function has a wrong type. #15407 (detailyang). - Fix race condition during MergeTree table rename and background cleanup. #15304 (alesapin).
- Fix rare race condition on server startup when system.logs are enabled. #15300 (alesapin).
- Fix MSan report in QueryLog. Uninitialized memory can be used for the field
memory_usage
. #15258 (alexey-milovidov). - Fix instance crash when using joinGet with LowCardinality types. This fixes #15214. #15220 (Amos Bird).
- Fix bug in table engine
Buffer
which does not allow to insert data of new structure intoBuffer
afterALTER
query. Fixes #15117. #15192 (alesapin). - Adjust decimals field size in mysql column definition packet. #15152 (maqroll).
- We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes #14908. #15033 (Amos Bird).
- If function
bar
was called with specifically crafted arguments, buffer overflow was possible. This closes #13926. #15028 (alexey-milovidov). - Fixed
Cannot rename ... errno: 22, strerror: Invalid argument
error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. #15024 (tavplubix). - Now settings
number_of_free_entries_in_pool_to_execute_mutation
andnumber_of_free_entries_in_pool_to_lower_max_size_of_merge
can be equal tobackground_pool_size
. #14975 (alesapin). - Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes #14847. #14937 (filimonov).
- Publish CPU frequencies per logical core in
system.asynchronous_metrics
. This fixes #14923. #14924 (Alexander Kuzmenkov). - Fixed
.metadata.tmp File exists
error when usingMaterializeMySQL
database engine. #14898 (Winter Zhang). - Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the
from_zk
include option). This fixes #14814. #14843 (Alexander Kuzmenkov). - Fix wrong monotonicity detection for shrunk
Int -> Int
cast of signed types. It might lead to incorrect query result. This bug is unveiled in #14513. #14783 (Amos Bird). - Fixed the incorrect sorting order of
Nullable
column. This fixes #14344. #14495 (Nikita Mikhaylov).
Improvement
- Now it's possible to change the type of version column for
VersionedCollapsingMergeTree
withALTER
query. #15442 (alesapin).
ClickHouse release v20.8.3.18-stable, 2020-09-18
Bug Fix
- Fix the issue when some invocations of
extractAllGroups
function may trigger "Memory limit exceeded" error. This fixes #13383. #14889 (alexey-milovidov). - Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). #14887 (Azat Khuzhin).
- Fix rare error in
SELECT
queries when the queried column hasDEFAULT
expression which depends on the other column which also hasDEFAULT
and not present in select query and not exists on disk. Partially fixes #14531. #14845 (alesapin). - Fixed missed default database name in metadata of materialized view when executing
ALTER ... MODIFY QUERY
. #14664 (tavplubix). - Fix bug when
ALTER UPDATE
mutation with Nullable column in assignment expression and constant value (likeUPDATE x = 42
) leads to incorrect value in column or segfault. Fixes #13634, #14045. #14646 (alesapin). - Fix wrong Decimal multiplication result caused wrong decimal scale of result column. #14603 (Artem Zuikov).
- Added the checker as neither calling
lc->isNullable()
nor callingls->getDictionaryPtr()->isNullable()
would return the correct result. #14591 (myrrc). - Cleanup data directory after Zookeeper exceptions during CreateQuery for StorageReplicatedMergeTree Engine. #14563 (Bharat Nallan).
- Fix rare segfaults in functions with combinator -Resample, which could appear in result of overflow with very large parameters. #14562 (Anton Popov).
Improvement
- Speed up server shutdown process if there are ongoing S3 requests. #14858 (Pavel Kovalenko).
- Allow using multi-volume storage configuration in storage Distributed. #14839 (Pavel Kovalenko).
- Speed up server shutdown process if there are ongoing S3 requests. #14496 (Pavel Kovalenko).
- Support custom codecs in compact parts. #12183 (Anton Popov).
ClickHouse release v20.8.2.3-stable, 2020-09-08
Backward Incompatible Change
- Now
OPTIMIZE FINAL
query does not recalculate TTL for parts that were added before TTL was created. UseALTER TABLE ... MATERIALIZE TTL
once to calculate them, after thatOPTIMIZE FINAL
will evaluate TTL's properly. This behavior never worked for replicated tables. #14220 (alesapin). - Extend
parallel_distributed_insert_select
setting, adding an option to runINSERT
into local table. The setting changes type fromBool
toUInt64
, so the valuesfalse
andtrue
are no longer supported. If you have these values in server configuration, the server will not start. Please replace them with0
and1
, respectively. #14060 (Azat Khuzhin). - Remove support for the
ODBCDriver
input/output format. This was a deprecated format once used for communication with the ClickHouse ODBC driver, now long superseded by theODBCDriver2
format. Resolves #13629. #13847 (hexiaoting). - When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to
Part ... intersects previous part
errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).
New Feature
- Add the ability to specify
Default
compression codec for columns that correspond to settings specified inconfig.xml
. Implements: #9074. #14049 (alesapin). - Support Kerberos authentication in Kafka, using
krb5
andcyrus-sasl
libraries. #12771 (Ilya Golshtein). - Add function
normalizeQuery
that replaces literals, sequences of literals and complex aliases with placeholders. Add functionnormalizedQueryHash
that returns identical 64bit hash values for similar queries. It helps to analyze query log. This closes #11271. #13816 (alexey-milovidov). - Add
time_zones
table. #13880 (Bharat Nallan). - Add function
defaultValueOfTypeName
that returns the default value for a given type. #13877 (hcz). - Add
countDigits(x)
function that count number of decimal digits in integer or decimal column. AddisDecimalOverflow(d, [p])
function that checks if the value in Decimal column is out of its (or specified) precision. #14151 (Artem Zuikov). - Add
quantileExactLow
andquantileExactHigh
implementations with respective aliases formedianExactLow
andmedianExactHigh
. #13818 (Bharat Nallan). - Added
date_trunc
function that truncates a date/time value to a specified date/time part. #13888 (Vladimir Golovchenko). - Add new optional section
<user_directories>
to the main config. #13425 (Vitaly Baranov). - Add
ALTER SAMPLE BY
statement that allows to change table sample clause. #13280 (Amos Bird). - Function
position
now supports optionalstart_pos
argument. #13237 (vdimir).
Bug Fix
- Fix visible data clobbering by progress bar in client in interactive mode. This fixes #12562 and #13369 and #13584 and fixes #12964. #13691 (alexey-milovidov).
- Fixed incorrect sorting order if
LowCardinality
column when sorting by multiple columns. This fixes #13958. #14223 (Nikita Mikhaylov). - Check for array size overflow in
topK
aggregate function. Without this check the user may send a query with carefully crafted parameters that will lead to server crash. This closes #14452. #14467 (alexey-milovidov). - Fix bug which can lead to wrong merges assignment if table has partitions with a single part. #14444 (alesapin).
- Stop query execution if exception happened in
PipelineExecutor
itself. This could prevent rare possible query hung. Continuation of #14334. #14402 #14334 (Nikolai Kochetov). - Fix crash during
ALTER
query for table which was createdAS table_function
. Fixes #14212. #14326 (alesapin). - Fix exception during ALTER LIVE VIEW query with REFRESH command. Live view is an experimental feature. #14320 (Bharat Nallan).
- Fix QueryPlan lifetime (for EXPLAIN PIPELINE graph=1) for queries with nested interpreter. #14315 (Azat Khuzhin).
- Fix segfault in
clickhouse-odbc-bridge
during schema fetch from some external sources. This PR fixes #13861. #14267 (Vitaly Baranov). - Fix crash in mark inclusion search introduced in #12277. #14225 (Amos Bird).
- Fix creation of tables with named tuples. This fixes #13027. #14143 (alexey-milovidov).
- Fix formatting of minimal negative decimal numbers. This fixes #14111. #14119 (Alexander Kuzmenkov).
- Fix
DistributedFilesToInsert
metric (zeroed when it should not). #14095 (Azat Khuzhin). - Fix
pointInPolygon
with const 2d array as polygon. #14079 (Alexey Ilyukhov). - Fixed wrong mount point in extra info for
Poco::Exception: no space left on device
. #14050 (tavplubix). - Fix GRANT ALL statement when executed on a non-global level. #13987 (Vitaly Baranov).
- Fix parser to reject create table as table function with engine. #13940 (hcz).
- Fix wrong results in select queries with
DISTINCT
keyword and subqueries with UNION ALL in caseoptimize_duplicate_order_by_and_distinct
setting is enabled. #13925 (Artem Zuikov). - Fixed potential deadlock when renaming
Distributed
table. #13922 (tavplubix). - Fix incorrect sorting for
FixedString
columns when sorting by multiple columns. Fixes #13182. #13887 (Nikolai Kochetov). - Fix potentially imprecise result of
topK
/topKWeighted
merge (with non-default parameters). #13817 (Azat Khuzhin). - Fix reading from MergeTree table with INDEX of type SET fails when comparing against NULL. This fixes #13686. #13793 (Amos Bird).
- Fix
arrayJoin
capturing in lambda (LOGICAL_ERROR). #13792 (Azat Khuzhin). - Add step overflow check in function
range
. #13790 (Azat Khuzhin). - Fixed
Directory not empty
error when concurrently executingDROP DATABASE
andCREATE TABLE
. #13756 (alexey-milovidov). - Add range check for
h3KRing
function. This fixes #13633. #13752 (alexey-milovidov). - Fix race condition between DETACH and background merges. Parts may revive after detach. This is continuation of #8602 that did not fix the issue but introduced a test that started to fail in very rare cases, demonstrating the issue. #13746 (alexey-milovidov).
- Fix logging Settings.Names/Values when log_queries_min_type > QUERY_START. #13737 (Azat Khuzhin).
- Fixes
/replicas_status
endpoint response status code when verbose=1. #13722 (javi santana). - Fix incorrect message in
clickhouse-server.init
while checking user and group. #13711 (ylchou). - Do not optimize any(arrayJoin()) -> arrayJoin() under
optimize_move_functions_out_of_any
setting. #13681 (Azat Khuzhin). - Fix crash in JOIN with StorageMerge and
set enable_optimize_predicate_expression=1
. #13679 (Artem Zuikov). - Fix typo in error message about
The value of 'number_of_free_entries_in_pool_to_lower_max_size_of_merge' setting
. #13678 (alexey-milovidov). - Concurrent
ALTER ... REPLACE/MOVE PARTITION ...
queries might cause deadlock. It's fixed. #13626 (tavplubix). - Fixed the behaviour when sometimes cache-dictionary returned default value instead of present value from source. #13624 (Nikita Mikhaylov).
- Fix secondary indices corruption in compact parts. Compact parts are experimental feature. #13538 (Anton Popov).
- Fix premature
ON CLUSTER
timeouts for queries that must be executed on a single replica. Fixes #6704, #7228, #13361, #11884. #13450 (alesapin). - Fix wrong code in function
netloc
. This fixes #13335. #13446 (alexey-milovidov). - Fix possible race in
StorageMemory
. #13416 (Nikolai Kochetov). - Fix missing or excessive headers in
TSV/CSVWithNames
formats in HTTP protocol. This fixes #12504. #13343 (Azat Khuzhin). - Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes #5779, #12527. #13199 (Vitaly Baranov).
- Fix access to
redis
dictionary after connection was dropped once. It may happen withcache
anddirect
dictionary layouts. #13082 (Anton Popov). - Removed wrong auth access check when using ClickHouseDictionarySource to query remote tables. #12756 (sundyli).
- Properly distinguish subqueries in some cases for common subexpression elimination. #8333. #8367 (Amos Bird).
Improvement
- Disallows
CODEC
onALIAS
column type. Fixes #13911. #14263 (Bharat Nallan). - When waiting for a dictionary update to complete, use the timeout specified by
query_wait_timeout_milliseconds
setting instead of a hard-coded value. #14105 (Nikita Mikhaylov). - Add setting
min_index_granularity_bytes
that protects against accidentally creating a table with very lowindex_granularity_bytes
setting. #14139 (Bharat Nallan). - Now it's possible to fetch partitions from clusters that use different ZooKeeper:
ALTER TABLE table_name FETCH PARTITION partition_expr FROM 'zk-name:/path-in-zookeeper'
. It's useful for shipping data to new clusters. #14155 (Amos Bird). - Slightly better performance of Memory table if it was constructed from a huge number of very small blocks (that's unlikely). Author of the idea: Mark Papadakis. Closes #14043. #14056 (alexey-milovidov).
- Conditional aggregate functions (for example:
avgIf
,sumIf
,maxIf
) should returnNULL
when miss rows and use nullable arguments. #13964 (Winter Zhang). - Increase limit in -Resample combinator to 1M. #13947 (Mikhail f. Shiryaev).
- Corrected an error in AvroConfluent format that caused the Kafka table engine to stop processing messages when an abnormally small, malformed, message was received. #13941 (Gervasio Varela).
- Fix wrong error for long queries. It was possible to get syntax error other than
Max query size exceeded
for correct query. #13928 (Nikolai Kochetov). - Better error message for null value of
TabSeparated
format. #13906 (jiang tao). - Function
arrayCompact
will compare NaNs bitwise if the type of array elements is Float32/Float64. In previous versions NaNs were always not equal if the type of array elements is Float32/Float64 and were always equal if the type is more complex, like Nullable(Float64). This closes #13857. #13868 (alexey-milovidov). - Fix data race in
lgamma
function. This race was caught only intsan
, no side effects a really happened. #13842 (Nikolai Kochetov). - Avoid too slow queries when arrays are manipulated as fields. Throw exception instead. #13753 (alexey-milovidov).
- Added Redis requirepass authorization (for redis dictionary source). #13688 (Ivan Torgashov).
- Add MergeTree Write-Ahead-Log (WAL) dump tool. WAL is an experimental feature. #13640 (BohuTANG).
- In previous versions
lcm
function may produce assertion violation in debug build if called with specifically crafted arguments. This fixes #13368. #13510 (alexey-milovidov). - Provide monotonicity for
toDate/toDateTime
functions in more cases. Monotonicity information is used for index analysis (more complex queries will be able to use index). Now the input arguments are saturated more naturally and provides better monotonicity. #13497 (Amos Bird). - Support compound identifiers for custom settings. Custom settings is an integration point of ClickHouse codebase with other codebases (no benefits for ClickHouse itself) #13496 (Vitaly Baranov).
- Move parts from DiskLocal to DiskS3 in parallel.
DiskS3
is an experimental feature. #13459 (Pavel Kovalenko). - Enable mixed granularity parts by default. #13449 (alesapin).
- Proper remote host checking in S3 redirects (security-related thing). #13404 (Vladimir Chebotarev).
- Add
QueryTimeMicroseconds
,SelectQueryTimeMicroseconds
andInsertQueryTimeMicroseconds
to system.events. #13336 (ianton-ru). - Fix debug assertion when Decimal has too large negative exponent. Fixes #13188. #13228 (alexey-milovidov).
- Added cache layer for DiskS3 (cache to local disk mark and index files).
DiskS3
is an experimental feature. #13076 (Pavel Kovalenko). - Fix readline so it dumps history to file now. #13600 (Amos Bird).
- Create
system
database withAtomic
engine by default (a preparation to enableAtomic
database engine by default everywhere). #13680 (tavplubix).
Performance Improvement
- Slightly optimize very short queries with
LowCardinality
. #14129 (Anton Popov). - Enable parallel INSERTs for table engines
Null
,Memory
,Distributed
andBuffer
when the settingmax_insert_threads
is set. #14120 (alexey-milovidov). - Fail fast if
max_rows_to_read
limit is exceeded on parts scan. The motivation behind this change is to skip ranges scan for all selected parts if it is clear thatmax_rows_to_read
is already exceeded. The change is quite noticeable for queries over big number of parts. #13677 (Roman Khavronenko). - Slightly improve performance of aggregation by UInt8/UInt16 keys. #13099 (alexey-milovidov).
- Optimize
has()
,indexOf()
andcountEqual()
functions forArray(LowCardinality(T))
and constant right arguments. #12550 (myrrc). - When performing trivial
INSERT SELECT
queries, automatically setmax_threads
to 1 ormax_insert_threads
, and setmax_block_size
tomin_insert_block_size_rows
. Related to #5907. #12195 (flynn).
Experimental Feature
- ClickHouse can work as MySQL replica - it is implemented by
MaterializeMySQL
database engine. Implements #4006. #10851 (Winter Zhang). - Add types
Int128
,Int256
,UInt256
and related functions for them. Extend Decimals with Decimal256 (precision up to 76 digits). New types are under the settingallow_experimental_bigint_types
. It is working extremely slow and bad. The implementation is incomplete. Please don't use this feature. #13097 (Artem Zuikov).
Build/Testing/Packaging Improvement
- Added
clickhouse install
script, that is useful if you only have a single binary. #13528 (alexey-milovidov). - Allow to run
clickhouse
binary without configuration. #13515 (alexey-milovidov). - Enable check for typos in code with
codespell
. #13513 #13511 (alexey-milovidov). - Enable Shellcheck in CI as a linter of .sh tests. This closes #13168. #13530 #13529 (alexey-milovidov).
- Add a CMake option to fail configuration instead of auto-reconfiguration, enabled by default. #13687 (Konstantin).
- Expose version of embedded tzdata via TZDATA_VERSION in system.build_options. #13648 (filimonov).
- Improve generation of system.time_zones table during build. Closes #14209. #14215 (filimonov).
- Build ClickHouse with the most fresh tzdata from package repository. #13623 (alexey-milovidov).
- Add the ability to write js-style comments in skip_list.json. #14159 (alesapin).
- Ensure that there is no copy-pasted GPL code. #13514 (alexey-milovidov).
- Switch tests docker images to use test-base parent. #14167 (Ilya Yatsishin).
- Adding retry logic when bringing up docker-compose cluster; Increasing COMPOSE_HTTP_TIMEOUT. #14112 (vzakaznikov).
- Enabled
system.text_log
in stress test to find more bugs. #13855 (Nikita Mikhaylov). - Testflows LDAP module: adding missing certificates and dhparam.pem for openldap4. #13780 (vzakaznikov).
- ZooKeeper cannot work reliably in unit tests in CI infrastructure. Using unit tests for ZooKeeper interaction with real ZooKeeper is bad idea from the start (unit tests are not supposed to verify complex distributed systems). We already using integration tests for this purpose and they are better suited. #13745 (alexey-milovidov).
- Added docker image for style check. Added style check that all docker and docker compose files are located in docker directory. #13724 (Ilya Yatsishin).
- Fix cassandra build on Mac OS. #13708 (Ilya Yatsishin).
- Fix link error in shared build. #13700 (Amos Bird).
- Updating LDAP user authentication suite to check that it works with RBAC. #13656 (vzakaznikov).
- Removed
-DENABLE_CURL_CLIENT
forcontrib/aws
. #13628 (Vladimir Chebotarev). - Increasing health-check timeouts for ClickHouse nodes and adding support to dump docker-compose logs if unhealthy containers found. #13612 (vzakaznikov).
- Make sure #10977 is invalid. #13539 (Amos Bird).
- Skip PR's from robot-clickhouse. #13489 (Nikita Mikhaylov).
- Move Dockerfiles from integration tests to
docker/test
directory. docker_compose files are available inrunner
docker container. Docker images are built in CI and not in integration tests. #13448 (Ilya Yatsishin).
ClickHouse release 20.7
ClickHouse release v20.7.2.30-stable, 2020-08-31
Backward Incompatible Change
- Function
modulo
(operator%
) with at least one floating point number as argument will calculate remainder of division directly on floating point numbers without converting both arguments to integers. It makes behaviour compatible with most of DBMS. This also applicable for Date and DateTime data types. Added aliasmod
. This closes #7323. #12585 (alexey-milovidov). - Deprecate special printing of zero Date/DateTime values as
0000-00-00
and0000-00-00 00:00:00
. #12442 (alexey-milovidov). - The function
groupArrayMoving*
was not working for distributed queries. It's result was calculated within incorrect data type (without promotion to the largest type). The functiongroupArrayMovingAvg
was returning integer number that was inconsistent with theavg
function. This fixes #12568. #12622 (alexey-milovidov). - Add sanity check for MergeTree settings. If the settings are incorrect, the server will refuse to start or to create a table, printing detailed explanation to the user. #13153 (alexey-milovidov).
- Protect from the cases when user may set
background_pool_size
to value lower thannumber_of_free_entries_in_pool_to_execute_mutation
ornumber_of_free_entries_in_pool_to_lower_max_size_of_merge
. In these cases ALTERs won't work or the maximum size of merge will be too limited. It will throw exception explaining what to do. This closes #10897. #12728 (alexey-milovidov). - When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to
Part ... intersects previous part
errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).
New Feature
- Polygon dictionary type that provides efficient "reverse geocoding" lookups - to find the region by coordinates in a dictionary of many polygons (world map). It is using carefully optimized algorithm with recursive grids to maintain low CPU and memory usage. #9278 (achulkov2).
- Added support of LDAP authentication for preconfigured users ("Simple Bind" method). #11234 (Denis Glazachev).
- Introduce setting
alter_partition_verbose_result
which outputs information about touched parts for some types ofALTER TABLE ... PARTITION ...
queries (currentlyATTACH
andFREEZE
). Closes #8076. #13017 (alesapin). - Add
bayesAB
function for bayesian-ab-testing. #12327 (achimbab). - Added
system.crash_log
table into which stack traces for fatal errors are collected. This table should be empty. #12316 (alexey-milovidov). - Added http headers
X-ClickHouse-Database
andX-ClickHouse-Format
which may be used to set default database and output format. #12981 (hcz). - Add
minMap
andmaxMap
functions support toSimpleAggregateFunction
. #12662 (Ildus Kurbangaliev). - Add setting
allow_non_metadata_alters
which restricts to executeALTER
queries which modify data on disk. Disabled be default. Closes #11547. #12635 (alesapin). - A function
formatRow
is added to support turning arbitrary expressions into a string via given format. It's useful for manipulating SQL outputs and is quite versatile combined with thecolumns
function. #12574 (Amos Bird). - Add
FROM_UNIXTIME
function for compatibility with MySQL, related to 12149. #12484 (flynn). - Allow Nullable types as keys in MergeTree tables if
allow_nullable_key
table setting is enabled. Closes #5319. #12433 (Amos Bird). - Integration with COS. #12386 (fastio).
- Add
mapAdd
andmapSubtract
functions for adding/subtracting key-mapped values. #11735 (Ildus Kurbangaliev).
Bug Fix
- Fix premature
ON CLUSTER
timeouts for queries that must be executed on a single replica. Fixes #6704, #7228, #13361, #11884. #13450 (alesapin). - Fix crash in mark inclusion search introduced in #12277. #14225 (Amos Bird).
- Fix race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
- Fix visible data clobbering by progress bar in client in interactive mode. This fixes #12562 and #13369 and #13584 and fixes #12964. #13691 (alexey-milovidov).
- Fixed incorrect sorting order for
LowCardinality
columns when ORDER BY multiple columns is used. This fixes #13958. #14223 (Nikita Mikhaylov). - Removed hardcoded timeout, which wrongly overruled
query_wait_timeout_milliseconds
setting for cache-dictionary. #14105 (Nikita Mikhaylov). - Fixed wrong mount point in extra info for
Poco::Exception: no space left on device
. #14050 (tavplubix). - Fix wrong query optimization of select queries with
DISTINCT
keyword when subqueries also haveDISTINCT
in caseoptimize_duplicate_order_by_and_distinct
setting is enabled. #13925 (Artem Zuikov). - Fixed potential deadlock when renaming
Distributed
table. #13922 (tavplubix). - Fix incorrect sorting for
FixedString
columns when ORDER BY multiple columns is used. Fixes #13182. #13887 (Nikolai Kochetov). - Fix potentially lower precision of
topK
/topKWeighted
aggregations (with non-default parameters). #13817 (Azat Khuzhin). - Fix reading from MergeTree table with INDEX of type SET fails when compared against NULL. This fixes #13686. #13793 (Amos Bird).
- Fix step overflow in function
range()
. #13790 (Azat Khuzhin). - Fixed
Directory not empty
error when concurrently executingDROP DATABASE
andCREATE TABLE
. #13756 (alexey-milovidov). - Add range check for
h3KRing
function. This fixes #13633. #13752 (alexey-milovidov). - Fix race condition between DETACH and background merges. Parts may revive after detach. This is continuation of #8602 that did not fix the issue but introduced a test that started to fail in very rare cases, demonstrating the issue. #13746 (alexey-milovidov).
- Fix logging Settings.Names/Values when
log_queries_min_type
greater thanQUERY_START
. #13737 (Azat Khuzhin). - Fix incorrect message in
clickhouse-server.init
while checking user and group. #13711 (ylchou). - Do not optimize
any(arrayJoin())
toarrayJoin()
underoptimize_move_functions_out_of_any
. #13681 (Azat Khuzhin). - Fixed possible deadlock in concurrent
ALTER ... REPLACE/MOVE PARTITION ...
queries. #13626 (tavplubix). - Fixed the behaviour when sometimes cache-dictionary returned default value instead of present value from source. #13624 (Nikita Mikhaylov).
- Fix secondary indices corruption in compact parts (compact parts is an experimental feature). #13538 (Anton Popov).
- Fix wrong code in function
netloc
. This fixes #13335. #13446 (alexey-milovidov). - Fix error in
parseDateTimeBestEffort
function when unix timestamp was passed as an argument. This fixes #13362. #13441 (alexey-milovidov). - Fix invalid return type for comparison of tuples with
NULL
elements. Fixes #12461. #13420 (Nikolai Kochetov). - Fix wrong optimization caused
aggregate function any(x) is found inside another aggregate function in query
error withSET optimize_move_functions_out_of_any = 1
and aliases insideany()
. #13419 (Artem Zuikov). - Fix possible race in
StorageMemory
. #13416 (Nikolai Kochetov). - Fix empty output for
Arrow
andParquet
formats in case if query return zero rows. It was done because empty output is not valid for this formats. #13399 (hcz). - Fix select queries with constant columns and prefix of primary key in
ORDER BY
clause. #13396 (Anton Popov). - Fix
PrettyCompactMonoBlock
for clickhouse-local. Fix extremes/totals withPrettyCompactMonoBlock
. Fixes #7746. #13394 (Azat Khuzhin). - Fixed deadlock in system.text_log. #12452 (alexey-milovidov). It is a part of #12339. This fixes #12325. #13386 (Nikita Mikhaylov).
- Fixed
File(TSVWithNames*)
(header was written multiple times), fixedclickhouse-local --format CSVWithNames*
(lacks header, broken after #12197), fixedclickhouse-local --format CSVWithNames*
with zero rows (lacks header). #13343 (Azat Khuzhin). - Fix segfault when function
groupArrayMovingSum
deserializes empty state. Fixes #13339. #13341 (alesapin). - Throw error on
arrayJoin()
function inJOIN ON
section. #13330 (Artem Zuikov). - Fix crash in
LEFT ASOF JOIN
withjoin_use_nulls=1
. #13291 (Artem Zuikov). - Fix possible error
Totals having transform was already added to pipeline
in case of a query from delayed replica. #13290 (Nikolai Kochetov). - The server may crash if user passed specifically crafted arguments to the function
h3ToChildren
. This fixes #13275. #13277 (alexey-milovidov). - Fix potentially low performance and slightly incorrect result for
uniqExact
,topK
,sumDistinct
and similar aggregate functions called on Float types withNaN
values. It also triggered assert in debug build. This fixes #12491. #13254 (alexey-milovidov). - Fix assertion in KeyCondition when primary key contains expression with monotonic function and query contains comparison with constant whose type is different. This fixes #12465. #13251 (alexey-milovidov).
- Return passed number for numbers with MSB set in function roundUpToPowerOfTwoOrZero(). It prevents potential errors in case of overflow of array sizes. #13234 (Azat Khuzhin).
- Fix function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. #13226 (alexey-milovidov).
- Fix assert in
arrayElement
function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. #13224 (alexey-milovidov). - Fix DateTime64 conversion functions with constant argument. #13205 (Azat Khuzhin).
- Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes #5779, #12527. #13199 (Vitaly Baranov).
- Fix access to
redis
dictionary after connection was dropped once. It may happen withcache
anddirect
dictionary layouts. #13082 (Anton Popov). - Fix wrong index analysis with functions. It could lead to some data parts being skipped when reading from
MergeTree
tables. Fixes #13060. Fixes #12406. #13081 (Anton Popov). - Fix error
Cannot convert column because it is constant but values of constants are different in source and result
for remote queries which use deterministic functions in scope of query, but not deterministic between queries, likenow()
,now64()
,randConstant()
. Fixes #11327. #13075 (Nikolai Kochetov). - Fix crash which was possible for queries with
ORDER BY
tuple and smallLIMIT
. Fixes #12623. #13009 (Nikolai Kochetov). - Fix
Block structure mismatch
error for queries withUNION
andJOIN
. Fixes #12602. #12989 (Nikolai Kochetov). - Corrected
merge_with_ttl_timeout
logic which did not work well when expiration affected more than one partition over one time interval. (Authored by @excitoon). #12982 (Alexander Kazakov). - Fix columns duplication for range hashed dictionary created from DDL query. This fixes #10605. #12857 (alesapin).
- Fix unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
- Fix rare bug when
ALTER DELETE
andALTER MODIFY COLUMN
queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows incount.txt
and as a consequence incorrect data in part. Also, fix a small bug with simultaneousALTER RENAME COLUMN
andALTER ADD COLUMN
. #12760 (alesapin). - Wrong credentials being used when using
clickhouse
dictionary source to query remote tables. #12756 (sundyli). - Fix
CAST(Nullable(String), Enum())
. #12745 (Azat Khuzhin). - Fix performance with large tuples, which are interpreted as functions in
IN
section. The case when user writesWHERE x IN tuple(1, 2, ...)
instead ofWHERE x IN (1, 2, ...)
for some obscure reason. #12700 (Anton Popov). - Fix memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 (Azat Khuzhin).
- Fix wrong optimization
optimize_move_functions_out_of_any=1
in case ofany(func(<lambda>))
. #12664 (Artem Zuikov). - Fixed #10572 fix bloom filter index with const expression. #12659 (Winter Zhang).
- Fix SIGSEGV in StorageKafka when broker is unavailable (and not only). #12658 (Azat Khuzhin).
- Add support for function
if
withArray(UUID)
arguments. This fixes #11066. #12648 (alexey-milovidov). - CREATE USER IF NOT EXISTS now does not throw exception if the user exists. This fixes #12507. #12646 (Vitaly Baranov).
- Exception
There is no supertype...
can be thrown duringALTER ... UPDATE
in unexpected cases (e.g. when subtracting from UInt64 column). This fixes #7306. This fixes #4165. #12633 (alexey-milovidov). - Fix possible
Pipeline stuck
error for queries with external sorting. Fixes #12617. #12618 (Nikolai Kochetov). - Fix error
Output of TreeExecutor is not sorted
forOPTIMIZE DEDUPLICATE
. Fixes #11572. #12613 (Nikolai Kochetov). - Fix the issue when alias on result of function
any
can be lost during query optimization. #12593 (Anton Popov). - Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 (Azat Khuzhin).
- Now ClickHouse will recalculate checksums for parts when file
checksums.txt
is absent. Broken since #9827. #12545 (alesapin). - Fix bug which lead to broken old parts after
ALTER DELETE
query whenenable_mixed_granularity_parts=1
. Fixes #12536. #12543 (alesapin). - Fixing race condition in live view tables which could cause data duplication. LIVE VIEW is an experimental feature. #12519 (vzakaznikov).
- Fix backwards compatibility in binary format of
AggregateFunction(avg, ...)
values. This fixes #12342. #12486 (alexey-milovidov). - Fix crash in JOIN with dictionary when we are joining over expression of dictionary key:
t JOIN dict ON expr(dict.id) = t.id
. Disable dictionary join optimisation for this case. #12458 (Artem Zuikov). - Fix overflow when very large LIMIT or OFFSET is specified. This fixes #10470. This fixes #11372. #12427 (alexey-milovidov).
- kafka: fix SIGSEGV if there is a message with error in the middle of the batch. #12302 (Azat Khuzhin).
Improvement
- Keep smaller amount of logs in ZooKeeper. Avoid excessive growing of ZooKeeper nodes in case of offline replicas when having many servers/tables/inserts. #13100 (alexey-milovidov).
- Now exceptions forwarded to the client if an error happened during ALTER or mutation. Closes #11329. #12666 (alesapin).
- Add
QueryTimeMicroseconds
,SelectQueryTimeMicroseconds
andInsertQueryTimeMicroseconds
tosystem.events
, along with system.metrics, processes, query_log, etc. #13028 (ianton-ru). - Added
SelectedRows
andSelectedBytes
tosystem.events
, along with system.metrics, processes, query_log, etc. #12638 (ianton-ru). - Added
current_database
information tosystem.query_log
. #12652 (Amos Bird). - Allow
TabSeparatedRaw
as input format. #12009 (hcz). - Now
joinGet
supports multi-key lookup. #12418 (Amos Bird). - Allow
*Map
aggregate functions to work on Arrays with NULLs. Fixes #13157. #13225 (alexey-milovidov). - Avoid overflow in parsing of DateTime values that will lead to negative unix timestamp in their timezone (for example,
1970-01-01 00:00:00
in Moscow). Saturate to zero instead. This fixes #3470. This fixes #4172. #12443 (alexey-milovidov). - AvroConfluent: Skip Kafka tombstone records - Support skipping broken records #13203 (Andrew Onyshchuk).
- Fix wrong error for long queries. It was possible to get syntax error other than
Max query size exceeded
for correct query. #13928 (Nikolai Kochetov). - Fix data race in
lgamma
function. This race was caught only intsan
, no side effects really happened. #13842 (Nikolai Kochetov). - Fix a 'Week'-interval formatting for ATTACH/ALTER/CREATE QUOTA-statements. #13417 (vladimir-golovchenko).
- Now broken parts are also reported when encountered in compact part processing. Compact parts is an experimental feature. #13282 (Amos Bird).
- Fix assert in
geohashesInBox
. This fixes #12554. #13229 (alexey-milovidov). - Fix assert in
parseDateTimeBestEffort
. This fixes #12649. #13227 (alexey-milovidov). - Minor optimization in Processors/PipelineExecutor: breaking out of a loop because it makes sense to do so. #13058 (Mark Papadakis).
- Support TRUNCATE table without TABLE keyword. #12653 (Winter Zhang).
- Fix explain query format overwrite by default. This fixes #12541. #12541 (BohuTANG).
- Allow to set JOIN kind and type in more standad way:
LEFT SEMI JOIN
instead ofSEMI LEFT JOIN
. For now both are correct. #12520 (Artem Zuikov). - Changes default value for
multiple_joins_rewriter_version
to 2. It enables new multiple joins rewriter that knows about column names. #12469 (Artem Zuikov). - Add several metrics for requests to S3 storages. #12464 (ianton-ru).
- Use correct default secure port for clickhouse-benchmark with
--secure
argument. This fixes #11044. #12440 (alexey-milovidov). - Rollback insertion errors in
Log
,TinyLog
,StripeLog
engines. In previous versions insertion error lead to inconsisent table state (this works as documented and it is normal for these table engines). This fixes #12402. #12426 (alexey-milovidov). - Implement
RENAME DATABASE
andRENAME DICTIONARY
forAtomic
database engine - Add implicit{uuid}
macro, which can be used in ZooKeeper path forReplicatedMergeTree
. It works withCREATE ... ON CLUSTER ...
queries. Setshow_table_uuid_in_table_create_query_if_not_nil
totrue
to use it. - MakeReplicatedMergeTree
engine arguments optional,/clickhouse/tables/{uuid}/{shard}/
and{replica}
are used by default. Closes #12135. - Minor fixes. - These changes break backward compatibility ofAtomic
database engine. Previously createdAtomic
databases must be manually converted to new format. Atomic database is an experimental feature. #12343 (tavplubix). - Separated
AWSAuthV4Signer
into different logger, removed excessiveAWSClient: AWSClient
from log messages. #12320 (Vladimir Chebotarev). - Better exception message in disk access storage. #12625 (alesapin).
- Better exception for function
in
with invalid number of arguments. #12529 (Anton Popov). - Fix error message about adaptive granularity. #12624 (alesapin).
- Fix SETTINGS parse after FORMAT. #12480 (Azat Khuzhin).
- If MergeTree table does not contain ORDER BY or PARTITION BY, it was possible to request ALTER to CLEAR all the columns and ALTER will stuck. Fixed #7941. #12382 (alexey-milovidov).
- Avoid re-loading completion from the history file after each query (to avoid history overlaps with other client sessions). #13086 (Azat Khuzhin).
Performance Improvement
- Lower memory usage for some operations up to 2 times. #12424 (alexey-milovidov).
- Optimize PK lookup for queries that match exact PK range. #12277 (Ivan Babrou).
- Slightly optimize very short queries with
LowCardinality
. #14129 (Anton Popov). - Slightly improve performance of aggregation by UInt8/UInt16 keys. #13091 and #13055 (alexey-milovidov).
- Push down
LIMIT
step for query plan (inside subqueries). #13016 (Nikolai Kochetov). - Parallel primary key lookup and skipping index stages on parts, as described in #11564. #12589 (Ivan Babrou).
- Converting String-type arguments of function "if" and "transform" into enum if
set optimize_if_transform_strings_to_enum = 1
. #12515 (Artem Zuikov). - Replaces monotonic functions with its argument in
ORDER BY
ifset optimize_monotonous_functions_in_order_by=1
. #12467 (Artem Zuikov). - Add order by optimization that rewrites
ORDER BY x, f(x)
withORDER by x
ifset optimize_redundant_functions_in_order_by = 1
. #12404 (Artem Zuikov). - Allow pushdown predicate when subquery contains
WITH
clause. This fixes #12293 #12663 (Winter Zhang). - Improve performance of reading from compact parts. Compact parts is an experimental feature. #12492 (Anton Popov).
- Attempt to implement streaming optimization in
DiskS3
. DiskS3 is an experimental feature. #12434 (Vladimir Chebotarev).
Build/Testing/Packaging Improvement
- Use
shellcheck
for sh tests linting. #13200 #13207 (alexey-milovidov). - Add script which set labels for pull requests in GitHub hook. #13183 (alesapin).
- Remove some of recursive submodules. See #13378. #13379 (alexey-milovidov).
- Ensure that all the submodules are from proper URLs. Continuation of #13379. This fixes #13378. #13397 (alexey-milovidov).
- Added support for user-declared settings, which can be accessed from inside queries. This is needed when ClickHouse engine is used as a component of another system. #13013 (Vitaly Baranov).
- Added testing for RBAC functionality of INSERT privilege in TestFlows. Expanded tables on which SELECT is being tested. Added Requirements to match new table engine tests. #13340 (MyroTk).
- Fix timeout error during server restart in the stress test. #13321 (alesapin).
- Now fast test will wait server with retries. #13284 (alesapin).
- Function
materialize()
(the function for ClickHouse testing) will work for NULL as expected - by transforming it to non-constant column. #13212 (alexey-milovidov). - Fix libunwind build in AArch64. This fixes #13204. #13208 (alexey-milovidov).
- Even more retries in zkutil gtest to prevent test flakiness. #13165 (alexey-milovidov).
- Small fixes to the RBAC TestFlows. #13152 (vzakaznikov).
- Fixing
00960_live_view_watch_events_live.py
test. #13108 (vzakaznikov). - Improve cache purge in documentation deploy script. #13107 (alesapin).
- Rewrote some orphan tests to gtest. Removed useless includes from tests. #13073 (Nikita Mikhaylov).
- Added tests for RBAC functionality of
SELECT
privilege in TestFlows. #13061 (Ritaank Tiwari). - Rerun some tests in fast test check. #12992 (alesapin).
- Fix MSan error in "rdkafka" library. This closes #12990. Updated
rdkafka
to version 1.5 (master). #12991 (alexey-milovidov). - Fix UBSan report in base64 if tests were run on server with AVX-512. This fixes #12318. Author: @qoega. #12441 (alexey-milovidov).
- Fix UBSan report in HDFS library. This closes #12330. #12453 (alexey-milovidov).
- Check an ability that we able to restore the backup from an old version to the new version. This closes #8979. #12959 (alesapin).
- Do not build helper_container image inside integrational tests. Build docker container in CI and use pre-built helper_container in integration tests. #12953 (Ilya Yatsishin).
- Add a test for
ALTER TABLE CLEAR COLUMN
query for primary key columns. #12951 (alesapin). - Increased timeouts in testflows tests. #12949 (vzakaznikov).
- Fix build of test under Mac OS X. This closes #12767. #12772 (alexey-milovidov).
- Connector-ODBC updated to mysql-connector-odbc-8.0.21. #12739 (Ilya Yatsishin).
- Adding RBAC syntax tests in TestFlows. #12642 (vzakaznikov).
- Improve performance of TestKeeper. This will speedup tests with heavy usage of Replicated tables. #12505 (alexey-milovidov).
- Now we check that server is able to start after stress tests run. This fixes #12473. #12496 (alesapin).
- Update fmtlib to master (7.0.1). #12446 (alexey-milovidov).
- Add docker image for fast tests. #12294 (alesapin).
- Rework configuration paths for integration tests. #12285 (Ilya Yatsishin).
- Add compiler option to control that stack frames are not too large. This will help to run the code in fibers with small stack size. #11524 (alexey-milovidov).
- Update gitignore-files. #13447 (vladimir-golovchenko).
ClickHouse release 20.6
ClickHouse release v20.6.3.28-stable
Backward Incompatible Change
- When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to
Part ... intersects previous part
errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).
New Feature
- Added an initial implementation of
EXPLAIN
query. Syntax:EXPLAIN SELECT ...
. This fixes #1118. #11873 (Nikolai Kochetov). - Added storage
RabbitMQ
. #11069 (Kseniia Sumarokova). - Implemented PostgreSQL-like
ILIKE
operator for #11710. #12125 (Mike). - Supported RIGHT and FULL JOIN with
SET join_algorithm = 'partial_merge'
. Only ALL strictness is allowed (ANY, SEMI, ANTI, ASOF are not). #12118 (Artem Zuikov). - Added a function
initializeAggregation
to initialize an aggregation based on a single value. #12109 (Guillaume Tassery). - Supported
ALTER TABLE ... [ADD|MODIFY] COLUMN ... FIRST
#4006. #12073 (Winter Zhang). - Added function
parseDateTimeBestEffortUS
. #12028 (flynn). - Support format
ORC
for output (was supported only for input). #11662 (Kruglov Pavel).
Bug Fix
- Fixed
aggregate function any(x) is found inside another aggregate function in query
error withSET optimize_move_functions_out_of_any = 1
and aliases insideany()
. #13419 (Artem Zuikov). - Fixed
PrettyCompactMonoBlock
for clickhouse-local. Fixed extremes/totals withPrettyCompactMonoBlock
. This fixes #7746. #13394 (Azat Khuzhin). - Fixed possible error
Totals having transform was already added to pipeline
in case of a query from delayed replica. #13290 (Nikolai Kochetov). - The server may crash if user passed specifically crafted arguments to the function
h3ToChildren
. This fixes #13275. #13277 (alexey-milovidov). - Fixed potentially low performance and slightly incorrect result for
uniqExact
,topK
,sumDistinct
and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes #12491. #13254 (alexey-milovidov). - Fixed function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. #13226 (alexey-milovidov).
- Fixed assert in
arrayElement
function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. #13224 (alexey-milovidov). - Fixed
DateTime64
conversion functions with constant argument. #13205 (Azat Khuzhin). - Fixed wrong index analysis with functions. It could lead to pruning wrong parts, while reading from
MergeTree
tables. Fixes #13060. Fixes #12406. #13081 (Anton Popov). - Fixed error
Cannot convert column because it is constant but values of constants are different in source and result
for remote queries which use deterministic functions in scope of query, but not deterministic between queries, likenow()
,now64()
,randConstant()
. Fixes #11327. #13075 (Nikolai Kochetov). - Fixed unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
- Fixed rare bug when
ALTER DELETE
andALTER MODIFY COLUMN
queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows incount.txt
and as a consequence incorrect data in part. Also, fix a small bug with simultaneousALTER RENAME COLUMN
andALTER ADD COLUMN
. #12760 (alesapin). - Fixed
CAST(Nullable(String), Enum())
. #12745 (Azat Khuzhin). - Fixed a performance with large tuples, which are interpreted as functions in
IN
section. The case when user writeWHERE x IN tuple(1, 2, ...)
instead ofWHERE x IN (1, 2, ...)
for some obscure reason. #12700 (Anton Popov). - Fixed memory tracking for
input_format_parallel_parsing
(by attaching thread to group). #12672 (Azat Khuzhin). - Fixed bloom filter index with const expression. This fixes #10572. #12659 (Winter Zhang).
- Fixed
SIGSEGV
inStorageKafka
when broker is unavailable (and not only). #12658 (Azat Khuzhin). - Added support for function
if
withArray(UUID)
arguments. This fixes #11066. #12648 (alexey-milovidov). CREATE USER IF NOT EXISTS
now does not throw exception if the user exists. This fixes #12507. #12646 (Vitaly Baranov).- Better exception message in disk access storage. #12625 (alesapin).
- The function
groupArrayMoving*
was not working for distributed queries. It's result was calculated within incorrect data type (without promotion to the largest type). The functiongroupArrayMovingAvg
was returning integer number that was inconsistent with theavg
function. This fixes #12568. #12622 (alexey-milovidov). - Fixed lack of aliases with function
any
. #12593 (Anton Popov). - Fixed race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
- Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 (Azat Khuzhin).
- Fixed bug which lead to broken old parts after
ALTER DELETE
query whenenable_mixed_granularity_parts=1
. Fixes #12536. #12543 (alesapin). - Better exception for function
in
with invalid number of arguments. #12529 (Anton Popov). - Fixing race condition in live view tables which could cause data duplication. #12519 (vzakaznikov).
- Fixed performance issue, while reading from compact parts. #12492 (Anton Popov).
- Fixed backwards compatibility in binary format of
AggregateFunction(avg, ...)
values. This fixes #12342. #12486 (alexey-milovidov). - Fixed SETTINGS parse after FORMAT. #12480 (Azat Khuzhin).
- Fixed the deadlock if
text_log
is enabled. #12452 (alexey-milovidov). - Fixed overflow when very large
LIMIT
orOFFSET
is specified. This fixes #10470. This fixes #11372. #12427 (alexey-milovidov). - Fixed possible segfault if
StorageMerge
. This fixes #12054. #12401 (tavplubix). - Reverted change introduced in #11079 to resolve #12098. #12397 (Mike).
- Additional check for arguments of bloom filter index. This fixes #11408. #12388 (alexey-milovidov).
- Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables. This fixes #11905. #12384 (alexey-milovidov).
- Allowed to
CLEAR
column even if there are dependingDEFAULT
expressions. This fixes #12333. #12378 (alexey-milovidov). - Fix
TOTALS/ROLLUP/CUBE
for aggregate functions with-State
andNullable
arguments. This fixes #12163. #12376 (alexey-milovidov). - Fixed error message and exit codes for
ALTER RENAME COLUMN
queries, whenRENAME
is not allowed. Fixes #12301 and #12303. #12335 (alesapin). - Fixed very rare race condition in
ReplicatedMergeTreeQueue
. #12315 (alexey-milovidov). - When using codec
Delta
orDoubleDelta
with non fixed width types, exception with codeLOGICAL_ERROR
was returned instead of exception with codeBAD_ARGUMENTS
(we ensure that exceptions with code logical error never happen). This fixes #12110. #12308 (alexey-milovidov). - Fixed order of columns in
WITH FILL
modifier. Previously order of columns ofORDER BY
statement wasn't respected. #12306 (Anton Popov). - Avoid "bad cast" exception when there is an expression that filters data by virtual columns (like
_table
inMerge
tables) or by "index" columns in system tables such as filtering by database name when querying fromsystem.tables
, and this expression returnsNullable
type. This fixes #12166. #12305 (alexey-milovidov). - Fixed
TTL
after renaming column, on which depends TTL expression. #12304 (Anton Popov). - Fixed SIGSEGV if there is an message with error in the middle of the batch in
Kafka
Engine. #12302 (Azat Khuzhin). - Fixed the situation when some threads might randomly hang for a few seconds during
DNS
cache updating. #12296 (tavplubix). - Fixed typo in setting name. #12292 (alexey-milovidov).
- Show error after
TrieDictionary
failed to load. #12290 (Vitaly Baranov). - The function
arrayFill
worked incorrectly for empty arrays that may lead to crash. This fixes #12263. #12279 (alexey-milovidov). - Implement conversions to the common type for
LowCardinality
types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes #8212. This fixes #4342. #12275 (alexey-milovidov). - Fixed the behaviour on reaching redirect limit in request to
S3
storage. #12256 (ianton-ru). - Fixed the behaviour when during multiple sequential inserts in
StorageFile
header for some special types was written more than once. This fixed #6155. #12197 (Nikita Mikhaylov). - Fixed logical functions for UInt8 values when they are not equal to 0 or 1. #12196 (Alexander Kazakov).
- Cap max_memory_usage* limits to the process resident memory. #12182 (Azat Khuzhin).
- Fix dictGet arguments check during
GROUP BY
injective functions elimination. #12179 (Azat Khuzhin). - Fixed the behaviour when
SummingMergeTree
engine sums up columns from partition key. Added an exception in case of explicit definition of columns to sum which intersects with partition key columns. This fixes #7867. #12173 (Nikita Mikhaylov). - Don't split the dictionary source's table name into schema and table name itself if ODBC connection does not support schema. #12165 (Vitaly Baranov).
- Fixed wrong logic in
ALTER DELETE
that leads to deleting of records when condition evaluates to NULL. This fixes #9088. This closes #12106. #12153 (alexey-milovidov). - Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. #12151 (alexey-milovidov).
- Fixed bad code in redundant ORDER BY optimization. The bug was introduced in #10067. #12148 (alexey-milovidov).
- Fixed potential overflow in integer division. This fixes #12119. #12140 (alexey-milovidov).
- Fixed potential infinite loop in
greatCircleDistance
,geoDistance
. This fixes #12117. #12137 (alexey-milovidov). - Normalize "pid" file handling. In previous versions the server may refuse to start if it was killed without proper shutdown and if there is another process that has the same pid as previously runned server. Also pid file may be removed in unsuccessful server startup even if there is another server running. This fixes #3501. #12133 (alexey-milovidov).
- Fixed bug which leads to incorrect table metadata in ZooKeepeer for ReplicatedVersionedCollapsingMergeTree tables. Fixes #12093. #12121 (alesapin).
- Avoid "There is no query" exception for materialized views with joins or with subqueries attached to system logs (system.query_log, metric_log, etc) or to engine=Buffer underlying table. #12120 (filimonov).
- Fixed handling dependency of table with ENGINE=Dictionary on dictionary. This fixes #10994. This fixes #10397. #12116 (Vitaly Baranov).
- Format
Parquet
now properly works withLowCardinality
andLowCardinality(Nullable)
types. Fixes #12086, #8406. #12108 (Nikolai Kochetov). - Fixed performance for selects with
UNION
caused by wrong limit for the total number of threads. Fixes #12030. #12103 (Nikolai Kochetov). - Fixed segfault with
-StateResample
combinators. #12092 (Anton Popov). - Fixed empty
result_rows
andresult_bytes
metrics insystem.quey_log
for selects. Fixes #11595. #12089 (Nikolai Kochetov). - Fixed unnecessary limiting the number of threads for selects from
VIEW
. Fixes #11937. #12085 (Nikolai Kochetov). - Fixed SIGSEGV in StorageKafka on DROP TABLE. #12075 (Azat Khuzhin).
- Fixed possible crash while using wrong type for
PREWHERE
. Fixes #12053, #12060. #12060 (Nikolai Kochetov). - Fixed error
Cannot capture column
for higher-order functions withTuple(LowCardinality)
argument. Fixes #9766. #12055 (Nikolai Kochetov). - Fixed constraints check if constraint is a constant expression. This fixes #11360. #12042 (alexey-milovidov).
- Fixed wrong result and potential crash when invoking function
if
with arguments of typeFixedString
with different sizes. This fixes #11362. #12021 (alexey-milovidov).
Improvement
- Allowed to set
JOIN
kind and type in more standard way:LEFT SEMI JOIN
instead ofSEMI LEFT JOIN
. For now both are correct. #12520 (Artem Zuikov). - lifetime_rows/lifetime_bytes for Buffer engine. #12421 (Azat Khuzhin).
- Write the detail exception message to the client instead of 'MySQL server has gone away'. #12383 (BohuTANG).
- Allows to change a charset which is used for printing grids borders. Available charsets are following: UTF-8, ASCII. Setting
output_format_pretty_grid_charset
enables this feature. #12372 (Sabyanin Maxim). - Supported MySQL 'SELECT DATABASE()' #9336 2. Add MySQL replacement query integration test. #12314 (BohuTANG).
- Added
KILL QUERY [connection_id]
for the MySQL client/driver to cancel the long query, issue #12038. #12152 (BohuTANG). - Added support for
%g
(two digit ISO year) and%G
(four digit ISO year) substitutions informatDateTime
function. #12136 (vivarum). - Added 'type' column in system.disks. #12115 (ianton-ru).
- Improved
REVOKE
command: now it requires grant/admin option for only access which will be revoked. For example, to executeREVOKE ALL ON *.* FROM user1
now it does not require to have full access rights granted with grant option. Added commandREVOKE ALL FROM user1
- it revokes all granted roles fromuser1
. #12083 (Vitaly Baranov). - Added replica priority for load_balancing (for manual prioritization of the load balancing). #11995 (Azat Khuzhin).
- Switched paths in S3 metadata to relative which allows to handle S3 blobs more easily. #11892 (Vladimir Chebotarev).
Performance Improvement
- Improved performace of 'ORDER BY' and 'GROUP BY' by prefix of sorting key (enabled with
optimize_aggregation_in_order
setting, disabled by default). #11696 (Anton Popov). - Removed injective functions inside
uniq*()
ifset optimize_injective_functions_inside_uniq=1
. #12337 (Ruslan Kamalov). - Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes #10574. #12062 (nvartolomei).
- Implemented single part uploads for DiskS3 (experimental feature). #12026 (Vladimir Chebotarev).
Experimental Feature
- Added new in-memory format of parts in
MergeTree
-family tables, which stores data in memory. Parts are written on disk at first merge. Part will be created in in-memory format if its size in rows or bytes is below thresholdsmin_rows_for_compact_part
andmin_bytes_for_compact_part
. Also optional support of Write-Ahead-Log is available, which is enabled by default and is controlled by settingin_memory_parts_enable_wal
. #10697 (Anton Popov).
Build/Testing/Packaging Improvement
- Implement AST-based query fuzzing mode for clickhouse-client. See this label for the list of issues we recently found by fuzzing. Most of them were found by this tool, and a couple by SQLancer and
00746_sql_fuzzy.pl
. #12111 (Alexander Kuzmenkov). - Add new type of tests based on Testflows framework. #12090 (vzakaznikov).
- Added S3 HTTPS integration test. #12412 (Pavel Kovalenko).
- Log sanitizer trap messages from separate thread. This will prevent possible deadlock under thread sanitizer. #12313 (alexey-milovidov).
- Now functional and stress tests will be able to run with old version of
clickhouse-test
script. #12287 (alesapin). - Remove strange file creation during build in
orc
. #12258 (Nikita Mikhaylov). - Place common docker compose files to integration docker container. #12168 (Ilya Yatsishin).
- Fix warnings from CodeQL.
CodeQL
is another static analyzer that we will use along withclang-tidy
andPVS-Studio
that we use already. #12138 (alexey-milovidov). - Minor CMake fixes for UNBUNDLED build. #12131 (Matwey V. Kornilov).
- Added a showcase of the minimal Docker image without using any Linux distribution. #12126 (alexey-milovidov).
- Perform an upgrade of system packages in the
clickhouse-server
docker image. #12124 (Ivan Blinkov). - Add
UNBUNDLED
flag tosystem.build_options
table. Move skip lists forclickhouse-test
to clickhouse repo. #12107 (alesapin). - Regular check by Anchore Container Analysis security analysis tool that looks for CVE in
clickhouse-server
Docker image. Also confirms thatDockerfile
is buildable. Runs daily onmaster
and on pull-requests toDockerfile
. #12102 (Ivan Blinkov). - Daily check by GitHub CodeQL security analysis tool that looks for CWE. #12101 (Ivan Blinkov).
- Install
ca-certificates
before the firstapt-get update
in Dockerfile. #12095 (Ivan Blinkov).
ClickHouse release 20.5
ClickHouse release v20.5.4.40-stable 2020-08-10
Bug Fix
- Fixed wrong index analysis with functions. It could lead to pruning wrong parts, while reading from
MergeTree
tables. Fixes #13060. Fixes #12406. #13081 (Anton Popov). - Fixed unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
- Fixed performance with large tuples, which are interpreted as functions in
IN
section. The case when user writeWHERE x IN tuple(1, 2, ...)
instead ofWHERE x IN (1, 2, ...)
for some obscure reason. #12700 (Anton Popov). - Fixed memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 (Azat Khuzhin).
- Fixed bloom filter index with const expression. This fixes #10572. #12659 (Winter Zhang).
- Fixed
SIGSEGV
inStorageKafka
when broker is unavailable (and not only). #12658 (Azat Khuzhin). - Added support for function
if
withArray(UUID)
arguments. This fixes #11066. #12648 (alexey-milovidov). - Fixed lack of aliases with function
any
. #12593 (Anton Popov). - Fixed race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
- Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 (Azat Khuzhin).
- Fixed bug which lead to broken old parts after
ALTER DELETE
query whenenable_mixed_granularity_parts=1
. Fixes #12536. #12543 (alesapin). - Better exception for function
in
with invalid number of arguments. #12529 (Anton Popov). - Fixed race condition in live view tables which could cause data duplication. #12519 (vzakaznikov).
- Fixed performance issue, while reading from compact parts. #12492 (Anton Popov).
- Fixed backwards compatibility in binary format of
AggregateFunction(avg, ...)
values. This fixes #12342. #12486 (alexey-milovidov). - Fixed the deadlock if
text_log
is enabled. #12452 (alexey-milovidov). - Fixed overflow when very large LIMIT or OFFSET is specified. This fixes #10470. This fixes #11372. #12427 (alexey-milovidov).
- Fixed possible segfault if StorageMerge. Closes #12054. #12401 (tavplubix).
- Reverts change introduced in #11079 to resolve #12098. #12397 (Mike).
- Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables. This fixes #11905. #12384 (alexey-milovidov).
- Allow to CLEAR column even if there are depending DEFAULT expressions. This fixes #12333. #12378 (alexey-milovidov).
- Fixed TOTALS/ROLLUP/CUBE for aggregate functions with
-State
andNullable
arguments. This fixes #12163. #12376 (alexey-milovidov). - Fixed SIGSEGV if there is an message with error in the middle of the batch in
Kafka
Engine. #12302 (Azat Khuzhin). - Fixed the behaviour when
SummingMergeTree
engine sums up columns from partition key. Added an exception in case of explicit definition of columns to sum which intersects with partition key columns. This fixes #7867. #12173 (Nikita Mikhaylov). - Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. #12151 (alexey-milovidov).
- Fixed bug which leads to incorrect table metadata in ZooKeepeer for ReplicatedVersionedCollapsingMergeTree tables. Fixes #12093. #12121 (alesapin).
- Fixed unnecessary limiting the number of threads for selects from
VIEW
. Fixes #11937. #12085 (Nikolai Kochetov). - Fixed crash in JOIN with LowCardinality type with
join_algorithm=partial_merge
. #12035 (Artem Zuikov). - Fixed wrong result for
if()
with NULLs in condition. #11807 (Artem Zuikov).
Performance Improvement
- Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes #10574. #12062 (nvartolomei).
Build/Testing/Packaging Improvement
- Install
ca-certificates
before the firstapt-get update
in Dockerfile. #12095 (Ivan Blinkov).
ClickHouse release v20.5.2.7-stable 2020-07-02
Backward Incompatible Change
- Return non-Nullable result from COUNT(DISTINCT), and
uniq
aggregate functions family. If all passed values are NULL, return zero instead. This improves SQL compatibility. #11661 (alexey-milovidov). - Added a check for the case when user-level setting is specified in a wrong place. User-level settings should be specified in
users.xml
inside<profile>
section for specific user profile (or in<default>
for default settings). The server won't start with exception message in log. This fixes #9051. If you want to skip the check, you can either move settings to the appropriate place or add<skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings>
to config.xml. #11449 (alexey-milovidov). - The setting
input_format_with_names_use_header
is enabled by default. It will affect parsing of input formats-WithNames
and-WithNamesAndTypes
. #10937 (alexey-milovidov). - Remove
experimental_use_processors
setting. It is enabled by default. #10924 (Nikolai Kochetov). - Update
zstd
to 1.4.4. It has some minor improvements in performance and compression ratio. If you run replicas with different versions of ClickHouse you may see reasonable error messagesData after merge is not byte-identical to data on another replicas.
with explanation. These messages are Ok and you should not worry. This change is backward compatible but we list it here in changelog in case you will wonder about these messages. #10663 (alexey-milovidov). - Added a check for meaningless codecs and a setting
allow_suspicious_codecs
to control this check. This closes #4966. #10645 (alexey-milovidov). - Several Kafka setting changes their defaults. See #11388.
- When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to
Part ... intersects previous part
errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).
New Feature
TTL DELETE WHERE
andTTL GROUP BY
for automatic data coarsening and rollup in tables. #10537 (expl0si0nn).- Implementation of PostgreSQL wire protocol. #10242 (Movses).
- Added system tables for users, roles, grants, settings profiles, quotas, row policies; added commands SHOW USER, SHOW [CURRENT|ENABLED] ROLES, SHOW SETTINGS PROFILES. #10387 (Vitaly Baranov).
- Support writes in ODBC Table function #10554 (ageraab). #10901 (tavplubix).
- Add query performance metrics based on Linux
perf_events
(these metrics are calculated with hardware CPU counters and OS counters). It is optional and requiresCAP_SYS_ADMIN
to be set on clickhouse binary. #9545 Andrey Skobtsov. #11226 (Alexander Kuzmenkov). - Now support
NULL
andNOT NULL
modifiers for data types inCREATE
query. #11057 (Павел Потемкин). - Add
ArrowStream
input and output format. #11088 (hcz). - Support Cassandra as external dictionary source. #4978 (favstovol).
- Added a new layout
direct
which loads all the data directly from the source for each query, without storing or caching data. #10622 (Artem Streltsov). - Added new
complex_key_direct
layout to dictionaries, that does not store anything locally during query execution. #10850 (Artem Streltsov). - Added support for MySQL style global variables syntax (stub). This is needed for compatibility of MySQL protocol. #11832 (alexey-milovidov).
- Added syntax highligting to
clickhouse-client
usingreplxx
. #11422 (Tagir Kuskarov). minMap
andmaxMap
functions were added. #11603 (Ildus Kurbangaliev).- Add the
system.asynchronous_metric_log
table that logs historical metrics fromsystem.asynchronous_metrics
. #11588 (Alexander Kuzmenkov). - Add functions
extractAllGroupsHorizontal(haystack, re)
andextractAllGroupsVertical(haystack, re)
. #11554 (Vasily Nemkov). - Add SHOW CLUSTER(S) queries. #11467 (hexiaoting).
- Add
netloc
function for extracting network location, similar tourlparse(url)
,netloc
in python. #11356 (Guillaume Tassery). - Add 2 more virtual columns for engine=Kafka to access message headers. #11283 (filimonov).
- Add
_timestamp_ms
virtual column for Kafka engine (type isNullable(DateTime64(3))
). #11260 (filimonov). - Add function
randomFixedString
. #10866 (Andrei Nekrashevich). - Add function
fuzzBits
that randomly flips bits in a string with given probability. #11237 (Andrei Nekrashevich). - Allow comparison of numbers with constant string in comparison operators, IN and VALUES sections. #11647 (alexey-milovidov).
- Add
round_robin
load_balancing mode. #11645 (Azat Khuzhin). - Add
cast_keep_nullable
setting. If setCAST(something_nullable AS Type)
returnNullable(Type)
. #11733 (Artem Zuikov). - Added column
position
tosystem.columns
table andcolumn_position
tosystem.parts_columns
table. It contains ordinal position of a column in a table starting with 1. This closes #7744. #11655 (alexey-milovidov). - ON CLUSTER support for SYSTEM {FLUSH DISTRIBUTED,STOP/START DISTRIBUTED SEND}. #11415 (Azat Khuzhin).
- Add system.distribution_queue table. #11394 (Azat Khuzhin).
- Support for all format settings in Kafka, expose some setting on table level, adjust the defaults for better performance. #11388 (filimonov).
- Add
port
function (to extract port from URL). #11120 (Azat Khuzhin). - Now
dictGet*
functions accept table names. #11050 (Vitaly Baranov). - The
clickhouse-format
tool is now able to format multiple queries when the-n
argument is used. #10852 (Darío). - Possibility to configure proxy-resolver for DiskS3. #10744 (Pavel Kovalenko).
- Make
pointInPolygon
work with non-constant polygon. PointInPolygon now can take Array(Array(Tuple(..., ...))) as second argument, array of polygon and holes. #10623 (Alexey Ilyukhov) #11421 (Alexey Ilyukhov). - Added
move_ttl_info
tosystem.parts
in order to provide introspection of move TTL functionality. #10591 (Vladimir Chebotarev). - Possibility to work with S3 through proxies. #10576 (Pavel Kovalenko).
- Add
NCHAR
andNVARCHAR
synonims for data types. #11025 (alexey-milovidov). - Resolved #7224: added
FailedQuery
,FailedSelectQuery
andFailedInsertQuery
metrics tosystem.events
table. #11151 (Nikita Orlov). - Add more
jemalloc
statistics tosystem.asynchronous_metrics
, and ensure that we see up-to-date values for them. #11748 (Alexander Kuzmenkov). - Allow to specify default S3 credentials and custom auth headers. #11134 (Grigory Pervakov).
- Added new functions to import/export DateTime64 as Int64 with various precision:
to-/fromUnixTimestamp64Milli/-Micro/-Nano
. #10923 (Vasily Nemkov). - Allow specifying
mongodb://
URI for MongoDB dictionaries. #10915 (Alexander Kuzmenkov). - OFFSET keyword can now be used without an affiliated LIMIT clause. #10802 (Guillaume Tassery).
- Added
system.licenses
table. This table contains licenses of third-party libraries that are located incontrib
directory. This closes #2890. #10795 (alexey-milovidov). - New function function toStartOfSecond(DateTime64) -> DateTime64 that nullifies sub-second part of DateTime64 value. #10722 (Vasily Nemkov).
- Add new input format
JSONAsString
that accepts a sequence of JSON objects separated by newlines, spaces and/or commas. #10607 (Kruglov Pavel). - Allowed to profile memory with finer granularity steps than 4 MiB. Added sampling memory profiler to capture random allocations/deallocations. #10598 (alexey-milovidov).
SimpleAggregateFunction
now also supportssumMap
. #10000 (Ildus Kurbangaliev).- Support
ALTER RENAME COLUMN
for the distributed table engine. Continuation of #10727. Fixes #10747. #10887 (alesapin).
Bug Fix
- Fix UBSan report in Decimal parse. This fixes #7540. #10512 (alexey-milovidov).
- Fix potential floating point exception when parsing DateTime64. This fixes #11374. #11875 (alexey-milovidov).
- Fix rare crash caused by using
Nullable
column in prewhere condition. #11895 #11608 #11869 (Nikolai Kochetov). - Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (alexey-milovidov).
- Fix wrong result of comparison of FixedString with constant String. This fixes #11393. This bug appeared in version 20.4. #11828 (alexey-milovidov).
- Fix wrong result for
if
with NULLs in condition. #11807 (Artem Zuikov). - Fix using too many threads for queries. #11788 (Nikolai Kochetov).
- Fixed
Scalar does not exist
exception when usingWITH <scalar subquery> ...
inSELECT ... FROM merge_tree_table ...
#11621. #11767 (Amos Bird). - Fix unexpected behaviour of queries like
SELECT *, xyz.*
which were success while an error expected. #11753 (hexiaoting). - Now replicated fetches will be cancelled during metadata alter. #11744 (alesapin).
- Parse metadata stored in zookeeper before checking for equality. #11739 (Azat Khuzhin).
- Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (tavplubix).
- Fix
ORDER BY ... WITH FILL
over const columns. #11697 (Anton Popov). - Fix very rare race condition in SYSTEM SYNC REPLICA. If the replicated table is created and at the same time from the separate connection another client is issuing
SYSTEM SYNC REPLICA
command on that table (this is unlikely, because another client should be aware that the table is created), it's possible to get nullptr dereference. #11691 (alexey-milovidov). - Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (alexey-milovidov).
- Fix
LIMIT n WITH TIES
usage together withORDER BY
statement, which contains aliases. #11689 (Anton Popov). - Fix possible
Pipeline stuck
for selects with parallelFINAL
. Fixes #11636. #11682 (Nikolai Kochetov). - Fix error which leads to an incorrect state of
system.mutations
. It may show that whole mutation is already done but the server still hasMUTATE_PART
tasks in the replication queue and tries to execute them. This fixes #11611. #11681 (alesapin). - Fix syntax hilite in CREATE USER query. #11664 (alexey-milovidov).
- Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (alexey-milovidov).
- Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. #11644 (alexey-milovidov).
- Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
- Without
-q
option the database does not get created at startup. #11604 (giordyb). - Fix error
Block structure mismatch
for queries with sampling reading fromBuffer
table. #11602 (Nikolai Kochetov). - Fix wrong exit code of the clickhouse-client, when
exception.code() % 256 == 0
. #11601 (filimonov). - Fix race conditions in CREATE/DROP of different replicas of ReplicatedMergeTree. Continue to work if the table was not removed completely from ZooKeeper or not created successfully. This fixes #11432. #11592 (alexey-milovidov).
- Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (alexey-milovidov).
- Fix error
Size of offsets does not match size of column
for queries withPREWHERE column in (subquery)
andARRAY JOIN
. #11580 (Nikolai Kochetov). - Fixed rare segfault in
SHOW CREATE TABLE
Fixes #11490. #11579 (tavplubix). - All queries in HTTP session have had the same query_id. It is fixed. #11578 (tavplubix).
- Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
- Fix the error
Data compressed with different methods
that can happen ifmin_bytes_to_use_direct_io
is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (alexey-milovidov). - Fix shard_num/replica_num for
<node>
(breaks use_compact_format_in_distributed_parts_names). #11528 (Azat Khuzhin). - Fix async INSERT into Distributed for prefer_localhost_replica=0 and w/o internal_replication. #11527 (Azat Khuzhin).
- Fix memory leak when exception is thrown in the middle of aggregation with
-State
functions. This fixes #8995. #11496 (alexey-milovidov). - Fix
Pipeline stuck
exception forINSERT SELECT FINAL
whereSELECT
(max_threads
>1) has multiple streams butINSERT
has only one (max_insert_threads
==0). #11455 (Azat Khuzhin). - Fix wrong result in queries like
select count() from t, u
. #11454 (Artem Zuikov). - Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
- Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
- Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. #11420 (alexey-milovidov).
- Fix crash in JOIN over
LowCarinality(T)
andNullable(T)
. #11380. #11414 (Artem Zuikov). - Fix error code for wrong
USING
key. #11373. #11404 (Artem Zuikov). - Fixed
geohashesInBox
with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov). - Better errors for
joinGet()
functions. #11389 (Artem Zuikov). - Fix possible
Pipeline stuck
error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov). - Remove redundant lock during parts send in ReplicatedMergeTree. #11354 (alesapin).
- Fix support for
\G
(vertical output) in clickhouse-client in multiline mode. This closes #9933. #11350 (alexey-milovidov). - Fix potential segfault when using
Lazy
database. #11348 (alexey-milovidov). - Fix crash in direct selects from
Join
table engine (without JOIN) and wrong nullability. #11340 (Artem Zuikov). - Fix crash in
quantilesExactWeightedArray
. #11337 (Nikolai Kochetov). - Now merges stopped before change metadata in
ALTER
queries. #11335 (alesapin). - Make writing to
MATERIALIZED VIEW
with settingparallel_view_processing = 1
parallel again. Fixes #10241. #11330 (Nikolai Kochetov). - Fix
visitParamExtractRaw
when extracted JSON has strings with unbalanced { or [. #11318 (Ewout). - Fix very rare race condition in ThreadPool. #11314 (alexey-milovidov).
- Fix insignificant data race in
clickhouse-copier
. Found by integration tests. #11313 (alexey-milovidov). - Fix potential uninitialized memory in conversion. Example:
SELECT toIntervalSecond(now64())
. #11311 (alexey-milovidov). - Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with
empty
ornotEmpty
functions. This fixes #11286. #11303 (alexey-milovidov). - Fix bug when query speed estimation can be incorrect and the limit of
min_execution_speed
may not work or work incorrectly if the query is throttled bymax_network_bandwidth
,max_execution_speed
orpriority
settings. Change the default value oftimeout_before_checking_execution_speed
to non-zero, because otherwise the settingsmin_execution_speed
andmax_execution_speed
have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar inclickhouse-client
. #11296 (alexey-milovidov). - Fix crash when
SET DEFAULT ROLE
is called with wrong arguments. This fixes #10586. #11278 (Vitaly Baranov). - Fix crash while reading malformed data in
Protobuf
format. This fixes #5957, fixes #11203. #11258 (Vitaly Baranov). - Fixed a bug when
cache
dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 (Nikita Mikhaylov). - Fix error
Block structure mismatch in QueryPipeline
while reading fromVIEW
with constants in inner query. Fixes #11181. #11205 (Nikolai Kochetov). - Fix possible exception
Invalid status for associated output
. #11200 (Nikolai Kochetov). - Now
primary.idx
will be checked if it's defined inCREATE
query. #11199 (alesapin). - Fix possible error
Cannot capture column
for higher-order functions withArray(Array(LowCardinality))
captured argument. #11185 (Nikolai Kochetov). - Fixed
S3
globbing which could fail in case of more than 1000 keys and some backends. #11179 (Vladimir Chebotarev). - If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
- Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #11145 (filimonov).
- Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 (Azat Khuzhin).
- Remove logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
- Fixed deadlock during server startup after update with changes in structure of system log tables. #11106 (alesapin).
- Fixed memory leak in registerDiskS3. #11074 (Pavel Kovalenko).
- Fix error
No such name in Block::erase()
when JOIN appears with PREWHERE oroptimize_move_to_prewhere
makes PREWHERE from WHERE. #11051 (Artem Zuikov). - Fixes the potential missed data during termination of Kafka engine table. #11048 (filimonov).
- Fixed parseDateTime64BestEffort argument resolution bugs. #10925. #11038 (Vasily Nemkov).
- Now it's possible to
ADD/DROP
andRENAME
the same one column in a singleALTER
query. Exception message for simultaneousMODIFY
andRENAME
became more clear. Partially fixes #10669. #11037 (alesapin). - Fixed parsing of S3 URLs. #11036 (Vladimir Chebotarev).
- Fix memory tracking for two-level
GROUP BY
when there is aLIMIT
. #11022 (Azat Khuzhin). - Fix very rare potential use-after-free error in MergeTree if table was not created successfully. #10986 (alexey-milovidov).
- Fix metadata (relative path for rename) and data (relative path for symlink) handling for Atomic database. #10980 (Azat Khuzhin).
- Fix server crash on concurrent
ALTER
andDROP DATABASE
queries withAtomic
database engine. #10968 (tavplubix). - Fix incorrect raw data size in method getRawData(). #10964 (Igr).
- Fix incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 (alexey-milovidov).
- Avoid sending partially written files by the DistributedBlockOutputStream. #10940 (Azat Khuzhin).
- Fix crash in
SELECT count(notNullIn(NULL, []))
. #10920 (Nikolai Kochetov). - Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #10910 (filimonov).
- Now it's possible to execute multiple
ALTER RENAME
likea TO b, c TO a
. #10895 (alesapin). - Fix possible race which could happen when you get result from aggregate function state from multiple thread for the same column. The only way (which I found) it can happen is when you use
finalizeAggregation
function while reading from table withMemory
engine which storesAggregateFunction
state forquanite*
function. #10890 (Nikolai Kochetov). - Fix backward compatibility with tuples in Distributed tables. #10889 (Anton Popov).
- Fix SIGSEGV in StringHashTable (if such key does not exist). #10870 (Azat Khuzhin).
- Fixed
WATCH
hangs afterLiveView
table was dropped from database withAtomic
engine. #10859 (tavplubix). - Fixed bug in
ReplicatedMergeTree
which might cause someALTER
onOPTIMIZE
query to hang waiting for some replica after it become inactive. #10849 (tavplubix). - Now constraints are updated if the column participating in
CONSTRAINT
expression was renamed. Fixes #10844. #10847 (alesapin). - Fix potential read of uninitialized memory in cache dictionary. #10834 (alexey-milovidov).
- Fix columns order after Block::sortColumns() (also add a test that shows that it affects some real use case - Buffer engine). #10826 (Azat Khuzhin).
- Fix the issue with ODBC bridge when no quoting of identifiers is requested. This fixes #7984. #10821 (alexey-milovidov).
- Fix UBSan and MSan report in DateLUT. #10798 (alexey-milovidov).
- Make use of
src_type
for correct type conversion in key conditions. Fixes #6287. #10791 (Andrew Onyshchuk). - Get rid of old libunwind patches. https://github.com/ClickHouse-Extras/libunwind/commit/500aa227911bd185a94bfc071d68f4d3b03cb3b1#r39048012 This allows to disable
-fno-omit-frame-pointer
inclang
builds that improves performance at least by 1% in average. #10761 (Amos Bird). - Fix avgWeighted when using floating-point weight over multiple shards. #10758 (Baudouin Giard).
- Fix
parallel_view_processing
behavior. Now all insertions intoMATERIALIZED VIEW
without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov). - Fix combinator -OrNull and -OrDefault when combined with -State. #10741 (hcz).
- Fix crash in
generateRandom
with nested types. Fixes #10583. #10734 (Nikolai Kochetov). - Fix data corruption for
LowCardinality(FixedString)
key column inSummingMergeTree
which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov). - Fix usage of primary key wrapped into a function with 'FINAL' modifier and 'ORDER BY' optimization. #10715 (Anton Popov).
- Fix possible buffer overflow in function
h3EdgeAngle
. #10711 (alexey-milovidov). - Fix disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes #10674. #10698 (Nikolai Kochetov).
- Fix atomicity of HTTP insert. This fixes #9666. #10687 (Andrew Onyshchuk).
- Fix multiple usages of
IN
operator with the identical set in one query. #10686 (Anton Popov). - Fixed bug, which causes http requests stuck on client close when
readonly=2
andcancel_http_readonly_queries_on_client_close=1
. Fixes #7939, #7019, #7736, #7091. #10684 (tavplubix). - Fix order of parameters in AggregateTransform constructor. #10667 (palasonic1).
- Fix the lack of parallel execution of remote queries with
distributed_aggregation_memory_efficient
enabled. Fixes #10655. #10664 (Nikolai Kochetov). - Fix possible incorrect number of rows for queries with
LIMIT
. Fixes #10566, #10709. #10660 (Nikolai Kochetov). - Fix bug which locks concurrent alters when table has a lot of parts. #10659 (alesapin).
- Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 (alexey-milovidov).
- Fix predicates optimization for distributed queries (
enable_optimize_predicate_expression=1
) for queries withHAVING
section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin). - Fix optimize_skip_unused_shards with LowCardinality. #10611 (Azat Khuzhin).
- Fix segfault in StorageBuffer when exception on server startup. Fixes #10550. #10609 (tavplubix).
- On
SYSTEM DROP DNS CACHE
query also drop caches, which are used to check if user is allowed to connect from some IP addresses. #10608 (tavplubix). - Fixed incorrect scalar results inside inner query of
MATERIALIZED VIEW
in case if this query contained dependent table. #10603 (Nikolai Kochetov). - Fixed handling condition variable for synchronous mutations. In some cases signals to that condition variable could be lost. #10588 (Vladimir Chebotarev).
- Fixes possible crash
createDictionary()
is called beforeloadStoredObject()
has finished. #10587 (Vitaly Baranov). - Fix error
the BloomFilter false positive must be a double number between 0 and 1
#10551. #10569 (Winter Zhang). - Fix SELECT of column ALIAS which default expression type different from column type. #10563 (Azat Khuzhin).
- Implemented comparison between DateTime64 and String values (just like for DateTime). #10560 (Vasily Nemkov).
- Fix index corruption, which may occur in some cases after merge compact parts into another compact part. #10531 (Anton Popov).
- Disable GROUP BY sharding_key optimization by default (
optimize_distributed_group_by_sharding_key
had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example isif
in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. #10516 (Azat Khuzhin). - Fixes: #10263 (after that PR dist send via INSERT had been postponing on each INSERT) Fixes: #8756 (that PR breaks distributed sends with all of the following conditions met (unlikely setup for now I guess):
internal_replication == false
, multiple local shards (activates the hardlinking code) anddistributed_storage_policy
(makeslink(2)
fails onEXDEV
)). #10486 (Azat Khuzhin). - Fixed error with "max_rows_to_sort" limit. #10268 (alexey-milovidov).
- Get dictionary and check access rights only once per each call of any function reading external dictionaries. #10928 (Vitaly Baranov).
Improvement
- Apply
TTL
for old data, afterALTER MODIFY TTL
query. This behaviour is controlled by settingmaterialize_ttl_after_modify
, which is enabled by default. #11042 (Anton Popov). - When parsing C-style backslash escapes in string literals, VALUES and various text formats (this is an extension to SQL standard that is endemic for ClickHouse and MySQL), keep backslash if unknown escape sequence is found (e.g.
\%
or\w
) that will make usage ofLIKE
andmatch
regular expressions more convenient (it's enough to writename LIKE 'used\_cars'
instead ofname LIKE 'used\\_cars'
) and more compatible at the same time. This fixes #10922. #11208 (alexey-milovidov). - When reading Decimal value, cut extra digits after point. This behaviour is more compatible with MySQL and PostgreSQL. This fixes #10202. #11831 (alexey-milovidov).
- Allow to DROP replicated table if the metadata in ZooKeeper was already removed and does not exist (this is also the case when using TestKeeper for testing and the server was restarted). Allow to RENAME replicated table even if there is an error communicating with ZooKeeper. This fixes #10720. #11652 (alexey-milovidov).
- Slightly improve diagnostic of reading decimal from string. This closes #10202. #11829 (alexey-milovidov).
- Fix sleep invocation in signal handler. It was sleeping for less amount of time than expected. #11825 (alexey-milovidov).
- (Only Linux) OS related performance metrics (for CPU and I/O) will work even without
CAP_NET_ADMIN
capability. #10544 (Alexander Kazakov). - Added
hostname
as an alias to functionhostName
. This feature was suggested by Victor Tarnavskiy from Yandex.Metrica. #11821 (alexey-milovidov). - Added support for distributed
DDL
(update/delete/drop partition) on cross replication clusters. #11703 (Nikita Mikhaylov). - Emit warning instead of error in server log at startup if we cannot listen one of the listen addresses (e.g. IPv6 is unavailable inside Docker). Note that if server fails to listen all listed addresses, it will refuse to startup as before. This fixes #4406. #11687 (alexey-milovidov).
- Default user and database creation on docker image starting. #10637 (Paramtamtam).
- When multiline query is printed to server log, the lines are joined. Make it to work correct in case of multiline string literals, identifiers and single-line comments. This fixes #3853. #11686 (alexey-milovidov).
- Multiple names are now allowed in commands: CREATE USER, CREATE ROLE, ALTER USER, SHOW CREATE USER, SHOW GRANTS and so on. #11670 (Vitaly Baranov).
- Add support for distributed DDL (
UPDATE/DELETE/DROP PARTITION
) on cross replication clusters. #11508 (frank lee). - Clear password from command line in
clickhouse-client
andclickhouse-benchmark
if the user has specified it with explicit value. This prevents password exposure byps
and similar tools. #11665 (alexey-milovidov). - Don't use debug info from ELF file if it does not correspond to the running binary. It is needed to avoid printing wrong function names and source locations in stack traces. This fixes #7514. #11657 (alexey-milovidov).
- Return NULL/zero when value is not parsed completely in parseDateTimeBestEffortOrNull/Zero functions. This fixes #7876. #11653 (alexey-milovidov).
- Skip empty parameters in requested URL. They may appear when you write
http://localhost:8123/?&a=b
orhttp://localhost:8123/?a=b&&c=d
. This closes #10749. #11651 (alexey-milovidov). - Allow using
groupArrayArray
andgroupUniqArrayArray
asSimpleAggregateFunction
. #11650 (Volodymyr Kuznetsov). - Allow comparison with constant strings by implicit conversions when analysing index conditions on other types. This may close #11630. #11648 (alexey-milovidov).
- https://github.com/ClickHouse/ClickHouse/pull/7572#issuecomment-642815377 Support config default HTTPHandlers. #11628 (Winter Zhang).
- Make more input formats to work with Kafka engine. Fix the issue with premature flushes. Fix the performance issue when
kafka_num_consumers
is greater than number of partitions in topic. #11599 (filimonov). - Improve
multiple_joins_rewriter_version=2
logic. Fix unknown columns error for lambda aliases. #11587 (Artem Zuikov). - Better exception message when cannot parse columns declaration list. This closes #10403. #11537 (alexey-milovidov).
- Improve
enable_optimize_predicate_expression=1
logic for VIEW. #11513 (Artem Zuikov). - Adding support for PREWHERE in live view tables. #11495 (vzakaznikov).
- Automatically update DNS cache, which is used to check if user is allowed to connect from an address. #11487 (tavplubix).
- OPTIMIZE FINAL will force merge even if concurrent merges are performed. This closes #11309 and closes #11322. #11346 (alexey-milovidov).
- Suppress output of cancelled queries in clickhouse-client. In previous versions result may continue to print in terminal even after you press Ctrl+C to cancel query. This closes #9473. #11342 (alexey-milovidov).
- Now history file is updated after each query and there is no race condition if multiple clients use one history file. This fixes #9897. #11453 (Tagir Kuskarov).
- Better log messages in while reloading configuration. #11341 (alexey-milovidov).
- Remove trailing whitespaces from formatted queries in
clickhouse-client
orclickhouse-format
in some cases. #11325 (alexey-milovidov). - Add setting "output_format_pretty_max_value_width". If value is longer, it will be cut to avoid output of too large values in terminal. This closes #11140. #11324 (alexey-milovidov).
- Better exception message in case when there is shortage of memory mappings. This closes #11027. #11316 (alexey-milovidov).
- Support (U)Int8, (U)Int16, Date in ASOF JOIN. #11301 (Artem Zuikov).
- Support kafka_client_id parameter for Kafka tables. It also changes the default
client.id
used by ClickHouse when communicating with Kafka to be more verbose and usable. #11252 (filimonov). - Keep the value of
DistributedFilesToInsert
metric on exceptions. In previous versions, the value was set when we are going to send some files, but it is zero, if there was an exception and some files are still pending. Now it corresponds to the number of pending files in filesystem. #11220 (alexey-milovidov). - Add support for multi-word data type names (such as
DOUBLE PRECISION
andCHAR VARYING
) for better SQL compatibility. #11214 (Павел Потемкин). - Provide synonyms for some data types. #10856 (Павел Потемкин).
- The query log is now enabled by default. #11184 (Ivan Blinkov).
- Show authentication type in table system.users and while executing SHOW CREATE USER query. #11080 (Vitaly Baranov).
- Remove data on explicit
DROP DATABASE
forMemory
database engine. Fixes #10557. #11021 (tavplubix). - Set thread names for internal threads of rdkafka library. Make logs from rdkafka available in server logs. #10983 (Azat Khuzhin).
- Support for unicode whitespaces in queries. This helps when queries are copy-pasted from Word or from web page. This fixes #10896. #10903 (alexey-milovidov).
- Allow large UInt types as the index in function
tupleElement
. #10874 (hcz). - Respect prefer_localhost_replica/load_balancing on INSERT into Distributed. #10867 (Azat Khuzhin).
- Introduce
min_insert_block_size_rows_for_materialized_views
,min_insert_block_size_bytes_for_materialized_views
settings. This settings are similar tomin_insert_block_size_rows
andmin_insert_block_size_bytes
, but applied only for blocks inserted intoMATERIALIZED VIEW
. It helps to control blocks squashing while pushing to MVs and avoid excessive memory usage. #10858 (Azat Khuzhin). - Get rid of exception from replicated queue during server shutdown. Fixes #10819. #10841 (alesapin).
- Ensure that
varSamp
,varPop
cannot return negative results due to numerical errors and thatstddevSamp
,stddevPop
cannot be calculated from negative variance. This fixes #10532. #10829 (alexey-milovidov). - Better DNS exception message. This fixes #10813. #10828 (alexey-milovidov).
- Change HTTP response code in case of some parse errors to 400 Bad Request. This fix #10636. #10640 (alexey-milovidov).
- Print a message if clickhouse-client is newer than clickhouse-server. #10627 (alexey-milovidov).
- Adding support for
INSERT INTO [db.]table WATCH
query. #10498 (vzakaznikov). - Allow to pass quota_key in clickhouse-client. This closes #10227. #10270 (alexey-milovidov).
Performance Improvement
- Allow multiple replicas to assign merges, mutations, partition drop, move and replace concurrently. This closes #10367. #11639 (alexey-milovidov) #11795 (alexey-milovidov).
- Optimization of GROUP BY with respect to table sorting key, enabled with
optimize_aggregation_in_order
setting. #9113 (dimarub2000). - Selects with final are executed in parallel. Added setting
max_final_threads
to limit the number of threads used. #10463 (Nikolai Kochetov). - Improve performance for INSERT queries via
INSERT SELECT
or INSERT with clickhouse-client when small blocks are generated (typical case with parallel parsing). This fixes #11275. Fix the issue that CONSTRAINTs were not working for DEFAULT fields. This fixes #11273. Fix the issue that CONSTRAINTS were ignored for TEMPORARY tables. This fixes #11274. #11276 (alexey-milovidov). - Optimization that eliminates min/max/any aggregators of GROUP BY keys in SELECT section, enabled with
optimize_aggregators_of_group_by_keys
setting. #11667 (xPoSx). #11806 (Azat Khuzhin). - New optimization that takes all operations out of
any
function, enabled withoptimize_move_functions_out_of_any
#11529 (Ruslan). - Improve performance of
clickhouse-client
in interactive mode when Pretty formats are used. In previous versions, significant amount of time can be spent calculating visible width of UTF-8 string. This closes #11323. #11323 (alexey-milovidov). - Improved performance for queries with
ORDER BY
and smallLIMIT
(less, thenmax_block_size
). #11171 (Albert Kidrachev). - Add runtime CPU detection to select and dispatch the best function implementation. Add support for codegeneration for multiple targets. This closes #1017. #10058 (DimasKovas).
- Enable
mlock
of clickhouse binary by default. It will prevent clickhouse executable from being paged out under high IO load. #11139 (alexey-milovidov). - Make queries with
sum
aggregate function and without GROUP BY keys to run multiple times faster. #10992 (alexey-milovidov). - Improving radix sort (used in
ORDER BY
with simple keys) by removing some redundant data moves. #10981 (Arslan Gumerov). - Sort bigger parts of the left table in MergeJoin. Buffer left blocks in memory. Add
partial_merge_join_left_table_buffer_bytes
setting to manage the left blocks buffers sizes. #10601 (Artem Zuikov). - Remove duplicate ORDER BY and DISTINCT from subqueries, this optimization is enabled with
optimize_duplicate_order_by_and_distinct
#10067 (Mikhail Malafeev). - This feature eliminates functions of other keys in GROUP BY section, enabled with
optimize_group_by_function_keys
#10051 (xPoSx). - New optimization that takes arithmetic operations out of aggregate functions, enabled with
optimize_arithmetic_operations_in_aggregate_functions
#10047 (Ruslan). - Use HTTP client for S3 based on Poco instead of curl. This will improve performance and lower memory usage of s3 storage and table functions. #11230 (Pavel Kovalenko).
- Fix Kafka performance issue related to reschedules based on limits, which were always applied. #11149 (filimonov).
- Enable percpu_arena:percpu for jemalloc (This will reduce memory fragmentation due to thread pool). #11084 (Azat Khuzhin).
- Optimize memory usage when reading a response from an S3 HTTP client. #11561 (Pavel Kovalenko).
- Adjust the default Kafka settings for better performance. #11388 (filimonov).
Experimental Feature
- Add data type
Point
(Tuple(Float64, Float64)) andPolygon
(Array(Array(Tuple(Float64, Float64))). #10678 (Alexey Ilyukhov). - Add's a
hasSubstr
function that allows for look for subsequences in arrays. Note: this function is likely to be renamed without further notice. #11071 (Ryad Zenine). - Added OpenCL support and bitonic sort algorithm, which can be used for sorting integer types of data in single column. Needs to be build with flag
-DENABLE_OPENCL=1
. For using bitonic sort algorithm instead of others you need to setbitonic_sort
for Setting's optionspecial_sort
and make sure that OpenCL is available. This feature does not improve performance or anything else, it is only provided as an example and for demonstration purposes. It is likely to be removed in near future if there will be no further development in this direction. #10232 (Ri).
Build/Testing/Packaging Improvement
- Enable clang-tidy for programs and utils. #10991 (alexey-milovidov).
- Remove dependency on
tzdata
: do not fail if/usr/share/zoneinfo
directory does not exist. Note that all timezones work in ClickHouse even without tzdata installed in system. #11827 (alexey-milovidov). - Added MSan and UBSan stress tests. Note that we already have MSan, UBSan for functional tests and "stress" test is another kind of tests. #10871 (alexey-milovidov).
- Print compiler build id in crash messages. It will make us slightly more certain about what binary has crashed. Added new function
buildId
. #11824 (alexey-milovidov). - Added a test to ensure that mutations continue to work after FREEZE query. #11820 (alexey-milovidov).
- Don't allow tests with "fail" substring in their names because it makes looking at the tests results in browser less convenient when you type Ctrl+F and search for "fail". #11817 (alexey-milovidov).
- Removes unused imports from HTTPHandlerFactory. #11660 (Bharat Nallan).
- Added a random sampling of instances where copier is executed. It is needed to avoid
Too many simultaneous queries
error. Also increased timeout and decreased fault probability. #11573 (Nikita Mikhaylov). - Fix missed include. #11525 (Matwey V. Kornilov).
- Speed up build by removing old example programs. Also found some orphan functional test. #11486 (alexey-milovidov).
- Increase ccache size for builds in CI. #11450 (alesapin).
- Leave only unit_tests_dbms in deb build. #11429 (Ilya Yatsishin).
- Update librdkafka to version 1.4.2. #11256 (filimonov).
- Refactor CMake build files. #11390 (Ivan).
- Fix several flaky integration tests. #11355 (alesapin).
- Add support for unit tests run with UBSan. #11345 (alexey-milovidov).
- Remove redundant timeout from integration test
test_insertion_sync_fails_with_timeout
. #11343 (alesapin). - Better check for hung queries in clickhouse-test. #11321 (alexey-milovidov).
- Emit a warning if server was build in debug or with sanitizers. #11304 (alexey-milovidov).
- Now clickhouse-test check the server aliveness before tests run. #11285 (alesapin).
- Fix potentially flacky test
00731_long_merge_tree_select_opened_files.sh
. It does not fail frequently but we have discovered potential race condition in this test while experimenting with ThreadFuzzer: #9814 See link for the example. #11270 (alexey-milovidov). - Repeat test in CI if
curl
invocation was timed out. It is possible due to system hangups for 10+ seconds that are typical in our CI infrastructure. This fixes #11267. #11268 (alexey-milovidov). - Add a test for Join table engine from @donmikel. This closes #9158. #11265 (alexey-milovidov).
- Fix several non significant errors in unit tests. #11262 (alesapin).
- Now parts of linker command for
cctz
library will not be shuffled with other libraries. #11213 (alesapin). - Split /programs/server into actual program and library. #11186 (Ivan).
- Improve build scripts for protobuf & gRPC. #11172 (Vitaly Baranov).
- Enable performance test that was not working. #11158 (alexey-milovidov).
- Create root S3 bucket for tests before any CH instance is started. #11142 (Pavel Kovalenko).
- Add performance test for non-constant polygons. #11141 (alexey-milovidov).
- Fixing
00979_live_view_watch_continuous_aggregates
test. #11024 (vzakaznikov). - Add ability to run zookeeper in integration tests over tmpfs. #11002 (alesapin).
- Wait for odbc-bridge with exponential backoff. Previous wait time of 200 ms was not enough in our CI environment. #10990 (alexey-milovidov).
- Fix non-deterministic test. #10989 (alexey-milovidov).
- Added a test for empty external data. #10926 (alexey-milovidov).
- Database is recreated for every test. This improves separation of tests. #10902 (alexey-milovidov).
- Added more asserts in columns code. #10833 (alexey-milovidov).
- Better cooperation with sanitizers. Print information about query_id in the message of sanitizer failure. #10832 (alexey-milovidov).
- Fix obvious race condition in "Split build smoke test" check. #10820 (alexey-milovidov).
- Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in #9968. #10801 (alexey-milovidov).
- Add MSan suppression for MariaDB Client library. #10800 (alexey-milovidov).
- GRPC make couldn't find protobuf files, changed make file by adding the right link. #10794 (mnkonkova).
- Enable extra warnings (
-Weverything
) for base, utils, programs. Note that we already have it for the most of the code. #10779 (alexey-milovidov). - Suppressions of warnings from libraries was mistakenly declared as public in #10396. #10776 (alexey-milovidov).
- Restore a patch that was accidentially deleted in #10396. #10774 (alexey-milovidov).
- Fix performance tests errors, part 2. #10773 (alexey-milovidov).
- Fix performance test errors. #10766 (alexey-milovidov).
- Update cross-builds to use clang-10 compiler. #10724 (Ivan).
- Update instruction to install RPM packages. This was suggested by Denis (TG login @ldviolet) and implemented by Arkady Shejn. #10707 (alexey-milovidov).
- Trying to fix
tests/queries/0_stateless/01246_insert_into_watch_live_view.py
test. #10670 (vzakaznikov). - Fixing and re-enabling 00979_live_view_watch_continuous_aggregates.py test. #10658 (vzakaznikov).
- Fix OOM in ASan stress test. #10646 (alexey-milovidov).
- Fix UBSan report (adding zero to nullptr) in HashTable that appeared after migration to clang-10. #10638 (alexey-milovidov).
- Remove external call to
ld
(bfd) linker during tzdata processing in compile time. #10634 (alesapin). - Allow to use
lld
to link blobs (resources). #10632 (alexey-milovidov). - Fix UBSan report in
LZ4
library. #10631 (alexey-milovidov). See also https://github.com/lz4/lz4/issues/857 - Update LZ4 to the latest dev branch. #10630 (alexey-milovidov).
- Added auto-generated machine-readable file with the list of stable versions. #10628 (alexey-milovidov).
- Fix
capnproto
version check forcapnp::UnalignedFlatArrayMessageReader
. #10618 (Matwey V. Kornilov). - Lower memory usage in tests. #10617 (alexey-milovidov).
- Fixing hard coded timeouts in new live view tests. #10604 (vzakaznikov).
- Increasing timeout when opening a client in tests/queries/0_stateless/helpers/client.py. #10599 (vzakaznikov).
- Enable ThinLTO for clang builds, continuation of #10435. #10585 (Amos Bird).
- Adding fuzzers and preparing for oss-fuzz integration. #10546 (kyprizel).
- Fix FreeBSD build. #10150 (Ivan).
- Add new build for query tests using pytest framework. #10039 (Ivan).
ClickHouse release v20.4
ClickHouse release v20.4.8.99-stable 2020-08-10
Bug Fix
- Fixed error in
parseDateTimeBestEffort
function when unix timestamp was passed as an argument. This fixes #13362. #13441 (alexey-milovidov). - Fixed potentially low performance and slightly incorrect result for
uniqExact
,topK
,sumDistinct
and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes #12491. #13254 (alexey-milovidov). - Fixed function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. #13226 (alexey-milovidov).
- Fixed assert in
arrayElement
function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. #13224 (alexey-milovidov). - Fixed wrong index analysis with functions. It could lead to pruning wrong parts, while reading from
MergeTree
tables. Fixes #13060. Fixes #12406. #13081 (Anton Popov). - Fixed unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
- Fixed possible extra overflow row in data which could appear for queries
WITH TOTALS
. #12747 (Nikolai Kochetov). - Fixed performance with large tuples, which are interpreted as functions in
IN
section. The case when user writeWHERE x IN tuple(1, 2, ...)
instead ofWHERE x IN (1, 2, ...)
for some obscure reason. #12700 (Anton Popov). - Fixed memory tracking for
input_format_parallel_parsing
(by attaching thread to group). #12672 (Azat Khuzhin). - Fixed #12293 allow push predicate when subquery contains with clause. #12663 (Winter Zhang).
- Fixed #10572 fix bloom filter index with const expression. #12659 (Winter Zhang).
- Fixed
SIGSEGV
inStorageKafka
when broker is unavailable (and not only). #12658 (Azat Khuzhin). - Added support for function
if
withArray(UUID)
arguments. This fixes #11066. #12648 (alexey-milovidov). - Fixed race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
- Removed data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 (Azat Khuzhin).
- Fixed bug which lead to broken old parts after
ALTER DELETE
query whenenable_mixed_granularity_parts=1
. Fixes #12536. #12543 (alesapin). - Better exception for function
in
with invalid number of arguments. #12529 (Anton Popov). - Fixed performance issue, while reading from compact parts. #12492 (Anton Popov).
- Fixed crash in JOIN with dictionary when we are joining over expression of dictionary key:
t JOIN dict ON expr(dict.id) = t.id
. Disable dictionary join optimisation for this case. #12458 (Artem Zuikov). - Fixed possible segfault if StorageMerge. Closes #12054. #12401 (tavplubix).
- Fixed order of columns in
WITH FILL
modifier. Previously order of columns ofORDER BY
statement wasn't respected. #12306 (Anton Popov). - Avoid "bad cast" exception when there is an expression that filters data by virtual columns (like
_table
inMerge
tables) or by "index" columns in system tables such as filtering by database name when querying fromsystem.tables
, and this expression returnsNullable
type. This fixes #12166. #12305 (alexey-milovidov). - Show error after TrieDictionary failed to load. #12290 (Vitaly Baranov).
- The function
arrayFill
worked incorrectly for empty arrays that may lead to crash. This fixes #12263. #12279 (alexey-milovidov). - Implemented conversions to the common type for
LowCardinality
types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes #8212. This fixes #4342. #12275 (alexey-milovidov). - Fixed the behaviour when during multiple sequential inserts in
StorageFile
header for some special types was written more than once. This fixed #6155. #12197 (Nikita Mikhaylov). - Fixed logical functions for UInt8 values when they are not equal to 0 or 1. #12196 (Alexander Kazakov).
- Cap max_memory_usage* limits to the process resident memory. #12182 (Azat Khuzhin).
- Fixed
dictGet
arguments check during GROUP BY injective functions elimination. #12179 (Azat Khuzhin). - Don't split the dictionary source's table name into schema and table name itself if ODBC connection does not support schema. #12165 (Vitaly Baranov).
- Fixed wrong logic in
ALTER DELETE
that leads to deleting of records when condition evaluates to NULL. This fixes #9088. This closes #12106. #12153 (alexey-milovidov). - Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. #12151 (alexey-milovidov).
- Fixed potential overflow in integer division. This fixes #12119. #12140 (alexey-milovidov).
- Fixed potential infinite loop in
greatCircleDistance
,geoDistance
. This fixes #12117. #12137 (alexey-milovidov). - Normalize "pid" file handling. In previous versions the server may refuse to start if it was killed without proper shutdown and if there is another process that has the same pid as previously runned server. Also pid file may be removed in unsuccessful server startup even if there is another server running. This fixes #3501. #12133 (alexey-milovidov).
- Fixed handling dependency of table with ENGINE=Dictionary on dictionary. This fixes #10994. This fixes #10397. #12116 (Vitaly Baranov).
- Fixed performance for selects with
UNION
caused by wrong limit for the total number of threads. Fixes #12030. #12103 (Nikolai Kochetov). - Fixed segfault with
-StateResample
combinators. #12092 (Anton Popov). - Fixed empty
result_rows
andresult_bytes
metrics insystem.quey_log
for selects. Fixes #11595. #12089 (Nikolai Kochetov). - Fixed unnecessary limiting the number of threads for selects from
VIEW
. Fixes #11937. #12085 (Nikolai Kochetov). - Fixed possible crash while using wrong type for
PREWHERE
. Fixes #12053, #12060. #12060 (Nikolai Kochetov). - Fixed error
Expected single dictionary argument for function
for functiondefaultValueOfArgumentType
withLowCardinality
type. Fixes #11808. #12056 (Nikolai Kochetov). - Fixed error
Cannot capture column
for higher-order functions withTuple(LowCardinality)
argument. Fixes #9766. #12055 (Nikolai Kochetov). - Parse tables metadata in parallel when loading database. This fixes slow server startup when there are large number of tables. #12045 (tavplubix).
- Make
topK
aggregate function return Enum for Enum types. This fixes #3740. #12043 (alexey-milovidov). - Fixed constraints check if constraint is a constant expression. This fixes #11360. #12042 (alexey-milovidov).
- Fixed incorrect comparison of tuples with
Nullable
columns. Fixes #11985. #12039 (Nikolai Kochetov). - Fixed calculation of access rights when allow_introspection_functions=0. #12031 (Vitaly Baranov).
- Fixed wrong result and potential crash when invoking function
if
with arguments of typeFixedString
with different sizes. This fixes #11362. #12021 (alexey-milovidov). - A query with function
neighbor
as the only returned expression may return empty result if the function is called with offset-9223372036854775808
. This fixes #11367. #12019 (alexey-milovidov). - Fixed calculation of access rights when allow_ddl=0. #12015 (Vitaly Baranov).
- Fixed potential array size overflow in generateRandom that may lead to crash. This fixes #11371. #12013 (alexey-milovidov).
- Fixed potential floating point exception. This closes #11378. #12005 (alexey-milovidov).
- Fixed wrong setting name in log message at server startup. #11997 (alexey-milovidov).
- Fixed
Query parameter was not set
inValues
format. Fixes #11918. #11936 (tavplubix). - Keep aliases for substitutions in query (parametrized queries). This fixes #11914. #11916 (alexey-milovidov).
- Fixed bug with no moves when changing storage policy from default one. #11893 (Vladimir Chebotarev).
- Fixed potential floating point exception when parsing
DateTime64
. This fixes #11374. #11875 (alexey-milovidov). - Fixed memory accounting via HTTP interface (can be significant with
wait_end_of_query=1
). #11840 (Azat Khuzhin). - Parse metadata stored in zookeeper before checking for equality. #11739 (Azat Khuzhin).
Performance Improvement
- Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes #10574. #12062 (nvartolomei).
Build/Testing/Packaging Improvement
- Install
ca-certificates
before the firstapt-get update
in Dockerfile. #12095 (Ivan Blinkov).
ClickHouse release v20.4.6.53-stable 2020-06-25
Bug Fix
- Fix rare crash caused by using
Nullable
column in prewhere condition. Continuation of #11608. #11869 (Nikolai Kochetov). - Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (alexey-milovidov).
- Fix wrong result of comparison of FixedString with constant String. This fixes #11393. This bug appeared in version 20.4. #11828 (alexey-milovidov).
- Fix wrong result for
if()
with NULLs in condition. #11807 (Artem Zuikov). - Fix using too many threads for queries. #11788 (Nikolai Kochetov).
- Fix unexpected behaviour of queries like
SELECT *, xyz.*
which were success while an error expected. #11753 (hexiaoting). - Now replicated fetches will be cancelled during metadata alter. #11744 (alesapin).
- Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (tavplubix).
- Fix
ORDER BY ... WITH FILL
over const columns. #11697 (Anton Popov). - Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (alexey-milovidov).
- Fix
LIMIT n WITH TIES
usage together withORDER BY
statement, which contains aliases. #11689 (Anton Popov). - Fix error which leads to an incorrect state of
system.mutations
. It may show that whole mutation is already done but the server still hasMUTATE_PART
tasks in the replication queue and tries to execute them. This fixes #11611. #11681 (alesapin). - Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (alexey-milovidov).
- Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. #11644 (alexey-milovidov).
- Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
- Fix rare crash caused by using
Nullable
column in prewhere condition. (Probably it is connected with #11572 somehow). #11608 (Nikolai Kochetov). - Fix error
Block structure mismatch
for queries with sampling reading fromBuffer
table. #11602 (Nikolai Kochetov). - Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. #11601 (filimonov).
- Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (alexey-milovidov).
- Fix error
Size of offsets does not match size of column
for queries withPREWHERE column in (subquery)
andARRAY JOIN
. #11580 (Nikolai Kochetov). - Fixed rare segfault in
SHOW CREATE TABLE
Fixes #11490. #11579 (tavplubix). - All queries in HTTP session have had the same query_id. It is fixed. #11578 (tavplubix).
- Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
- Fix shard_num/replica_num for
<node>
(breaks use_compact_format_in_distributed_parts_names). #11528 (Azat Khuzhin). - Fix race condition which may lead to an exception during table drop. It's a bit tricky and not dangerous at all. If you want an explanation, just notice me in telegram. #11523 (alesapin).
- Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. #11496 (alexey-milovidov).
- If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
- Get rid of old libunwind patches. https://github.com/ClickHouse-Extras/libunwind/commit/500aa227911bd185a94bfc071d68f4d3b03cb3b1#r39048012 This allows to disable
-fno-omit-frame-pointer
inclang
builds that improves performance at least by 1% in average. #10761 (Amos Bird). - Fix usage of primary key wrapped into a function with 'FINAL' modifier and 'ORDER BY' optimization. #10715 (Anton Popov).
Build/Testing/Packaging Improvement
- Fix several non significant errors in unit tests. #11262 (alesapin).
- Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in #9968. #10801 (alexey-milovidov).
ClickHouse release v20.4.5.36-stable 2020-06-10
Bug Fix
- Fix the error
Data compressed with different methods
that can happen ifmin_bytes_to_use_direct_io
is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (alexey-milovidov). - Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
- Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
- Fix pointInPolygon with nan as point. Fixes #11375. #11421 (Alexey Ilyukhov).
- Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. #11420 (alexey-milovidov).
- Fixed geohashesInBox with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov).
- Fix possible
Pipeline stuck
error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov). - Remove redundant lock during parts send in ReplicatedMergeTree. #11354 (alesapin).
- Fix support for
\G
(vertical output) in clickhouse-client in multiline mode. This closes #9933. #11350 (alexey-milovidov). - Fix potential segfault when using
Lazy
database. #11348 (alexey-milovidov). - Fix crash in
quantilesExactWeightedArray
. #11337 (Nikolai Kochetov). - Now merges stopped before change metadata in
ALTER
queries. #11335 (alesapin). - Make writing to
MATERIALIZED VIEW
with settingparallel_view_processing = 1
parallel again. Fixes #10241. #11330 (Nikolai Kochetov). - Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 (Ewout).
- Fix very rare race condition in ThreadPool. #11314 (alexey-milovidov).
- Fix insignificant data race in clickhouse-copier. Found by integration tests. #11313 (alexey-milovidov).
- Fix potential uninitialized memory in conversion. Example:
SELECT toIntervalSecond(now64())
. #11311 (alexey-milovidov). - Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with
empty
ornotEmpty
functions. This fixes #11286. #11303 (alexey-milovidov). - Fix bug when query speed estimation can be incorrect and the limit of
min_execution_speed
may not work or work incorrectly if the query is throttled bymax_network_bandwidth
,max_execution_speed
orpriority
settings. Change the default value oftimeout_before_checking_execution_speed
to non-zero, because otherwise the settingsmin_execution_speed
andmax_execution_speed
have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar inclickhouse-client
. #11296 (alexey-milovidov). - Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes #10586. #11278 (Vitaly Baranov).
- Fix crash while reading malformed data in Protobuf format. This fixes #5957, fixes #11203. #11258 (Vitaly Baranov).
- Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 (Nikita Mikhaylov).
- Fix error
Block structure mismatch in QueryPipeline
while reading fromVIEW
with constants in inner query. Fixes #11181. #11205 (Nikolai Kochetov). - Fix possible exception
Invalid status for associated output
. #11200 (Nikolai Kochetov). - Fix possible error
Cannot capture column
for higher-order functions withArray(Array(LowCardinality))
captured argument. #11185 (Nikolai Kochetov). - Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 (Vladimir Chebotarev).
- If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
- Fix Kafka performance issue related to reschedules based on limits, which were always applied. #11149 (filimonov).
- Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #11145 (filimonov).
- Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 (Azat Khuzhin).
- Fix predicates optimization for distributed queries (
enable_optimize_predicate_expression=1
) for queries withHAVING
section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin).
Build/Testing/Packaging Improvement
ClickHouse release v20.4.4.18-stable 2020-05-26
No changes compared to v20.4.3.16-stable.
ClickHouse release v20.4.3.16-stable 2020-05-23
Bug Fix
- Removed logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
- Fixed memory leak in registerDiskS3. #11074 (Pavel Kovalenko).
- Fixed the potential missed data during termination of Kafka engine table. #11048 (filimonov).
- Fixed
parseDateTime64BestEffort
argument resolution bugs. #11038 (Vasily Nemkov). - Fixed very rare potential use-after-free error in
MergeTree
if table was not created successfully. #10986, #10970 (alexey-milovidov). - Fixed metadata (relative path for rename) and data (relative path for symlink) handling for Atomic database. #10980 (Azat Khuzhin).
- Fixed server crash on concurrent
ALTER
andDROP DATABASE
queries withAtomic
database engine. #10968 (tavplubix). - Fixed incorrect raw data size in
getRawData()
method. #10964 (Igr). - Fixed incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 (alexey-milovidov).
- Fixed sending partially written files by the
DistributedBlockOutputStream
. #10940 (Azat Khuzhin). - Fixed crash in
SELECT count(notNullIn(NULL, []))
. #10920 (Nikolai Kochetov). - Fixed the hang which was happening sometimes during
DROP
ofKafka
table engine. (or during server restarts). #10910 (filimonov). - Fixed the impossibility of executing multiple
ALTER RENAME
likea TO b, c TO a
. #10895 (alesapin). - Fixed possible race which could happen when you get result from aggregate function state from multiple thread for the same column. The only way it can happen is when you use
finalizeAggregation
function while reading from table withMemory
engine which storesAggregateFunction
state forquantile*
function. #10890 (Nikolai Kochetov). - Fixed backward compatibility with tuples in Distributed tables. #10889 (Anton Popov).
- Fixed
SIGSEGV
inStringHashTable
if such a key does not exist. #10870 (Azat Khuzhin). - Fixed
WATCH
hangs afterLiveView
table was dropped from database withAtomic
engine. #10859 (tavplubix). - Fixed bug in
ReplicatedMergeTree
which might cause someALTER
onOPTIMIZE
query to hang waiting for some replica after it become inactive. #10849 (tavplubix). - Now constraints are updated if the column participating in
CONSTRAINT
expression was renamed. Fixes #10844. #10847 (alesapin). - Fixed potential read of uninitialized memory in cache-dictionary. #10834 (alexey-milovidov).
- Fixed columns order after
Block::sortColumns()
. #10826 (Azat Khuzhin). - Fixed the issue with
ODBC
bridge when no quoting of identifiers is requested. Fixes #7984. #10821 (alexey-milovidov). - Fixed
UBSan
andMSan
report inDateLUT
. #10798 (alexey-milovidov). - Fixed incorrect type conversion in key conditions. Fixes #6287. #10791 (Andrew Onyshchuk).
- Fixed
parallel_view_processing
behavior. Now all insertions intoMATERIALIZED VIEW
without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov). - Fixed combinator
-OrNull
and-OrDefault
when combined with-State
. #10741 (hcz). - Fixed possible buffer overflow in function
h3EdgeAngle
. #10711 (alexey-milovidov). - Fixed bug which locks concurrent alters when table has a lot of parts. #10659 (alesapin).
- Fixed
nullptr
dereference inStorageBuffer
if server was shutdown before table startup. #10641 (alexey-milovidov). - Fixed
optimize_skip_unused_shards
withLowCardinality
. #10611 (Azat Khuzhin). - Fixed handling condition variable for synchronous mutations. In some cases signals to that condition variable could be lost. #10588 (Vladimir Chebotarev).
- Fixed possible crash when
createDictionary()
is called beforeloadStoredObject()
has finished. #10587 (Vitaly Baranov). - Fixed
SELECT
of columnALIAS
which default expression type different from column type. #10563 (Azat Khuzhin). - Implemented comparison between DateTime64 and String values. #10560 (Vasily Nemkov).
- Disable
GROUP BY
sharding_key optimization by default (optimize_distributed_group_by_sharding_key
had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example isif
in sharding key) and fix it forWITH ROLLUP/CUBE/TOTALS
. #10516 (Azat Khuzhin). - Fixed #10263. #10486 (Azat Khuzhin).
- Added tests about
max_rows_to_sort
setting. #10268 (alexey-milovidov). - Added backward compatibility for create bloom filter index. #10551. #10569 (Winter Zhang).
ClickHouse release v20.4.2.9, 2020-05-12
Backward Incompatible Change
- System tables (e.g. system.query_log, system.trace_log, system.metric_log) are using compact data part format for parts smaller than 10 MiB in size. Compact data part format is supported since version 20.3. If you are going to downgrade to version less than 20.3, you should manually delete table data for system logs in
/var/lib/clickhouse/data/system/
. - When string comparison involves FixedString and compared arguments are of different sizes, do comparison as if smaller string is padded to the length of the larger. This is intented for SQL compatibility if we imagine that FixedString data type corresponds to SQL CHAR. This closes #9272. #10363 (alexey-milovidov)
- Make SHOW CREATE TABLE multiline. Now it is more readable and more like MySQL. #10049 (Azat Khuzhin)
- Added a setting
validate_polygons
that is used inpointInPolygon
function and enabled by default. #9857 (alexey-milovidov)
New Feature
- Add support for secured connection from ClickHouse to Zookeeper #10184 (Konstantin Lebedev)
- Support custom HTTP handlers. See #5436 for description. #7572 (Winter Zhang)
- Add MessagePack Input/Output format. #9889 (Kruglov Pavel)
- Add Regexp input format. #9196 (Kruglov Pavel)
- Added output format
Markdown
for embedding tables in markdown documents. #10317 (Kruglov Pavel) - Added support for custom settings section in dictionaries. Also fixes issue #2829. #10137 (Artem Streltsov)
- Added custom settings support in DDL-queries for
CREATE DICTIONARY
#10465 (Artem Streltsov) - Add simple server-wide memory profiler that will collect allocation contexts when server memory usage becomes higher than the next allocation threshold. #10444 (alexey-milovidov)
- Add setting
always_fetch_merged_part
which restrict replica to merge parts by itself and always prefer dowloading from other replicas. #10379 (alesapin) - Add function
JSONExtractKeysAndValuesRaw
which extracts raw data from JSON objects #10378 (hcz) - Add memory usage from OS to
system.asynchronous_metrics
. #10361 (alexey-milovidov) - Added generic variants for functions
least
andgreatest
. Now they work with arbitrary number of arguments of arbitrary types. This fixes #4767 #10318 (alexey-milovidov) - Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration:
strict_max_lifetime_seconds
, which ismax_lifetime
by default, andquery_wait_timeout_milliseconds
, which is one minute by default. The first settings is also useful withallow_read_expired_keys
settings (to forbid reading very expired keys). #10337 (Nikita Mikhaylov) - Add log_queries_min_type to filter which entries will be written to query_log #10053 (Azat Khuzhin)
- Added function
isConstant
. This function checks whether its argument is constant expression and returns 1 or 0. It is intended for development, debugging and demonstration purposes. #10198 (alexey-milovidov) - add joinGetOrNull to return NULL when key is missing instead of returning the default value. #10094 (Amos Bird)
- Consider
NULL
to be equal toNULL
inIN
operator, if the optiontransform_null_in
is set. #10085 (achimbab) - Add
ALTER TABLE ... RENAME COLUMN
for MergeTree table engines family. #9948 (alesapin) - Support parallel distributed INSERT SELECT. #9759 (vxider)
- Add ability to query Distributed over Distributed (w/o
distributed_group_by_no_merge
) ... #9923 (Azat Khuzhin) - Add function
arrayReduceInRanges
which aggregates array elements in given ranges. #9598 (hcz) - Add Dictionary Status on prometheus exporter. #9622 (Guillaume Tassery)
- Add function
arrayAUC
#8698 (taiyang-li) - Support
DROP VIEW
statement for better TPC-H compatibility. #9831 (Amos Bird) - Add 'strict_order' option to windowFunnel() #9773 (achimbab)
- Support
DATE
andTIMESTAMP
SQL operators, e.g.SELECT date '2001-01-01'
#9691 (Artem Zuikov)
Experimental Feature
- Added experimental database engine Atomic. It supports non-blocking
DROP
andRENAME TABLE
queries and atomicEXCHANGE TABLES t1 AND t2
query #7512 (tavplubix) - Initial support for ReplicatedMergeTree over S3 (it works in suboptimal way) #10126 (Pavel Kovalenko)
Bug Fix
- Fixed incorrect scalar results inside inner query of
MATERIALIZED VIEW
in case if this query contained dependent table #10603 (Nikolai Kochetov) - Fixed bug, which caused HTTP requests to get stuck on client closing connection when
readonly=2
andcancel_http_readonly_queries_on_client_close=1
. #10684 (tavplubix) - Fix segfault in StorageBuffer when exception is thrown on server startup. Fixes #10550 #10609 (tavplubix)
- The query
SYSTEM DROP DNS CACHE
now also drops caches used to check if user is allowed to connect from some IP addresses #10608 (tavplubix) - Fix usage of multiple
IN
operators with an identical set in one query. Fixes #10539 #10686 (Anton Popov) - Fix crash in
generateRandom
with nested types. Fixes #10583. #10734 (Nikolai Kochetov) - Fix data corruption for
LowCardinality(FixedString)
key column inSummingMergeTree
which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov) - Fix logic for aggregation_memory_efficient_merge_threads setting. #10667 (palasonic1)
- Fix disappearing totals. Totals could have being filtered if query had
JOIN
or subquery with externalWHERE
condition. Fixes #10674 #10698 (Nikolai Kochetov) - Fix the lack of parallel execution of remote queries with
distributed_aggregation_memory_efficient
enabled. Fixes #10655 #10664 (Nikolai Kochetov) - Fix possible incorrect number of rows for queries with
LIMIT
. Fixes #10566, #10709 #10660 (Nikolai Kochetov) - Fix index corruption, which may occur in some cases after merging compact parts into another compact part. #10531 (Anton Popov)
- Fix the situation, when mutation finished all parts, but hung up in
is_done=0
. #10526 (alesapin) - Fix overflow at beginning of unix epoch for timezones with fractional offset from UTC. Fixes #9335. #10513 (alexey-milovidov)
- Better diagnostics for input formats. Fixes #10204 #10418 (tavplubix)
- Fix numeric overflow in
simpleLinearRegression()
over large integers #10474 (hcz) - Fix use-after-free in Distributed shutdown, avoid waiting for sending all batches #10491 (Azat Khuzhin)
- Add CA certificates to clickhouse-server docker image #10476 (filimonov)
- Fix a rare endless loop that might have occurred when using the
addressToLine
function or AggregateFunctionState columns. #10466 (Alexander Kuzmenkov) - Handle zookeeper "no node error" during distributed query #10050 (Daniel Chen)
- Fix bug when server cannot attach table after column's default was altered. #10441 (alesapin)
- Implicitly cast the default expression type to the column type for the ALIAS columns #10563 (Azat Khuzhin)
- Don't remove metadata directory if
ATTACH DATABASE
fails #10442 (Winter Zhang) - Avoid dependency on system tzdata. Fixes loading of
Africa/Casablanca
timezone on CentOS 8. Fixes #10211 #10425 (alexey-milovidov) - Fix some issues if data is inserted with quorum and then gets deleted (DROP PARTITION, TTL, etc.). It led to stuck of INSERTs or false-positive exceptions in SELECTs. Fixes #9946 #10188 (Nikita Mikhaylov)
- Check the number and type of arguments when creating BloomFilter index #9623 #10431 (Winter Zhang)
- Prefer
fallback_to_stale_replicas
overskip_unavailable_shards
, otherwise when both settings specified and there are no up-to-date replicas the query will fail (patch from @alex-zaitsev ) #10422 (Azat Khuzhin) - Fix the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. Fixes #10226. #10427 (Vadim Plakhtinskiy)
- Add database name to dictionary name after DETACH/ATTACH. Fixes system.dictionaries table and
SYSTEM RELOAD
query #10415 (Azat Khuzhin) - Fix possible incorrect result for extremes in processors pipeline. #10131 (Nikolai Kochetov)
- Fix possible segfault when the setting
distributed_group_by_no_merge
is enabled (introduced in 20.3.7.46 by #10131). #10399 (Nikolai Kochetov) - Fix wrong flattening of
Array(Tuple(...))
data types. Fixes #10259 #10390 (alexey-milovidov) - Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN #9950 (Alexander Kuzmenkov)
- Fix order of columns after Block::sortColumns() #10826 (Azat Khuzhin)
- Fix possible
Pipeline stuck
error inConcatProcessor
which may happen in remote query. #10381 (Nikolai Kochetov) - Don't make disk reservations for aggregations. Fixes #9241 #10375 (Azat Khuzhin)
- Fix wrong behaviour of datetime functions for timezones that has altered between positive and negative offsets from UTC (e.g. Pacific/Kiritimati). Fixes #7202 #10369 (alexey-milovidov)
- Avoid infinite loop in
dictIsIn
function. Fixes #515 #10365 (alexey-milovidov) - Disable GROUP BY sharding_key optimization by default and fix it for WITH ROLLUP/CUBE/TOTALS #10516 (Azat Khuzhin)
- Check for error code when checking parts and don't mark part as broken if the error is like "not enough memory". Fixes #6269 #10364 (alexey-milovidov)
- Show information about not loaded dictionaries in system tables. #10234 (Vitaly Baranov)
- Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 (alexey-milovidov)
- Fixed
DROP
vsOPTIMIZE
race inReplicatedMergeTree
.DROP
could left some garbage in replica path in ZooKeeper if there was concurrentOPTIMIZE
query. #10312 (tavplubix) - Fix 'Logical error: CROSS JOIN has expressions' error for queries with comma and names joins mix. Fixes #9910 #10311 (Artem Zuikov)
- Fix queries with
max_bytes_before_external_group_by
. #10302 (Artem Zuikov) - Fix the issue with limiting maximum recursion depth in parser in certain cases. This fixes #10283 This fix may introduce minor incompatibility: long and deep queries via clickhouse-client may refuse to work, and you should adjust settings
max_query_size
andmax_parser_depth
accordingly. #10295 (alexey-milovidov) - Allow to use
count(*)
with multiple JOINs. Fixes #9853 #10291 (Artem Zuikov) - Fix error
Pipeline stuck
withmax_rows_to_group_by
andgroup_by_overflow_mode = 'break'
. #10279 (Nikolai Kochetov) - Fix 'Cannot add column' error while creating
range_hashed
dictionary using DDL query. Fixes #10093. #10235 (alesapin) - Fix rare possible exception
Cannot drain connections: cancel first
. #10239 (Nikolai Kochetov) - Fixed bug where ClickHouse would throw "Unknown function lambda." error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. #10237 (Alexander Kazakov)
- Fixed reasonably rare segfault in StorageSystemTables that happens when SELECT ... FROM system.tables is run on a database with Lazy engine. #10209 (Alexander Kazakov)
- Fix possible infinite query execution when the query actually should stop on LIMIT, while reading from infinite source like
system.numbers
orsystem.zeros
. #10206 (Nikolai Kochetov) - Fixed "generateRandom" function for Date type. This fixes #9973. Fix an edge case when dates with year 2106 are inserted to MergeTree tables with old-style partitioning but partitions are named with year 1970. #10218 (alexey-milovidov)
- Convert types if the table definition of a View does not correspond to the SELECT query. This fixes #10180 and #10022 #10217 (alexey-milovidov)
- Fix
parseDateTimeBestEffort
for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10082 #10214 (alexey-milovidov) - Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. #10207 (alexey-milovidov)
- Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes #10092 #10195 (alexey-milovidov)
- Fix issue with separator appearing in SCRAMBLE for native mysql-connector-java (JDBC) #10140 (BohuTANG)
- Fix using the current database for an access checking when the database isn't specified. #10192 (Vitaly Baranov)
- Fix ALTER of tables with compact parts. #10130 (Anton Popov)
- Add the ability to relax the restriction on non-deterministic functions usage in mutations with
allow_nondeterministic_mutations
setting. #10186 (filimonov) - Fix
DROP TABLE
invoked for dictionary #10165 (Azat Khuzhin) - Convert blocks if structure does not match when doing
INSERT
into Distributed table #10135 (Azat Khuzhin) - The number of rows was logged incorrectly (as sum across all parts) when inserted block is split by parts with partition key. #10138 (alexey-milovidov)
- Add some arguments check and support identifier arguments for MySQL Database Engine #10077 (Winter Zhang)
- Fix incorrect
index_granularity_bytes
check while creating new replica. Fixes #10098. #10121 (alesapin) - Fix bug in
CHECK TABLE
query when table contain skip indices. #10068 (alesapin) - Fix Distributed-over-Distributed with the only one shard in a nested table #9997 (Azat Khuzhin)
- Fix possible rows loss for queries with
JOIN
andUNION ALL
. Fixes #9826, #10113. ... #10099 (Nikolai Kochetov) - Fix bug in dictionary when local clickhouse server is used as source. It may caused memory corruption if types in dictionary and source are not compatible. #10071 (alesapin)
- Fixed replicated tables startup when updating from an old ClickHouse version where
/table/replicas/replica_name/metadata
node does not exist. Fixes #10037. #10095 (alesapin) - Fix error
Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform
. It happened when settingdistributed_aggregation_memory_efficient
was enabled, and distributed query read aggregating data with mixed single and two-level aggregation from different shards. #10063 (Nikolai Kochetov) - Fix deadlock when database with materialized view failed attach at start #10054 (Azat Khuzhin)
- Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (#8636, #8925). ... #10025 (Alexander Kuzmenkov)
- Fix wrong results of distributed queries when alias could override qualified column name. Fixes #9672 #9714 #9972 (Artem Zuikov)
- Fix possible deadlock in
SYSTEM RESTART REPLICAS
#9955 (tavplubix) - Fix the number of threads used for remote query execution (performance regression, since 20.3). This happened when query from
Distributed
table was executed simultaneously on local and remote shards. Fixes #9965 #9971 (Nikolai Kochetov) - Fixed
DeleteOnDestroy
logic inATTACH PART
which could lead to automatic removal of attached part and added few tests #9410 (Vladimir Chebotarev) - Fix a bug with
ON CLUSTER
DDL queries freezing on server startup. #9927 (Gagan Arneja) - Fix bug in which the necessary tables weren't retrieved at one of the processing stages of queries to some databases. Fixes #9699. #9949 (achulkov2)
- Fix 'Not found column in block' error when
JOIN
appears withTOTALS
. Fixes #9839 #9939 (Artem Zuikov) - Fix parsing multiple hosts set in the CREATE USER command #9924 (Vitaly Baranov)
- Fix
TRUNCATE
for Join table engine (#9917). #9920 (Amos Bird) - Fix race condition between drop and optimize in
ReplicatedMergeTree
. #9901 (alesapin) - Fix
DISTINCT
for Distributed whenoptimize_skip_unused_shards
is set. #9808 (Azat Khuzhin) - Fix "scalar does not exist" error in ALTERs (#9878). ... #9904 (Amos Bird)
- Fix error with qualified names in
distributed_product_mode=\'local\'
. Fixes #4756 #9891 (Artem Zuikov) - For INSERT queries shards now do clamp the settings from the initiator to their constraints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix #9447. #9852 (Vitaly Baranov)
- Add some retries when commiting offsets to Kafka broker, since it can reject commit if during
offsets.commit.timeout.ms
there were no enough replicas available for the__consumer_offsets
topic #9884 (filimonov) - Fix Distributed engine behavior when virtual columns of the underlying table used in
WHERE
#9847 (Azat Khuzhin) - Fixed some cases when timezone of the function argument wasn't used properly. #9574 (Vasily Nemkov)
- Fix 'Different expressions with the same alias' error when query has PREWHERE and WHERE on distributed table and
SET distributed_product_mode = 'local'
. #9871 (Artem Zuikov) - Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. #9860 (alesapin)
- Fix calculating grants for introspection functions from the setting
allow_introspection_functions
. #9840 (Vitaly Baranov) - Fix max_distributed_connections (w/ and w/o Processors) #9673 (Azat Khuzhin)
- Fix possible exception
Got 0 in totals chunk, expected 1
on client. It happened for queries withJOIN
in case if right joined table had zero rows. Example:select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;
. Fixes #9777. ... #9823 (Nikolai Kochetov) - Fix 'COMMA to CROSS JOIN rewriter is not enabled or cannot rewrite query' error in case of subqueries with COMMA JOIN out of tables lists (i.e. in WHERE). Fixes #9782 #9830 (Artem Zuikov)
- Fix server crashing when
optimize_skip_unused_shards
is set and expression for key can't be converted to its field type #9804 (Azat Khuzhin) - Fix empty string handling in
splitByString
. #9767 (hcz) - Fix broken
ALTER TABLE DELETE COLUMN
query for compact parts. #9779 (alesapin) - Fixed missing
rows_before_limit_at_least
for queries over http (with processors pipeline). Fixes #9730 #9757 (Nikolai Kochetov) - Fix excessive memory consumption in
ALTER
queries (mutations). This fixes #9533 and #9670. #9754 (alesapin) - Fix possible permanent "Cannot schedule a task" error. #9154 (Azat Khuzhin)
- Fix bug in backquoting in external dictionaries DDL. Fixes #9619. #9734 (alesapin)
- Fixed data race in
text_log
. It does not correspond to any real bug. #9726 (alexey-milovidov) - Fix bug in a replication that does not allow replication to work if the user has executed mutations on the previous version. This fixes #9645. #9652 (alesapin)
- Fixed incorrect internal function names for
sumKahan
andsumWithOverflow
. It led to exception while using this functions in remote queries. #9636 (Azat Khuzhin) - Add setting
use_compact_format_in_distributed_parts_names
which allows to write files forINSERT
queries intoDistributed
table with more compact format. This fixes #9647. #9653 (alesapin) - Fix RIGHT and FULL JOIN with LowCardinality in JOIN keys. #9610 (Artem Zuikov)
- Fix possible exceptions
Size of filter does not match size of column
andInvalid number of rows in Chunk
inMergeTreeRangeReader
. They could appear while executingPREWHERE
in some cases. #9612 (Anton Popov) - Allow
ALTER ON CLUSTER
of Distributed tables with internal replication. This fixes #3268 #9617 (shinoi2) - Fix issue when timezone was not preserved if you write a simple arithmetic expression like
time + 1
(in contrast to an expression liketime + INTERVAL 1 SECOND
). This fixes #5743 #9323 (alexey-milovidov)
Improvement
- Use time zone when comparing DateTime with string literal. This fixes #5206. #10515 (alexey-milovidov)
- Print verbose diagnostic info if Decimal value cannot be parsed from text input format. #10205 (alexey-milovidov)
- Add tasks/memory metrics for distributed/buffer schedule pools #10449 (Azat Khuzhin)
- Display result as soon as it's ready for SELECT DISTINCT queries in clickhouse-local and HTTP interface. This fixes #8951 #9559 (alexey-milovidov)
- Allow to use
SAMPLE OFFSET
query instead ofcityHash64(PRIMARY KEY) % N == n
for splitting inclickhouse-copier
. To use this feature, pass--experimental-use-sample-offset 1
as a command line argument. #10414 (Nikita Mikhaylov) - Allow to parse BOM in TSV if the first column cannot contain BOM in its value. This fixes #10301 #10424 (alexey-milovidov)
- Add Avro nested fields insert support #10354 (Andrew Onyshchuk)
- Allowed to alter column in non-modifying data mode when the same type is specified. #10382 (Vladimir Chebotarev)
- Auto
distributed_group_by_no_merge
on GROUP BY sharding key (ifoptimize_skip_unused_shards
is set) #10341 (Azat Khuzhin) - Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key #10373 (Azat Khuzhin)
- Added a setting
max_server_memory_usage
to limit total memory usage of the server. The metricMemoryTracking
is now calculated without a drift. The settingmax_memory_usage_for_all_queries
is now obsolete and does nothing. This closes #10293. #10362 (alexey-milovidov) - Add config option
system_tables_lazy_load
. If it's set to false, then system tables with logs are loaded at the server startup. Alexander Burmak, Svyatoslav Tkhon Il Pak, #9642 #10359 (alexey-milovidov) - Use background thread pool (background_schedule_pool_size) for distributed sends #10263 (Azat Khuzhin)
- Use background thread pool for background buffer flushes. #10315 (Azat Khuzhin)
- Support for one special case of removing incompletely written parts. This fixes #9940. #10221 (alexey-milovidov)
- Use isInjective() over manual list of such functions for GROUP BY optimization. #10342 (Azat Khuzhin)
- Avoid printing error message in log if client sends RST packet immediately on connect. It is typical behaviour of IPVS balancer with keepalived and VRRP. This fixes #1851 #10274 (alexey-milovidov)
- Allow to parse
+inf
for floating point types. This closes #1839 #10272 (alexey-milovidov) - Implemented
generateRandom
table function for Nested types. This closes #9903 #10219 (alexey-milovidov) - Provide
max_allowed_packed
in MySQL compatibility interface that will help some clients to communicate with ClickHouse via MySQL protocol. #10199 (BohuTANG) - Allow literals for GLOBAL IN (i.e.
SELECT * FROM remote('localhost', system.one) WHERE dummy global in (0)
) #10196 (Azat Khuzhin) - Fix various small issues in interactive mode of clickhouse-client #10194 (alexey-milovidov)
- Avoid superfluous dictionaries load (system.tables, DROP/SHOW CREATE TABLE) #10164 (Azat Khuzhin)
- Update to RWLock: timeout parameter for getLock() + implementation reworked to be phase fair #10073 (Alexander Kazakov)
- Enhanced compatibility with native mysql-connector-java(JDBC) #10021 (BohuTANG)
- The function
toString
is considered monotonic and can be used for index analysis even when applied in tautological cases with String or LowCardinality(String) argument. #10110 (Amos Bird) - Add
ON CLUSTER
clause support to commands{CREATE|DROP} USER/ROLE/ROW POLICY/SETTINGS PROFILE/QUOTA
,GRANT
. #9811 (Vitaly Baranov) - Virtual hosted-style support for S3 URI #9998 (Pavel Kovalenko)
- Now layout type for dictionaries with no arguments can be specified without round brackets in dictionaries DDL-queries. Fixes #10057. #10064 (alesapin)
- Add ability to use number ranges with leading zeros in filepath #9989 (Olga Khvostikova)
- Better memory usage in CROSS JOIN. #10029 (Artem Zuikov)
- Try to connect to all shards in cluster when getting structure of remote table and skip_unavailable_shards is set. #7278 (nvartolomei)
- Add
total_rows
/total_bytes
into thesystem.tables
table. #9919 (Azat Khuzhin) - System log tables now use polymorpic parts by default. #9905 (Anton Popov)
- Add type column into system.settings/merge_tree_settings #9909 (Azat Khuzhin)
- Check for available CPU instructions at server startup as early as possible. #9888 (alexey-milovidov)
- Remove
ORDER BY
stage from mutations because we read from a single ordered part in a single thread. Also add check that the rows in mutation are ordered by sorting key and this order is not violated. #9886 (alesapin) - Implement operator LIKE for FixedString at left hand side. This is needed to better support TPC-DS queries. #9890 (alexey-milovidov)
- Add
force_optimize_skip_unused_shards_no_nested
that will disableforce_optimize_skip_unused_shards
for nested Distributed table #9812 (Azat Khuzhin) - Now columns size is calculated only once for MergeTree data parts. #9827 (alesapin)
- Evaluate constant expressions for
optimize_skip_unused_shards
(i.e.SELECT * FROM foo_dist WHERE key=xxHash32(0)
) #8846 (Azat Khuzhin) - Check for using
Date
orDateTime
column from TTL expressions was removed. #9967 (Vladimir Chebotarev) - DiskS3 hard links optimal implementation. #9760 (Pavel Kovalenko)
- If
set multiple_joins_rewriter_version = 2
enables second version of multiple JOIN rewrites that keeps not clashed column names as is. It supports multiple JOINs withUSING
and allowselect *
for JOINs with subqueries. #9739 (Artem Zuikov) - Implementation of "non-blocking" alter for StorageMergeTree #9606 (alesapin)
- Add MergeTree full support for DiskS3 #9646 (Pavel Kovalenko)
- Extend
splitByString
to support empty strings as separators. #9742 (hcz) - Add a
timestamp_ns
column tosystem.trace_log
. It contains a high-definition timestamp of the trace event, and allows to build timelines of thread profiles ("flame charts"). #9696 (Alexander Kuzmenkov) - When the setting
send_logs_level
is enabled, avoid intermixing of log messages and query progress. #9634 (Azat Khuzhin) - Added support of
MATERIALIZE TTL IN PARTITION
. #9581 (Vladimir Chebotarev) - Support complex types inside Avro nested fields #10502 (Andrew Onyshchuk)
Performance Improvement
- Better insert logic for right table for Partial MergeJoin. #10467 (Artem Zuikov)
- Improved performance of row-oriented formats (more than 10% for CSV and more than 35% for Avro in case of narrow tables). #10503 (Andrew Onyshchuk)
- Improved performance of queries with explicitly defined sets at right side of IN operator and tuples on the left side. #10385 (Anton Popov)
- Use less memory for hash table in HashJoin. #10416 (Artem Zuikov)
- Special HashJoin over StorageDictionary. Allow rewrite
dictGet()
functions with JOINs. It's not backward incompatible itself but could uncover #8400 on some installations. #10133 (Artem Zuikov) - Enable parallel insert of materialized view when its target table supports. #10052 (vxider)
- Improved performance of index analysis with monotonic functions. #9607#10026 (Anton Popov)
- Using SSE2 or SSE4.2 SIMD intrinsics to speed up tokenization in bloom filters. #9968 (Vasily Nemkov)
- Improved performance of queries with explicitly defined sets at right side of
IN
operator. This fixes performance regression in version 20.3. #9740 (Anton Popov) - Now clickhouse-copier splits each partition in number of pieces and copies them independently. #9075 (Nikita Mikhaylov)
- Adding more aggregation methods. For example TPC-H query 1 will now pick
FixedHashMap<UInt16, AggregateDataPtr>
and gets 25% performance gain #9829 (Amos Bird) - Use single row counter for multiple streams in pre-limit transform. This helps to avoid uniting pipeline streams in queries with
limit
but withoutorder by
(likeselect f(x) from (select x from t limit 1000000000)
) and use multiple threads for further processing. #9602 (Nikolai Kochetov)
Build/Testing/Packaging Improvement
- Use a fork of AWS SDK libraries from ClickHouse-Extras #10527 (Pavel Kovalenko)
- Add integration tests for new ALTER RENAME COLUMN query. #10654 (vzakaznikov)
- Fix possible signed integer overflow in invocation of function
now64
with wrong arguments. This fixes #8973 #10511 (alexey-milovidov) - Split fuzzer and sanitizer configurations to make build config compatible with Oss-fuzz. #10494 (kyprizel)
- Fixes for clang-tidy on clang-10. #10420 (alexey-milovidov)
- Display absolute paths in error messages. Otherwise KDevelop fails to navigate to correct file and opens a new file instead. #10434 (alexey-milovidov)
- Added
ASAN_OPTIONS
environment variable to investigate errors in CI stress tests with Address sanitizer. #10440 (Nikita Mikhaylov) - Enable ThinLTO for clang builds (experimental). #10435 (alexey-milovidov)
- Remove accidential dependency on Z3 that may be introduced if the system has Z3 solver installed. #10426 (alexey-milovidov)
- Move integration tests docker files to docker/ directory. #10335 (Ilya Yatsishin)
- Allow to use
clang-10
in CI. It ensures that #10238 is fixed. #10384 (alexey-milovidov) - Update OpenSSL to upstream master. Fixed the issue when TLS connections may fail with the message
OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
andSSL Exception: error:2400006E:random number generator::error retrieving entropy
. The issue was present in version 20.1. #8956 (alexey-milovidov) - Fix clang-10 build. #10238 #10370 (Amos Bird)
- Add performance test for Parallel INSERT for materialized view. #10345 (vxider)
- Fix flaky test
test_settings_constraints_distributed.test_insert_clamps_settings
. #10346 (Vitaly Baranov) - Add util to test results upload in CI ClickHouse #10330 (Ilya Yatsishin)
- Convert test results to JSONEachRow format in junit_to_html tool #10323 (Ilya Yatsishin)
- Update cctz. #10215 (alexey-milovidov)
- Allow to create HTML report from the purest JUnit XML report. #10247 (Ilya Yatsishin)
- Update the check for minimal compiler version. Fix the root cause of the issue #10250 #10256 (alexey-milovidov)
- Initial support for live view tables over distributed #10179 (vzakaznikov)
- Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in #9968. #10801 (alexey-milovidov)
- clickhouse-docker-util #10151 (filimonov)
- Update pdqsort to recent version #10171 (Ivan)
- Update libdivide to v3.0 #10169 (Ivan)
- Add check with enabled polymorphic parts. #10086 (Anton Popov)
- Add cross-compile build for FreeBSD. This fixes #9465 #9643 (Ivan)
- Add performance test for #6924 #6980 (filimonov)
- Add support of
/dev/null
in theFile
engine for better performance testing #8455 (Amos Bird) - Move all folders inside /dbms one level up #9974 (Ivan)
- Add a test that checks that read from MergeTree with single thread is performed in order. Addition to #9670 #9762 (alexey-milovidov)
- Fix the
00964_live_view_watch_events_heartbeat.py
test to avoid race condition. #9944 (vzakaznikov) - Fix integration test
test_settings_constraints
#9962 (Vitaly Baranov) - Every function in its own file, part 12. #9922 (alexey-milovidov)
- Added performance test for the case of extremely slow analysis of array of tuples. #9872 (alexey-milovidov)
- Update zstd to 1.4.4. It has some minor improvements in performance and compression ratio. If you run replicas with different versions of ClickHouse you may see reasonable error messages
Data after merge is not byte-identical to data on another replicas.
with explanation. These messages are Ok and you should not worry. #10663 (alexey-milovidov) - Fix TSan report in
system.stack_trace
. #9832 (alexey-milovidov) - Removed dependency on
clock_getres
. #9833 (alexey-milovidov) - Added identifier names check with clang-tidy. #9799 (alexey-milovidov)
- Update "builder" docker image. This image is not used in CI but is useful for developers. #9809 (alexey-milovidov)
- Remove old
performance-test
tool that is no longer used in CI.clickhouse-performance-test
is great but now we are using way superior tool that is doing comparison testing with sophisticated statistical formulas to achieve confident results regardless to various changes in environment. #9796 (alexey-milovidov) - Added most of clang-static-analyzer checks. #9765 (alexey-milovidov)
- Update Poco to 1.9.3 in preparation for MongoDB URI support. #6892 (Alexander Kuzmenkov)
- Fix build with
-DUSE_STATIC_LIBRARIES=0 -DENABLE_JEMALLOC=0
#9651 (Artem Zuikov) - For change log script, if merge commit was cherry-picked to release branch, take PR name from commit description. #9708 (Nikolai Kochetov)
- Support
vX.X-conflicts
tag in backport script. #9705 (Nikolai Kochetov) - Fix
auto-label
for backporting script. #9685 (Nikolai Kochetov) - Use libc++ in Darwin cross-build to make it consistent with native build. #9665 (Hui Wang)
- Fix flacky test
01017_uniqCombined_memory_usage
. Continuation of #7236. #9667 (alexey-milovidov) - Fix build for native macOS Clang compiler #9649 (Ivan)
- Allow to add various glitches around
pthread_mutex_lock
,pthread_mutex_unlock
functions. #9635 (alexey-milovidov) - Add support for
clang-tidy
inpackager
script. #9625 (alexey-milovidov) - Add ability to use unbundled msgpack. #10168 (Azat Khuzhin)
ClickHouse release v20.3
ClickHouse release v20.3.21.2-lts, 2020-11-02
Bug Fix
- Fix dictGet in sharding_key (and similar places, i.e. when the function context is stored permanently). #16205 (Azat Khuzhin).
- Fix incorrect empty result for query from
Distributed
table if query hasWHERE
,PREWHERE
andGLOBAL IN
. Fixes #15792. #15933 (Nikolai Kochetov). - Fix missing or excessive headers in
TSV/CSVWithNames
formats. This fixes #12504. #13343 (Azat Khuzhin).
ClickHouse release v20.3.20.6-lts, 2020-10-09
Bug Fix
- Mutation might hang waiting for some non-existent part after
MOVE
orREPLACE PARTITION
or, in rare cases, afterDETACH
orDROP PARTITION
. It's fixed. #15724, #15537 (tavplubix). - Fix hang of queries with a lot of subqueries to same table of
MySQL
engine. Previously, if there were more than 16 subqueries to sameMySQL
table in query, it hang forever. #15299 (Anton Popov). - Fix 'Unknown identifier' in GROUP BY when query has JOIN over Merge table. #15242 (Artem Zuikov).
- Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes #14847. #14937 (filimonov).
- Concurrent
ALTER ... REPLACE/MOVE PARTITION ...
queries might cause deadlock. It's fixed. #13626 (tavplubix).
ClickHouse release v20.3.19.4-lts, 2020-09-18
Bug Fix
- Fix rare error in
SELECT
queries when the queried column hasDEFAULT
expression which depends on the other column which also hasDEFAULT
and not present in select query and not exists on disk. Partially fixes #14531. #14845 (alesapin). - Fix bug when
ALTER UPDATE
mutation with Nullable column in assignment expression and constant value (likeUPDATE x = 42
) leads to incorrect value in column or segfault. Fixes #13634, #14045. #14646 (alesapin). - Fix wrong Decimal multiplication result caused wrong decimal scale of result column. #14603 (Artem Zuikov).
Improvement
- Support custom codecs in compact parts. #12183 (Anton Popov).
ClickHouse release v20.3.18.10-lts, 2020-09-08
Bug Fix
- Stop query execution if exception happened in
PipelineExecutor
itself. This could prevent rare possible query hung. Continuation of #14334. #14402 (Nikolai Kochetov). - Fixed the behaviour when sometimes cache-dictionary returned default value instead of present value from source. #13624 (Nikita Mikhaylov).
- Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes #5779, #12527. #13199 (Vitaly Baranov).
- Fix CAST(Nullable(String), Enum()). #12745 (Azat Khuzhin).
- Fixed data race in
text_log
. It does not correspond to any real bug. #9726 (alexey-milovidov).
Improvement
- Fix wrong error for long queries. It was possible to get syntax error other than
Max query size exceeded
for correct query. #13928 (Nikolai Kochetov). - Return NULL/zero when value is not parsed completely in parseDateTimeBestEffortOrNull/Zero functions. This fixes #7876. #11653 (alexey-milovidov).
Performance Improvement
- Slightly optimize very short queries with LowCardinality. #14129 (Anton Popov).
Build/Testing/Packaging Improvement
- Fix UBSan report (adding zero to nullptr) in HashTable that appeared after migration to clang-10. #10638 (alexey-milovidov).
ClickHouse release v20.3.17.173-lts, 2020-08-15
Bug Fix
- Fix crash in JOIN with StorageMerge and
set enable_optimize_predicate_expression=1
. #13679 (Artem Zuikov). - Fix invalid return type for comparison of tuples with
NULL
elements. Fixes #12461. #13420 (Nikolai Kochetov). - Fix queries with constant columns and
ORDER BY
prefix of primary key. #13396 (Anton Popov). - Return passed number for numbers with MSB set in roundUpToPowerOfTwoOrZero(). #13234 (Azat Khuzhin).
ClickHouse release v20.3.16.165-lts 2020-08-10
Bug Fix
- Fixed error in
parseDateTimeBestEffort
function when unix timestamp was passed as an argument. This fixes #13362. #13441 (alexey-milovidov). - Fixed potentially low performance and slightly incorrect result for
uniqExact
,topK
,sumDistinct
and similar aggregate functions called on Float types withNaN
values. It also triggered assert in debug build. This fixes #12491. #13254 (alexey-milovidov). - Fixed function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. #13226 (alexey-milovidov).
- Fixed assert in
arrayElement
function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. #13224 (alexey-milovidov). - Fixed unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
- Fixed possible extra overflow row in data which could appear for queries
WITH TOTALS
. #12747 (Nikolai Kochetov). - Fixed performance with large tuples, which are interpreted as functions in
IN
section. The case when user writeWHERE x IN tuple(1, 2, ...)
instead ofWHERE x IN (1, 2, ...)
for some obscure reason. #12700 (Anton Popov). - Fixed memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 (Azat Khuzhin).
- Fixed #12293 allow push predicate when subquery contains with clause. #12663 (Winter Zhang).
- Fixed #10572 fix bloom filter index with const expression. #12659 (Winter Zhang).
- Fixed SIGSEGV in StorageKafka when broker is unavailable (and not only). #12658 (Azat Khuzhin).
- Fixed race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
- Fixed bug which lead to broken old parts after
ALTER DELETE
query whenenable_mixed_granularity_parts=1
. Fixes #12536. #12543 (alesapin). - Better exception for function
in
with invalid number of arguments. #12529 (Anton Popov). - Fixed performance issue, while reading from compact parts. #12492 (Anton Popov).
- Fixed the deadlock if
text_log
is enabled. #12452 (alexey-milovidov). - Fixed possible segfault if StorageMerge. Closes #12054. #12401 (tavplubix).
- Fixed
TOTALS/ROLLUP/CUBE
for aggregate functions with-State
andNullable
arguments. This fixes #12163. #12376 (alexey-milovidov). - Fixed order of columns in
WITH FILL
modifier. Previously order of columns ofORDER BY
statement wasn't respected. #12306 (Anton Popov). - Avoid "bad cast" exception when there is an expression that filters data by virtual columns (like
_table
inMerge
tables) or by "index" columns in system tables such as filtering by database name when querying fromsystem.tables
, and this expression returnsNullable
type. This fixes #12166. #12305 (alexey-milovidov). - Show error after
TrieDictionary
failed to load. #12290 (Vitaly Baranov). - The function
arrayFill
worked incorrectly for empty arrays that may lead to crash. This fixes #12263. #12279 (alexey-milovidov). - Implement conversions to the common type for
LowCardinality
types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes #8212. This fixes #4342. #12275 (alexey-milovidov). - Fixed the behaviour when during multiple sequential inserts in
StorageFile
header for some special types was written more than once. This fixed #6155. #12197 (Nikita Mikhaylov). - Fixed logical functions for UInt8 values when they are not equal to 0 or 1. #12196 (Alexander Kazakov).
- Fixed
dictGet
arguments check during GROUP BY injective functions elimination. #12179 (Azat Khuzhin). - Fixed wrong logic in
ALTER DELETE
that leads to deleting of records when condition evaluates to NULL. This fixes #9088. This closes #12106. #12153 (alexey-milovidov). - Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. #12151 (alexey-milovidov).
- Fixed potential overflow in integer division. This fixes #12119. #12140 (alexey-milovidov).
- Fixed potential infinite loop in
greatCircleDistance
,geoDistance
. This fixes #12117. #12137 (alexey-milovidov). - Avoid
There is no query
exception for materialized views with joins or with subqueries attached to system logs (system.query_log, metric_log, etc) or to engine=Buffer underlying table. #12120 (filimonov). - Fixed performance for selects with
UNION
caused by wrong limit for the total number of threads. Fixes #12030. #12103 (Nikolai Kochetov). - Fixed segfault with
-StateResample
combinators. #12092 (Anton Popov). - Fixed unnecessary limiting the number of threads for selects from
VIEW
. Fixes #11937. #12085 (Nikolai Kochetov). - Fixed possible crash while using wrong type for
PREWHERE
. Fixes #12053, #12060. #12060 (Nikolai Kochetov). - Fixed error
Expected single dictionary argument for function
for functiondefaultValueOfArgumentType
withLowCardinality
type. Fixes #11808. #12056 (Nikolai Kochetov). - Fixed error
Cannot capture column
for higher-order functions withTuple(LowCardinality)
argument. Fixes #9766. #12055 (Nikolai Kochetov). - Parse tables metadata in parallel when loading database. This fixes slow server startup when there are large number of tables. #12045 (tavplubix).
- Make
topK
aggregate function return Enum for Enum types. This fixes #3740. #12043 (alexey-milovidov). - Fixed constraints check if constraint is a constant expression. This fixes #11360. #12042 (alexey-milovidov).
- Fixed incorrect comparison of tuples with
Nullable
columns. Fixes #11985. #12039 (Nikolai Kochetov). - Fixed wrong result and potential crash when invoking function
if
with arguments of typeFixedString
with different sizes. This fixes #11362. #12021 (alexey-milovidov). - A query with function
neighbor
as the only returned expression may return empty result if the function is called with offset-9223372036854775808
. This fixes #11367. #12019 (alexey-milovidov). - Fixed potential array size overflow in generateRandom that may lead to crash. This fixes #11371. #12013 (alexey-milovidov).
- Fixed potential floating point exception. This closes #11378. #12005 (alexey-milovidov).
- Fixed wrong setting name in log message at server startup. #11997 (alexey-milovidov).
- Fixed
Query parameter was not set
inValues
format. Fixes #11918. #11936 (tavplubix). - Keep aliases for substitutions in query (parametrized queries). This fixes #11914. #11916 (alexey-milovidov).
- Fixed potential floating point exception when parsing DateTime64. This fixes #11374. #11875 (alexey-milovidov).
- Fixed memory accounting via
HTTP
interface (can be significant withwait_end_of_query=1
). #11840 (Azat Khuzhin). - Fixed wrong result for
if()
with NULLs in condition. #11807 (Artem Zuikov). - Parse metadata stored in zookeeper before checking for equality. #11739 (Azat Khuzhin).
- Fixed
LIMIT n WITH TIES
usage together withORDER BY
statement, which contains aliases. #11689 (Anton Popov). - Fix potential read of uninitialized memory in cache dictionary. #10834 (alexey-milovidov).
Performance Improvement
- Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes #10574. #12062 (nvartolomei).
ClickHouse release v20.3.12.112-lts 2020-06-25
Bug Fix
- Fix rare crash caused by using
Nullable
column in prewhere condition. Continuation of #11608. #11869 (Nikolai Kochetov). - Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (alexey-milovidov).
- Fix using too many threads for queries. #11788 (Nikolai Kochetov).
- Fix unexpected behaviour of queries like
SELECT *, xyz.*
which were success while an error expected. #11753 (hexiaoting). - Now replicated fetches will be cancelled during metadata alter. #11744 (alesapin).
- Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (tavplubix).
- Fix
ORDER BY ... WITH FILL
over const columns. #11697 (Anton Popov). - Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (alexey-milovidov).
- Fix error which leads to an incorrect state of
system.mutations
. It may show that whole mutation is already done but the server still hasMUTATE_PART
tasks in the replication queue and tries to execute them. This fixes #11611. #11681 (alesapin). - Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (alexey-milovidov).
- Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. #11644 (alexey-milovidov).
- Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
- Fix rare crash caused by using
Nullable
column in prewhere condition. (Probably it is connected with #11572 somehow). #11608 (Nikolai Kochetov). - Fix error
Block structure mismatch
for queries with sampling reading fromBuffer
table. #11602 (Nikolai Kochetov). - Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. #11601 (filimonov).
- Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (alexey-milovidov).
- Fix error
Size of offsets does not match size of column
for queries withPREWHERE column in (subquery)
andARRAY JOIN
. #11580 (Nikolai Kochetov). - All queries in HTTP session have had the same query_id. It is fixed. #11578 (tavplubix).
- Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
- Fix shard_num/replica_num for
<node>
(breaks use_compact_format_in_distributed_parts_names). #11528 (Azat Khuzhin). - Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. #11496 (alexey-milovidov).
- Fix wrong results of distributed queries when alias could override qualified column name. Fixes #9672 #9714. #9972 (Artem Zuikov).
ClickHouse release v20.3.11.97-lts 2020-06-10
New Feature
- Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration:
strict_max_lifetime_seconds
, which ismax_lifetime
by default andquery_wait_timeout_milliseconds
, which is one minute by default. The first settings is also useful withallow_read_expired_keys
settings (to forbid reading very expired keys). #10337 (Nikita Mikhaylov).
Bug Fix
- Fix the error
Data compressed with different methods
that can happen ifmin_bytes_to_use_direct_io
is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (alexey-milovidov). - Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
- Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
- Fix pointInPolygon with nan as point. Fixes #11375. #11421 (Alexey Ilyukhov).
- Fix crash in JOIN over LowCarinality(T) and Nullable(T). #11380. #11414 (Artem Zuikov).
- Fix error code for wrong
USING
key. #11373. #11404 (Artem Zuikov). - Fixed geohashesInBox with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov).
- Better errors for
joinGet()
functions. #11389 (Artem Zuikov). - Fix possible
Pipeline stuck
error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov). - Remove redundant lock during parts send in ReplicatedMergeTree. #11354 (alesapin).
- Fix support for
\G
(vertical output) in clickhouse-client in multiline mode. This closes #9933. #11350 (alexey-milovidov). - Fix crash in direct selects from StorageJoin (without JOIN) and wrong nullability. #11340 (Artem Zuikov).
- Fix crash in
quantilesExactWeightedArray
. #11337 (Nikolai Kochetov). - Now merges stopped before change metadata in
ALTER
queries. #11335 (alesapin). - Make writing to
MATERIALIZED VIEW
with settingparallel_view_processing = 1
parallel again. Fixes #10241. #11330 (Nikolai Kochetov). - Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 (Ewout).
- Fix very rare race condition in ThreadPool. #11314 (alexey-milovidov).
- Fix potential uninitialized memory in conversion. Example:
SELECT toIntervalSecond(now64())
. #11311 (alexey-milovidov). - Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with
empty
ornotEmpty
functions. This fixes #11286. #11303 (alexey-milovidov). - Fix bug when query speed estimation can be incorrect and the limit of
min_execution_speed
may not work or work incorrectly if the query is throttled bymax_network_bandwidth
,max_execution_speed
orpriority
settings. Change the default value oftimeout_before_checking_execution_speed
to non-zero, because otherwise the settingsmin_execution_speed
andmax_execution_speed
have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar inclickhouse-client
. #11296 (alexey-milovidov). - Fix crash while reading malformed data in Protobuf format. This fixes #5957, fixes #11203. #11258 (Vitaly Baranov).
- Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 (Nikita Mikhaylov).
- Fix error
Block structure mismatch in QueryPipeline
while reading fromVIEW
with constants in inner query. Fixes #11181. #11205 (Nikolai Kochetov). - Fix possible exception
Invalid status for associated output
. #11200 (Nikolai Kochetov). - Fix possible error
Cannot capture column
for higher-order functions withArray(Array(LowCardinality))
captured argument. #11185 (Nikolai Kochetov). - Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 (Vladimir Chebotarev).
- If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
- Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 (Azat Khuzhin).
- Fix predicates optimization for distributed queries (
enable_optimize_predicate_expression=1
) for queries withHAVING
section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin). - Introduce commit retry logic to decrease the possibility of getting duplicates from Kafka in rare cases when offset commit was failed. #9884 (filimonov).
Performance Improvement
- Get dictionary and check access rights only once per each call of any function reading external dictionaries. #10928 (Vitaly Baranov).
Build/Testing/Packaging Improvement
ClickHouse release v20.3.10.75-lts 2020-05-23
Bug Fix
- Removed logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
- Fixed
parseDateTime64BestEffort
argument resolution bugs. #11038 (Vasily Nemkov). - Fixed incorrect raw data size in method
getRawData()
. #10964 (Igr). - Fixed incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of
GROUP BY
result is large and aggregation is performed by a singleString
field. It leads to several unmerged rows for a single key in result. #10952 (alexey-milovidov). - Fixed backward compatibility with tuples in
Distributed
tables. #10889 (Anton Popov). - Fixed
SIGSEGV
inStringHashTable
if such a key does not exist. #10870 (Azat Khuzhin). - Fixed bug in
ReplicatedMergeTree
which might cause someALTER
onOPTIMIZE
query to hang waiting for some replica after it become inactive. #10849 (tavplubix). - Fixed columns order after
Block::sortColumns()
. #10826 (Azat Khuzhin). - Fixed the issue with
ODBC
bridge when no quoting of identifiers is requested. Fixes #7984. #10821 (alexey-milovidov). - Fixed
UBSan
andMSan
report inDateLUT
. #10798 (alexey-milovidov). - Fixed incorrect type conversion in key conditions. Fixes #6287. #10791 (Andrew Onyshchuk)
- Fixed
parallel_view_processing
behavior. Now all insertions intoMATERIALIZED VIEW
without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov). - Fixed combinator -
OrNull
and-OrDefault
when combined with-State
. #10741 (hcz). - Fixed crash in
generateRandom
with nested types. Fixes #10583. #10734 (Nikolai Kochetov). - Fixed data corruption for
LowCardinality(FixedString)
key column inSummingMergeTree
which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov). - Fixed possible buffer overflow in function
h3EdgeAngle
. #10711 (alexey-milovidov). - Fixed disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes #10674. #10698 (Nikolai Kochetov).
- Fixed multiple usages of
IN
operator with the identical set in one query. #10686 (Anton Popov). - Fixed bug, which causes http requests stuck on client close when
readonly=2
andcancel_http_readonly_queries_on_client_close=1
. Fixes #7939, #7019, #7736, #7091. #10684 (tavplubix). - Fixed order of parameters in
AggregateTransform
constructor. #10667 (palasonic1). - Fixed the lack of parallel execution of remote queries with
distributed_aggregation_memory_efficient
enabled. Fixes #10655. #10664 (Nikolai Kochetov). - Fixed possible incorrect number of rows for queries with
LIMIT
. Fixes #10566, #10709. #10660 (Nikolai Kochetov). - Fixed a bug which locks concurrent alters when table has a lot of parts. #10659 (alesapin).
- Fixed a bug when on
SYSTEM DROP DNS CACHE
query also drop caches, which are used to check if user is allowed to connect from some IP addresses. #10608 (tavplubix). - Fixed incorrect scalar results inside inner query of
MATERIALIZED VIEW
in case if this query contained dependent table. #10603 (Nikolai Kochetov). - Fixed
SELECT
of columnALIAS
which default expression type different from column type. #10563 (Azat Khuzhin). - Implemented comparison between DateTime64 and String values. #10560 (Vasily Nemkov).
- Fixed index corruption, which may occur in some cases after merge compact parts into another compact part. #10531 (Anton Popov).
- Fixed the situation, when mutation finished all parts, but hung up in
is_done=0
. #10526 (alesapin). - Fixed overflow at beginning of unix epoch for timezones with fractional offset from
UTC
. This fixes #9335. #10513 (alexey-milovidov). - Fixed improper shutdown of
Distributed
storage. #10491 (Azat Khuzhin). - Fixed numeric overflow in
simpleLinearRegression
over large integers. #10474 (hcz).
Build/Testing/Packaging Improvement
- Fix UBSan report in LZ4 library. #10631 (alexey-milovidov).
- Fix clang-10 build. #10238. #10370 (Amos Bird).
- Added failing tests about
max_rows_to_sort
setting. #10268 (alexey-milovidov). - Added some improvements in printing diagnostic info in input formats. Fixes #10204. #10418 (tavplubix).
- Added CA certificates to clickhouse-server docker image. #10476 (filimonov).
Bug fix
- Fix error
the BloomFilter false positive must be a double number between 0 and 1
#10551. #10569 (Winter Zhang).
ClickHouse release v20.3.8.53, 2020-04-23
Bug Fix
- Fixed wrong behaviour of datetime functions for timezones that has altered between positive and negative offsets from UTC (e.g. Pacific/Kiritimati). This fixes #7202 #10369 (alexey-milovidov)
- Fix possible segfault with
distributed_group_by_no_merge
enabled (introduced in 20.3.7.46 by #10131). #10399 (Nikolai Kochetov) - Fix wrong flattening of
Array(Tuple(...))
data types. This fixes #10259 #10390 (alexey-milovidov) - Drop disks reservation in Aggregator. This fixes bug in disk space reservation, which may cause big external aggregation to fail even if it could be completed successfully #10375 (Azat Khuzhin)
- Fixed
DROP
vsOPTIMIZE
race inReplicatedMergeTree
.DROP
could left some garbage in replica path in ZooKeeper if there was concurrentOPTIMIZE
query. #10312 (tavplubix) - Fix bug when server cannot attach table after column default was altered. #10441 (alesapin)
- Do not remove metadata directory when attach database fails before loading tables. #10442 (Winter Zhang)
- Fixed several bugs when some data was inserted with quorum, then deleted somehow (DROP PARTITION, TTL) and this leaded to the stuck of INSERTs or false-positive exceptions in SELECTs. This fixes #9946 #10188 (Nikita Mikhaylov)
- Fix possible
Pipeline stuck
error inConcatProcessor
which could have happened in remote query. #10381 (Nikolai Kochetov) - Fixed wrong behavior in HashTable that caused compilation error when trying to read HashMap from buffer. #10386 (palasonic1)
- Allow to use
count(*)
with multiple JOINs. Fixes #9853 #10291 (Artem Zuikov) - Prefer
fallback_to_stale_replicas
overskip_unavailable_shards
, otherwise when both settings specified and there are no up-to-date replicas the query will fail (patch from @alex-zaitsev). Fixes: #2564. #10422 (Azat Khuzhin) - Fix the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. This fixes #10226. Author: Vadim Plakhtinskiy. #10427 (alexey-milovidov)
- Check the number and type of arguments when creating BloomFilter index #9623 #10431 (Winter Zhang)
Performance Improvement
- Improved performance of queries with explicitly defined sets at right side of
IN
operator and tuples in the left side. This fixes performance regression in version 20.3. #9740, #10385 (Anton Popov)
ClickHouse release v20.3.7.46, 2020-04-17
Bug Fix
- Fix
Logical error: CROSS JOIN has expressions
error for queries with comma and names joins mix. #10311 (Artem Zuikov). - Fix queries with
max_bytes_before_external_group_by
. #10302 (Artem Zuikov). - Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes #10092. #10195 (alexey-milovidov).
- Add the ability to relax the restriction on non-deterministic functions usage in mutations with
allow_nondeterministic_mutations
setting. #10186 (filimonov).
ClickHouse release v20.3.6.40, 2020-04-16
New Feature
- Added function
isConstant
. This function checks whether its argument is constant expression and returns 1 or 0. It is intended for development, debugging and demonstration purposes. #10198 (alexey-milovidov).
Bug Fix
- Fix error
Pipeline stuck
withmax_rows_to_group_by
andgroup_by_overflow_mode = 'break'
. #10279 (Nikolai Kochetov). - Fix rare possible exception
Cannot drain connections: cancel first
. #10239 (Nikolai Kochetov). - Fixed bug where ClickHouse would throw "Unknown function lambda." error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. #10237 (Alexander Kazakov).
- Fixed "generateRandom" function for Date type. This fixes #9973. Fix an edge case when dates with year 2106 are inserted to MergeTree tables with old-style partitioning but partitions are named with year 1970. #10218 (alexey-milovidov).
- Convert types if the table definition of a View does not correspond to the SELECT query. This fixes #10180 and #10022. #10217 (alexey-milovidov).
- Fix
parseDateTimeBestEffort
for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10082. #10214 (alexey-milovidov). - Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. #10207 (alexey-milovidov).
- Fix possible inifinite query execution when the query actually should stop on LIMIT, while reading from infinite source like
system.numbers
orsystem.zeros
. #10206 (Nikolai Kochetov). - Fix using the current database for access checking when the database isn't specified. #10192 (Vitaly Baranov).
- Convert blocks if structure does not match on INSERT into Distributed(). #10135 (Azat Khuzhin).
- Fix possible incorrect result for extremes in processors pipeline. #10131 (Nikolai Kochetov).
- Fix some kinds of alters with compact parts. #10130 (Anton Popov).
- Fix incorrect
index_granularity_bytes
check while creating new replica. Fixes #10098. #10121 (alesapin). - Fix SIGSEGV on INSERT into Distributed table when its structure differs from the underlying tables. #10105 (Azat Khuzhin).
- Fix possible rows loss for queries with
JOIN
andUNION ALL
. Fixes #9826, #10113. #10099 (Nikolai Kochetov). - Fixed replicated tables startup when updating from an old ClickHouse version where
/table/replicas/replica_name/metadata
node does not exist. Fixes #10037. #10095 (alesapin). - Add some arguments check and support identifier arguments for MySQL Database Engine. #10077 (Winter Zhang).
- Fix bug in clickhouse dictionary source from localhost clickhouse server. The bug may lead to memory corruption if types in dictionary and source are not compatible. #10071 (alesapin).
- Fix bug in
CHECK TABLE
query when table contain skip indices. #10068 (alesapin). - Fix error
Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform
. It happened when settingdistributed_aggregation_memory_efficient
was enabled, and distributed query read aggregating data with different level from different shards (mixed single and two level aggregation). #10063 (Nikolai Kochetov). - Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (#8636, #8925). #10025 (Alexander Kuzmenkov).
- Fix the number of threads used for remote query execution (performance regression, since 20.3). This happened when query from
Distributed
table was executed simultaneously on local and remote shards. Fixes #9965. #9971 (Nikolai Kochetov). - Fix bug in which the necessary tables weren't retrieved at one of the processing stages of queries to some databases. Fixes #9699. #9949 (achulkov2).
- Fix 'Not found column in block' error when
JOIN
appears withTOTALS
. Fixes #9839. #9939 (Artem Zuikov). - Fix a bug with
ON CLUSTER
DDL queries freezing on server startup. #9927 (Gagan Arneja). - Fix parsing multiple hosts set in the CREATE USER command, e.g.
CREATE USER user6 HOST NAME REGEXP 'lo.?*host', NAME REGEXP 'lo*host'
. #9924 (Vitaly Baranov). - Fix
TRUNCATE
for Join table engine (#9917). #9920 (Amos Bird). - Fix "scalar does not exist" error in ALTERs (#9878). #9904 (Amos Bird).
- Fix race condition between drop and optimize in
ReplicatedMergeTree
. #9901 (alesapin). - Fix error with qualified names in
distributed_product_mode='local'
. Fixes #4756. #9891 (Artem Zuikov). - Fix calculating grants for introspection functions from the setting 'allow_introspection_functions'. #9840 (Vitaly Baranov).
Build/Testing/Packaging Improvement
- Fix integration test
test_settings_constraints
. #9962 (Vitaly Baranov). - Removed dependency on
clock_getres
. #9833 (alexey-milovidov).
ClickHouse release v20.3.5.21, 2020-03-27
Bug Fix
- Fix 'Different expressions with the same alias' error when query has PREWHERE and WHERE on distributed table and
SET distributed_product_mode = 'local'
. #9871 (Artem Zuikov). - Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. #9860 (alesapin).
- For INSERT queries shard now clamps the settings got from the initiator to the shard's constaints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix #9447. #9852 (Vitaly Baranov).
- Fix 'COMMA to CROSS JOIN rewriter is not enabled or cannot rewrite query' error in case of subqueries with COMMA JOIN out of tables lists (i.e. in WHERE). Fixes #9782. #9830 (Artem Zuikov).
- Fix possible exception
Got 0 in totals chunk, expected 1
on client. It happened for queries withJOIN
in case if right joined table had zero rows. Example:select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;
. Fixes #9777. #9823 (Nikolai Kochetov). - Fix SIGSEGV with optimize_skip_unused_shards when type cannot be converted. #9804 (Azat Khuzhin).
- Fix broken
ALTER TABLE DELETE COLUMN
query for compact parts. #9779 (alesapin). - Fix max_distributed_connections (w/ and w/o Processors). #9673 (Azat Khuzhin).
- Fixed a few cases when timezone of the function argument wasn't used properly. #9574 (Vasily Nemkov).
Improvement
- Remove order by stage from mutations because we read from a single ordered part in a single thread. Also add check that the order of rows in mutation is ordered in sorting key order and this order is not violated. #9886 (alesapin).
ClickHouse release v20.3.4.10, 2020-03-20
Bug Fix
- This release also contains all bug fixes from 20.1.8.41
- Fix missing
rows_before_limit_at_least
for queries over http (with processors pipeline). This fixes #9730. #9757 (Nikolai Kochetov)
ClickHouse release v20.3.3.6, 2020-03-17
Bug Fix
- This release also contains all bug fixes from 20.1.7.38
- Fix bug in a replication that does not allow replication to work if the user has executed mutations on the previous version. This fixes #9645. #9652 (alesapin). It makes version 20.3 backward compatible again.
- Add setting
use_compact_format_in_distributed_parts_names
which allows to write files forINSERT
queries intoDistributed
table with more compact format. This fixes #9647. #9653 (alesapin). It makes version 20.3 backward compatible again.
ClickHouse release v20.3.2.1, 2020-03-12
Backward Incompatible Change
- Fixed the issue
file name too long
when sending data forDistributed
tables for a large number of replicas. Fixed the issue that replica credentials were exposed in the server log. The format of directory name on disk was changed to[shard{shard_index}[_replica{replica_index}]]
. #8911 (Mikhail Korotov) After you upgrade to the new version, you will not be able to downgrade without manual intervention, because old server version does not recognize the new directory format. If you want to downgrade, you have to manually rename the corresponding directories to the old format. This change is relevant only if you have used asynchronousINSERT
s toDistributed
tables. In the version 20.3.3 we will introduce a setting that will allow you to enable the new format gradually. - Changed the format of replication log entries for mutation commands. You have to wait for old mutations to process before installing the new version.
- Implement simple memory profiler that dumps stacktraces to
system.trace_log
every N bytes over soft allocation limit #8765 (Ivan) #9472 (alexey-milovidov) The column ofsystem.trace_log
was renamed fromtimer_type
totrace_type
. This will require changes in third-party performance analysis and flamegraph processing tools. - Use OS thread id everywhere instead of internal thread number. This fixes #7477 Old
clickhouse-client
cannot receive logs that are send from the server when the settingsend_logs_level
is enabled, because the names and types of the structured log messages were changed. On the other hand, different server versions can send logs with different types to each other. When you don't use thesend_logs_level
setting, you should not care. #8954 (alexey-milovidov) - Remove
indexHint
function #9542 (alexey-milovidov) - Remove
findClusterIndex
,findClusterValue
functions. This fixes #8641. If you were using these functions, send an email to[email protected]
#9543 (alexey-milovidov) - Now it's not allowed to create columns or add columns with
SELECT
subquery as default expression. #9481 (alesapin) - Require aliases for subqueries in JOIN. #9274 (Artem Zuikov)
- Improved
ALTER MODIFY/ADD
queries logic. Now you cannotADD
column without type,MODIFY
default expression does not change type of column andMODIFY
type does not loose default expression value. Fixes #8669. #9227 (alesapin) - Require server to be restarted to apply the changes in logging configuration. This is a temporary workaround to avoid the bug where the server logs to a deleted log file (see #8696). #8707 (Alexander Kuzmenkov)
- The setting
experimental_use_processors
is enabled by default. This setting enables usage of the new query pipeline. This is internal refactoring and we expect no visible changes. If you will see any issues, set it to back zero. #8768 (alexey-milovidov)
New Feature
- Add
Avro
andAvroConfluent
input/output formats #8571 (Andrew Onyshchuk) #8957 (Andrew Onyshchuk) #8717 (alexey-milovidov) - Multi-threaded and non-blocking updates of expired keys in
cache
dictionaries (with optional permission to read old ones). #8303 (Nikita Mikhaylov) - Add query
ALTER ... MATERIALIZE TTL
. It runs mutation that forces to remove expired data by TTL and recalculates meta-information about TTL in all parts. #8775 (Anton Popov) - Switch from HashJoin to MergeJoin (on disk) if needed #9082 (Artem Zuikov)
- Added
MOVE PARTITION
command forALTER TABLE
#4729 #6168 (Guillaume Tassery) - Reloading storage configuration from configuration file on the fly. #8594 (Vladimir Chebotarev)
- Allowed to change
storage_policy
to not less rich one. #8107 (Vladimir Chebotarev) - Added support for globs/wildcards for S3 storage and table function. #8851 (Vladimir Chebotarev)
- Implement
bitAnd
,bitOr
,bitXor
,bitNot
forFixedString(N)
datatype. #9091 (Guillaume Tassery) - Added function
bitCount
. This fixes #8702. #8708 (alexey-milovidov) #8749 (ikopylov) - Add
generateRandom
table function to generate random rows with given schema. Allows to populate arbitrary test table with data. #8994 (Ilya Yatsishin) JSONEachRowFormat
: support special case when objects enclosed in top-level array. #8860 (Kruglov Pavel)- Now it's possible to create a column with
DEFAULT
expression which depends on a column with defaultALIAS
expression. #9489 (alesapin) - Allow to specify
--limit
more than the source data size inclickhouse-obfuscator
. The data will repeat itself with different random seed. #9155 (alexey-milovidov) - Added
groupArraySample
function (similar togroupArray
) with reservior sampling algorithm. #8286 (Amos Bird) - Now you can monitor the size of update queue in
cache
/complex_key_cache
dictionaries via system metrics. #9413 (Nikita Mikhaylov) - Allow to use CRLF as a line separator in CSV output format with setting
output_format_csv_crlf_end_of_line
is set to 1 #8934 #8935 #8963 (Mikhail Korotov) - Implement more functions of the H3 API:
h3GetBaseCell
,h3HexAreaM2
,h3IndexesAreNeighbors
,h3ToChildren
,h3ToString
andstringToH3
#8938 (Nico Mandery) - New setting introduced:
max_parser_depth
to control maximum stack size and allow large complex queries. This fixes #6681 and #7668. #8647 (Maxim Smirnov) - Add a setting
force_optimize_skip_unused_shards
setting to throw if skipping of unused shards is not possible #8805 (Azat Khuzhin) - Allow to configure multiple disks/volumes for storing data for send in
Distributed
engine #8756 (Azat Khuzhin) - Support storage policy (
<tmp_policy>
) for storing temporary data. #8750 (Azat Khuzhin) - Added
X-ClickHouse-Exception-Code
HTTP header that is set if exception was thrown before sending data. This implements #4971. #8786 (Mikhail Korotov) - Added function
ifNotFinite
. It is just a syntactic sugar:ifNotFinite(x, y) = isFinite(x) ? x : y
. #8710 (alexey-milovidov) - Added
last_successful_update_time
column insystem.dictionaries
table #9394 (Nikita Mikhaylov) - Add
blockSerializedSize
function (size on disk without compression) #8952 (Azat Khuzhin) - Add function
moduloOrZero
#9358 (hcz) - Added system tables
system.zeros
andsystem.zeros_mt
as well as tale functionszeros()
andzeros_mt()
. Tables (and table functions) contain single column with namezero
and typeUInt8
. This column contains zeros. It is needed for test purposes as the fastest method to generate many rows. This fixes #6604 #9593 (Nikolai Kochetov)
Experimental Feature
- Add new compact format of parts in
MergeTree
-family tables in which all columns are stored in one file. It helps to increase performance of small and frequent inserts. The old format (one file per column) is now called wide. Data storing format is controlled by settingsmin_bytes_for_wide_part
andmin_rows_for_wide_part
. #8290 (Anton Popov) - Support for S3 storage for
Log
,TinyLog
andStripeLog
tables. #8862 (Pavel Kovalenko)
Bug Fix
- Fixed inconsistent whitespaces in log messages. #9322 (alexey-milovidov)
- Fix bug in which arrays of unnamed tuples were flattened as Nested structures on table creation. #8866 (achulkov2)
- Fixed the issue when "Too many open files" error may happen if there are too many files matching glob pattern in
File
table orfile
table function. Now files are opened lazily. This fixes #8857 #8861 (alexey-milovidov) - DROP TEMPORARY TABLE now drops only temporary table. #8907 (Vitaly Baranov)
- Remove outdated partition when we shutdown the server or DETACH/ATTACH a table. #8602 (Guillaume Tassery)
- For how the default disk calculates the free space from
data
subdirectory. Fixed the issue when the amount of free space is not calculated correctly if thedata
directory is mounted to a separate device (rare case). This fixes #7441 #9257 (Mikhail Korotov) - Allow comma (cross) join with IN () inside. #9251 (Artem Zuikov)
- Allow to rewrite CROSS to INNER JOIN if there's [NOT] LIKE operator in WHERE section. #9229 (Artem Zuikov)
- Fix possible incorrect result after
GROUP BY
with enabled settingdistributed_aggregation_memory_efficient
. Fixes #9134. #9289 (Nikolai Kochetov) - Found keys were counted as missed in metrics of cache dictionaries. #9411 (Nikita Mikhaylov)
- Fix replication protocol incompatibility introduced in #8598. #9412 (alesapin)
- Fixed race condition on
queue_task_handle
at the startup ofReplicatedMergeTree
tables. #9552 (alexey-milovidov) - The token
NOT
did not work inSHOW TABLES NOT LIKE
query #8727 #8940 (alexey-milovidov) - Added range check to function
h3EdgeLengthM
. Without this check, buffer overflow is possible. #8945 (alexey-milovidov) - Fixed up a bug in batched calculations of ternary logical OPs on multiple arguments (more than 10). #8718 (Alexander Kazakov)
- Fix error of PREWHERE optimization, which could lead to segfaults or
Inconsistent number of columns got from MergeTreeRangeReader
exception. #9024 (Anton Popov) - Fix unexpected
Timeout exceeded while reading from socket
exception, which randomly happens on secure connection before timeout actually exceeded and when query profiler is enabled. Also addconnect_timeout_with_failover_secure_ms
settings (default 100ms), which is similar toconnect_timeout_with_failover_ms
, but is used for secure connections (because SSL handshake is slower, than ordinary TCP connection) #9026 (tavplubix) - Fix bug with mutations finalization, when mutation may hang in state with
parts_to_do=0
andis_done=0
. #9022 (alesapin) - Use new ANY JOIN logic with
partial_merge_join
setting. It's possible to makeANY|ALL|SEMI LEFT
andALL INNER
joins withpartial_merge_join=1
now. #8932 (Artem Zuikov) - Shard now clamps the settings got from the initiator to the shard's constaints instead of throwing an exception. This fix allows to send queries to a shard with another constraints. #9447 (Vitaly Baranov)
- Fixed memory management problem in
MergeTreeReadPool
. #8791 (Vladimir Chebotarev) - Fix
toDecimal*OrNull()
functions family when called with stringe
. Fixes #8312 #8764 (Artem Zuikov) - Make sure that
FORMAT Null
sends no data to the client. #8767 (Alexander Kuzmenkov) - Fix bug that timestamp in
LiveViewBlockInputStream
will not updated.LIVE VIEW
is an experimental feature. #8644 (vxider) #8625 (vxider) - Fixed
ALTER MODIFY TTL
wrong behavior which did not allow to delete old TTL expressions. #8422 (Vladimir Chebotarev) - Fixed UBSan report in MergeTreeIndexSet. This fixes #9250 #9365 (alexey-milovidov)
- Fixed the behaviour of
match
andextract
functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes #9160 #9163 (alexey-milovidov) #9345 (alexey-milovidov) - Avoid throwing from destructor in Apache Avro 3rd-party library. #9066 (Andrew Onyshchuk)
- Don't commit a batch polled from
Kafka
partially as it can lead to holes in data. #8876 (filimonov) - Fix
joinGet
with nullable return types. #8919 #9014 (Amos Bird) - Fix data incompatibility when compressed with
T64
codec. #9016 (Artem Zuikov) Fix data type ids inT64
compression codec that leads to wrong (de)compression in affected versions. #9033 (Artem Zuikov) - Add setting
enable_early_constant_folding
and disable it in some cases that leads to errors. #9010 (Artem Zuikov) - Fix pushdown predicate optimizer with VIEW and enable the test #9011 (Winter Zhang)
- Fix segfault in
Merge
tables, that can happen when reading fromFile
storages #9387 (tavplubix) - Added a check for storage policy in
ATTACH PARTITION FROM
,REPLACE PARTITION
,MOVE TO TABLE
. Otherwise it could make data of part inaccessible after restart and prevent ClickHouse to start. #9383 (Vladimir Chebotarev) - Fix alters if there is TTL set for table. #8800 (Anton Popov)
- Fix race condition that can happen when
SYSTEM RELOAD ALL DICTIONARIES
is executed while some dictionary is being modified/added/removed. #8801 (Vitaly Baranov) - In previous versions
Memory
database engine use empty data path, so tables are created inpath
directory (e.g./var/lib/clickhouse/
), not in data directory of database (e.g./var/lib/clickhouse/db_name
). #8753 (tavplubix) - Fixed wrong log messages about missing default disk or policy. #9530 (Vladimir Chebotarev)
- Fix not(has()) for the bloom_filter index of array types. #9407 (achimbab)
- Allow first column(s) in a table with
Log
engine be an alias #9231 (Ivan) - Fix order of ranges while reading from
MergeTree
table in one thread. It could lead to exceptions fromMergeTreeRangeReader
or wrong query results. #9050 (Anton Popov) - Make
reinterpretAsFixedString
to returnFixedString
instead ofString
. #9052 (Andrew Onyshchuk) - Avoid extremely rare cases when the user can get wrong error message (
Success
instead of detailed error description). #9457 (alexey-milovidov) - Do not crash when using
Template
format with empty row template. #8785 (Alexander Kuzmenkov) - Metadata files for system tables could be created in wrong place #8653 (tavplubix) Fixes #8581.
- Fix data race on exception_ptr in cache dictionary #8303. #9379 (Nikita Mikhaylov)
- Do not throw an exception for query
ATTACH TABLE IF NOT EXISTS
. Previously it was thrown if table already exists, despite theIF NOT EXISTS
clause. #8967 (Anton Popov) - Fixed missing closing paren in exception message. #8811 (alexey-milovidov)
- Avoid message
Possible deadlock avoided
at the startup of clickhouse-client in interactive mode. #9455 (alexey-milovidov) - Fixed the issue when padding at the end of base64 encoded value can be malformed. Update base64 library. This fixes #9491, closes #9492 #9500 (alexey-milovidov)
- Prevent losing data in
Kafka
in rare cases when exception happens after reading suffix but before commit. Fixes #9378 #9507 (filimonov) - Fixed exception in
DROP TABLE IF EXISTS
#8663 (Nikita Vasilev) - Fix crash when a user tries to
ALTER MODIFY SETTING
for old-formatedMergeTree
table engines family. #9435 (alesapin) - Support for UInt64 numbers that don't fit in Int64 in JSON-related functions. Update SIMDJSON to master. This fixes #9209 #9344 (alexey-milovidov)
- Fixed execution of inversed predicates when non-strictly monotinic functional index is used. #9223 (Alexander Kazakov)
- Don't try to fold
IN
constant inGROUP BY
#8868 (Amos Bird) - Fix bug in
ALTER DELETE
mutations which leads to index corruption. This fixes #9019 and #8982. Additionally fix extremely rare race conditions inReplicatedMergeTree
ALTER
queries. #9048 (alesapin) - When the setting
compile_expressions
is enabled, you can getunexpected column
inLLVMExecutableFunction
when we useNullable
type #8910 (Guillaume Tassery) - Multiple fixes for
Kafka
engine: 1) fix duplicates that were appearing during consumer group rebalance. 2) Fix rare 'holes' appeared when data were polled from several partitions with one poll and committed partially (now we always process / commit the whole polled block of messages). 3) Fix flushes by block size (before that only flushing by timeout was working properly). 4) better subscription procedure (with assignment feedback). 5) Make tests work faster (with default intervals and timeouts). Due to the fact that data was not flushed by block size before (as it should according to documentation), that PR may lead to some performance degradation with default settings (due to more often & tinier flushes which are less optimal). If you encounter the performance issue after that change - please increasekafka_max_block_size
in the table to the bigger value ( for exampleCREATE TABLE ...Engine=Kafka ... SETTINGS ... kafka_max_block_size=524288
). Fixes #7259 #8917 (filimonov) - Fix
Parameter out of bound
exception in some queries after PREWHERE optimizations. #8914 (Baudouin Giard) - Fixed the case of mixed-constness of arguments of function
arrayZip
. #8705 (alexey-milovidov) - When executing
CREATE
query, fold constant expressions in storage engine arguments. Replace empty database name with current database. Fixes #6508, #3492 #9262 (tavplubix) - Now it's not possible to create or add columns with simple cyclic aliases like
a DEFAULT b, b DEFAULT a
. #9603 (alesapin) - Fixed a bug with double move which may corrupt original part. This is relevant if you use
ALTER TABLE MOVE
#8680 (Vladimir Chebotarev) - Allow
interval
identifier to correctly parse without backticks. Fixed issue when a query cannot be executed even if theinterval
identifier is enclosed in backticks or double quotes. This fixes #9124. #9142 (alexey-milovidov) - Fixed fuzz test and incorrect behaviour of
bitTestAll
/bitTestAny
functions. #9143 (alexey-milovidov) - Fix possible crash/wrong number of rows in
LIMIT n WITH TIES
when there are a lot of rows equal to n'th row. #9464 (tavplubix) - Fix mutations with parts written with enabled
insert_quorum
. #9463 (alesapin) - Fix data race at destruction of
Poco::HTTPServer
. It could happen when server is started and immediately shut down. #9468 (Anton Popov) - Fix bug in which a misleading error message was shown when running
SHOW CREATE TABLE a_table_that_does_not_exist
. #8899 (achulkov2) - Fixed
Parameters are out of bound
exception in some rare cases when we have a constant in theSELECT
clause when we have anORDER BY
and aLIMIT
clause. #8892 (Guillaume Tassery) - Fix mutations finalization, when already done mutation can have status
is_done=0
. #9217 (alesapin) - Prevent from executing
ALTER ADD INDEX
for MergeTree tables with old syntax, because it does not work. #8822 (Mikhail Korotov) - During server startup do not access table, which
LIVE VIEW
depends on, so server will be able to start. Also removeLIVE VIEW
dependencies when detachingLIVE VIEW
.LIVE VIEW
is an experimental feature. #8824 (tavplubix) - Fix possible segfault in
MergeTreeRangeReader
, while executingPREWHERE
. #9106 (Anton Popov) - Fix possible mismatched checksums with column TTLs. #9451 (Anton Popov)
- Fixed a bug when parts were not being moved in background by TTL rules in case when there is only one volume. #8672 (Vladimir Chebotarev)
- Fixed the issue
Method createColumn() is not implemented for data type Set
. This fixes #7799. #8674 (alexey-milovidov) - Now we will try finalize mutations more frequently. #9427 (alesapin)
- Fix
intDiv
by minus one constant #9351 (hcz) - Fix possible race condition in
BlockIO
. #9356 (Nikolai Kochetov) - Fix bug leading to server termination when trying to use / drop
Kafka
table created with wrong parameters. #9513 (filimonov) - Added workaround if OS returns wrong result for
timer_create
function. #8837 (alexey-milovidov) - Fixed error in usage of
min_marks_for_seek
parameter. Fixed the error message when there is no sharding key in Distributed table and we try to skip unused shards. #8908 (Azat Khuzhin)
Improvement
- Implement
ALTER MODIFY/DROP
queries on top of mutations forReplicatedMergeTree*
engines family. NowALTERS
blocks only at the metadata update stage, and don't block after that. #8701 (alesapin) - Add ability to rewrite CROSS to INNER JOINs with
WHERE
section containing unqialified names. #9512 (Artem Zuikov) - Make
SHOW TABLES
andSHOW DATABASES
queries support theWHERE
expressions andFROM
/IN
#9076 (sundyli) - Added a setting
deduplicate_blocks_in_dependent_materialized_views
. #9070 (urykhy) - After recent changes MySQL client started to print binary strings in hex thereby making them not readable (#9032). The workaround in ClickHouse is to mark string columns as UTF-8, which is not always, but usually the case. #9079 (Yuriy Baranov)
- Add support of String and FixedString keys for
sumMap
#8903 (Baudouin Giard) - Support string keys in SummingMergeTree maps #8933 (Baudouin Giard)
- Signal termination of thread to the thread pool even if the thread has thrown exception #8736 (Ding Xiang Fei)
- Allow to set
query_id
inclickhouse-benchmark
#9416 (Anton Popov) - Don't allow strange expressions in
ALTER TABLE ... PARTITION partition
query. This addresses #7192 #8835 (alexey-milovidov) - The table
system.table_engines
now provides information about feature support (likesupports_ttl
orsupports_sort_order
). #8830 (Max Akhmedov) - Enable
system.metric_log
by default. It will contain rows with values of ProfileEvents, CurrentMetrics collected with "collect_interval_milliseconds" interval (one second by default). The table is very small (usually in order of megabytes) and collecting this data by default is reasonable. #9225 (alexey-milovidov) - Initialize query profiler for all threads in a group, e.g. it allows to fully profile insert-queries. Fixes #6964 #8874 (Ivan)
- Now temporary
LIVE VIEW
is created byCREATE LIVE VIEW name WITH TIMEOUT [42] ...
instead ofCREATE TEMPORARY LIVE VIEW ...
, because the previous syntax was not consistent withCREATE TEMPORARY TABLE ...
#9131 (tavplubix) - Add text_log.level configuration parameter to limit entries that goes to
system.text_log
table #8809 (Azat Khuzhin) - Allow to put downloaded part to a disks/volumes according to TTL rules #8598 (Vladimir Chebotarev)
- For external MySQL dictionaries, allow to mutualize MySQL connection pool to "share" them among dictionaries. This option significantly reduces the number of connections to MySQL servers. #9409 (Clément Rodriguez)
- Show nearest query execution time for quantiles in
clickhouse-benchmark
output instead of interpolated values. It's better to show values that correspond to the execution time of some queries. #8712 (alexey-milovidov) - Possibility to add key & timestamp for the message when inserting data to Kafka. Fixes #7198 #8969 (filimonov)
- If server is run from terminal, highlight thread number, query id and log priority by colors. This is for improved readability of correlated log messages for developers. #8961 (alexey-milovidov)
- Better exception message while loading tables for
Ordinary
database. #9527 (alexey-milovidov) - Implement
arraySlice
for arrays with aggregate function states. This fixes #9388 #9391 (alexey-milovidov) - Allow constant functions and constant arrays to be used on the right side of IN operator. #8813 (Anton Popov)
- If zookeeper exception has happened while fetching data for system.replicas, display it in a separate column. This implements #9137 #9138 (alexey-milovidov)
- Atomically remove MergeTree data parts on destroy. #8402 (Vladimir Chebotarev)
- Support row-level security for Distributed tables. #8926 (Ivan)
- Now we recognize suffix (like KB, KiB...) in settings values. #8072 (Mikhail Korotov)
- Prevent out of memory while constructing result of a large JOIN. #8637 (Artem Zuikov)
- Added names of clusters to suggestions in interactive mode in
clickhouse-client
. #8709 (alexey-milovidov) - Initialize query profiler for all threads in a group, e.g. it allows to fully profile insert-queries #8820 (Ivan)
- Added column
exception_code
insystem.query_log
table. #8770 (Mikhail Korotov) - Enabled MySQL compatibility server on port
9004
in the default server configuration file. Fixed password generation command in the example in configuration. #8771 (Yuriy Baranov) - Prevent abort on shutdown if the filesystem is readonly. This fixes #9094 #9100 (alexey-milovidov)
- Better exception message when length is required in HTTP POST query. #9453 (alexey-milovidov)
- Add
_path
and_file
virtual columns toHDFS
andFile
engines andhdfs
andfile
table functions #8489 (Olga Khvostikova) - Fix error
Cannot find column
while inserting intoMATERIALIZED VIEW
in case if new column was added to view's internal table. #8766 #8788 (vzakaznikov) #8788 #8806 (Nikolai Kochetov) #8803 (Nikolai Kochetov) - Fix progress over native client-server protocol, by send progress after final update (like logs). This may be relevant only to some third-party tools that are using native protocol. #9495 (Azat Khuzhin)
- Add a system metric tracking the number of client connections using MySQL protocol (#9013). #9015 (Eugene Klimov)
- From now on, HTTP responses will have
X-ClickHouse-Timezone
header set to the same timezone value thatSELECT timezone()
would report. #9493 (Denis Glazachev)
Performance Improvement
- Improve performance of analysing index with IN #9261 (Anton Popov)
- Simpler and more efficient code in Logical Functions + code cleanups. A followup to #8718 #8728 (Alexander Kazakov)
- Overall performance improvement (in range of 5%..200% for affected queries) by ensuring even more strict aliasing with C++20 features. #9304 (Amos Bird)
- More strict aliasing for inner loops of comparison functions. #9327 (alexey-milovidov)
- More strict aliasing for inner loops of arithmetic functions. #9325 (alexey-milovidov)
- A ~3 times faster implementation for ColumnVector::replicate(), via which ColumnConst::convertToFullColumn() is implemented. Also will be useful in tests when materializing constants. #9293 (Alexander Kazakov)
- Another minor performance improvement to
ColumnVector::replicate()
(this speeds up thematerialize
function and higher order functions) an even further improvement to #9293 #9442 (Alexander Kazakov) - Improved performance of
stochasticLinearRegression
aggregate function. This patch is contributed by Intel. #8652 (alexey-milovidov) - Improve performance of
reinterpretAsFixedString
function. #9342 (alexey-milovidov) - Do not send blocks to client for
Null
format in processors pipeline. #8797 (Nikolai Kochetov) #8767 (Alexander Kuzmenkov)
Build/Testing/Packaging Improvement
- Exception handling now works correctly on Windows Subsystem for Linux. See https://github.com/ClickHouse-Extras/libunwind/pull/3 This fixes #6480 #9564 (sobolevsv)
- Replace
readline
withreplxx
for interactive line editing inclickhouse-client
#8416 (Ivan) - Better build time and less template instantiations in FunctionsComparison. #9324 (alexey-milovidov)
- Added integration with
clang-tidy
in CI. See also #6044 #9566 (alexey-milovidov) - Now we link ClickHouse in CI using
lld
even forgcc
. #9049 (alesapin) - Allow to randomize thread scheduling and insert glitches when
THREAD_FUZZER_*
environment variables are set. This helps testing. #9459 (alexey-milovidov) - Enable secure sockets in stateless tests #9288 (tavplubix)
- Make SPLIT_SHARED_LIBRARIES=OFF more robust #9156 (Azat Khuzhin)
- Make "performance_introspection_and_logging" test reliable to random server stuck. This may happen in CI environment. See also #9515 #9528 (alexey-milovidov)
- Validate XML in style check. #9550 (alexey-milovidov)
- Fixed race condition in test
00738_lock_for_inner_table
. This test relied on sleep. #9555 (alexey-milovidov) - Remove performance tests of type
once
. This is needed to run all performance tests in statistical comparison mode (more reliable). #9557 (alexey-milovidov) - Added performance test for arithmetic functions. #9326 (alexey-milovidov)
- Added performance test for
sumMap
andsumMapWithOverflow
aggregate functions. Follow-up for #8933 #8947 (alexey-milovidov) - Ensure style of ErrorCodes by style check. #9370 (alexey-milovidov)
- Add script for tests history. #8796 (alesapin)
- Add GCC warning
-Wsuggest-override
to locate and fix all places whereoverride
keyword must be used. #8760 (kreuzerkrieg) - Ignore weak symbol under Mac OS X because it must be defined #9538 (Deleted user)
- Normalize running time of some queries in performance tests. This is done in preparation to run all the performance tests in comparison mode. #9565 (alexey-milovidov)
- Fix some tests to support pytest with query tests #9062 (Ivan)
- Enable SSL in build with MSan, so server will not fail at startup when running stateless tests #9531 (tavplubix)
- Fix database substitution in test results #9384 (Ilya Yatsishin)
- Build fixes for miscellaneous platforms #9381 (proller) #8755 (proller) #8631 (proller)
- Added disks section to stateless-with-coverage test docker image #9213 (Pavel Kovalenko)
- Get rid of in-source-tree files when building with GRPC #9588 (Amos Bird)
- Slightly faster build time by removing SessionCleaner from Context. Make the code of SessionCleaner more simple. #9232 (alexey-milovidov)
- Updated checking for hung queries in clickhouse-test script #8858 (Alexander Kazakov)
- Removed some useless files from repository. #8843 (alexey-milovidov)
- Changed type of math perftests from
once
toloop
. #8783 (Nikolai Kochetov) - Add docker image which allows to build interactive code browser HTML report for our codebase. #8781 (alesapin) See Woboq Code Browser
- Suppress some test failures under MSan. #8780 (Alexander Kuzmenkov)
- Speedup "exception while insert" test. This test often time out in debug-with-coverage build. #8711 (alexey-milovidov)
- Updated
libcxx
andlibcxxabi
to master. In preparation to #9304 #9308 (alexey-milovidov) - Fix flacky test
00910_zookeeper_test_alter_compression_codecs
. #9525 (alexey-milovidov) - Clean up duplicated linker flags. Make sure the linker won't look up an unexpected symbol. #9433 (Amos Bird)
- Add
clickhouse-odbc
driver into test images. This allows to test interaction of ClickHouse with ClickHouse via its own ODBC driver. #9348 (filimonov) - Fix several bugs in unit tests. #9047 (alesapin)
- Enable
-Wmissing-include-dirs
GCC warning to eliminate all non-existing includes - mostly as a result of CMake scripting errors #8704 (kreuzerkrieg) - Describe reasons if query profiler cannot work. This is intended for #9049 #9144 (alexey-milovidov)
- Update OpenSSL to upstream master. Fixed the issue when TLS connections may fail with the message
OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
andSSL Exception: error:2400006E:random number generator::error retrieving entropy
. The issue was present in version 20.1. #8956 (alexey-milovidov) - Update Dockerfile for server #8893 (Ilya Mazaev)
- Minor fixes in build-gcc-from-sources script #8774 (Michael Nacharov)
- Replace
numbers
tozeros
in perftests wherenumber
column is not used. This will lead to more clean test results. #9600 (Nikolai Kochetov) - Fix stack overflow issue when using initializer_list in Column constructors. #9367 (Deleted user)
- Upgrade librdkafka to v1.3.0. Enable bundled
rdkafka
andgsasl
libraries on Mac OS X. #9000 (Andrew Onyshchuk) - build fix on GCC 9.2.0 #9306 (vxider)
ClickHouse release v20.1
ClickHouse release v20.1.16.120-stable 2020-60-26
Bug Fix
- Fix rare crash caused by using
Nullable
column in prewhere condition. Continuation of #11608. #11869 (Nikolai Kochetov). - Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (alexey-milovidov).
- Fix unexpected behaviour of queries like
SELECT *, xyz.*
which were success while an error expected. #11753 (hexiaoting). - Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (tavplubix).
- Fix
ORDER BY ... WITH FILL
over const columns. #11697 (Anton Popov). - Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (alexey-milovidov).
- Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (alexey-milovidov).
- Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
- Fix rare crash caused by using
Nullable
column in prewhere condition. (