iceberg_insert_max_bytes_in_data_file
Max bytes of iceberg parquet data file on insert operation. The default mirrors the documented default of the Iceberg table propertywrite.target-file-size-bytes (512 MiB),
see https://iceberg.apache.org/docs/1.5.2/configuration/. The limit is compared against the size of the file
written so far, that is, the compressed size, so a file can exceed it by what the writer still buffers, such as
incomplete Parquet row groups and the ones being encoded. iceberg_insert_max_rows_in_data_file caps the file
independently of its size.
iceberg_insert_max_data_files_in_upload
The maximum number of finished data files of one insert stream whose upload is not waited for while theINSERT writes the next
file, for object storages that upload in the background (S3, Azure). Such a file holds its upload buffers until its upload
completes, so higher values hide more upload latency at the cost of memory. 0 waits for the upload of each data file before
writing the next one.
iceberg_insert_max_partitions
Max allowed partitions count per one insert operation for Iceberg table engine.iceberg_insert_max_rows_in_data_file
Max rows of iceberg parquet data file on insert operation.iceberg_insert_max_sort_buffer_bytes
Max uncompressed bytes anINSERT into an Iceberg table with a sort order buffers across all partitions before
writing them sorted. Past it, the largest buffer is written early as a smaller sorted data file. 0 means unlimited.
iceberg_insert_single_manifest_for_all_partitions
AnINSERT into a partitioned Iceberg table writes one manifest file for all the partitions it touches, like Spark
and PyIceberg, instead of one manifest file per partition. This holds per insert stream: every stream commits its own
snapshot with its own manifest, so an INSERT writes exactly one manifest only with max_insert_threads = 1.
OPTIMIZE TABLE ... MANIFEST then also merges manifests of different partitions into one.
ALTER TABLE ... DROP PARTITION cannot drop a partition whose manifest is shared with other partitions.