system.iceberg_metadata_log
The system.iceberg_metadata_log
table records metadata access and parsing events for Iceberg tables read by ClickHouse. It provides detailed information about each metadata file or entry processed, which is useful for debugging, auditing, and understanding Iceberg table structure evolution.
Purpose
This table logs every metadata file and entry read from Iceberg tables, including root metadata files, manifest lists, and manifest entries. It helps users trace how ClickHouse interprets Iceberg table metadata and diagnose issues related to schema evolution, file resolution, or query planning.
This table is primarily intended for debugging purposes.
Columns
Name | Type | Description |
---|---|---|
event_date | Date | Date of the log entry. |
event_time | DateTime | Timestamp of the event. |
query_id | String | Query ID that triggered the metadata read. |
content_type | Enum8 | Type of metadata content (see below). |
table_path | String | Path to the Iceberg table. |
file_path | String | Path to the root metadata JSON file, Avro manifest list, or manifest file. |
content | String | Content in JSON format (raw metadata from .json, Avro metadata, or Avro entry). |
row_in_file | Nullable(UInt64) | Row number in the file, if applicable. Present for ManifestListEntry and ManifestFileEntry content types. |
content_type
values
None
: No content.Metadata
: Root metadata file.ManifestListMetadata
: Manifest list metadata.ManifestListEntry
: Entry in a manifest list.ManifestFileMetadata
: Manifest file metadata.ManifestFileEntry
: Entry in a manifest file.
The data in this system table is held locally on each node in ClickHouse Cloud. Obtaining a complete view of all data, therefore, requires the clusterAllReplicas
function. See here for further details.
Controlling log verbosity
You can control which metadata events are logged using the iceberg_metadata_log_level
setting.
To log all metadata used in the current query:
To log only the root metadata JSON file used in the current query:
See more information in the description of the iceberg_metadata_log_level
setting.
Good To Know
- Use
iceberg_metadata_log_level
at the query level only when you need to investigate your Iceberg table in detail. Otherwise, you may populate the log table with excessive metadata and experience performance degradation. - The table may contain duplicate entries, as it is intended primarily for debugging and does not guarantee uniqueness per entity.
- If you use a
content_type
more verbose thanManifestListMetadata
, the Iceberg metadata cache is disabled for manifest lists. - Similarly, if you use a
content_type
more verbose thanManifestFileMetadata
, the Iceberg metadata cache is disabled for manifest files.