Data catalog

Data catalogs are essential for modern data architectures, but they're often misunderstood or overlooked. Let's explore what data catalogs are, why they matter for open table formats, and how they enable powerful analytical capabilities.

A data catalog is a centralized repository that stores metadata about an organization's data assets. It serves as an index or inventory system for data, making data assets discoverable, understandable, and manageable.

Fundamentally, a data catalog is an architectural component rather than a standalone product - it's a system that tracks where data and metadata reside without necessarily storing the actual data itself.

In this article, we will focus on data catalogs for open table formats - a specialized type of catalog designed to work with modern analytical data stored in columnar file formats like Apache Parquet and managed through table frameworks such as Apache Iceberg, Delta Lake, or Apache Hudi. These specialized catalogs address the unique requirements of data lakehouse architectures, where performance, scalability, and transactional capabilities are paramount.

Unlike general-purpose data catalogs that might cover a wide range of data assets (from traditional databases to unstructured documents), open table format catalogs are optimized for tracking table states, schema versions, and optimization metadata specifically for analytical data workloads. They function as the critical lookup system for query engines to locate and process data stored across distributed storage systems efficiently. They provide the technical metadata for operations like partition pruning, predicate pushdown, and time travel queries.

These catalogs enable multiple processing engines to consistently access the same datasets with transactional guarantees, even as the underlying data evolves through updates, schema changes, and optimization operations.

Now we know what a data catalog is, let’s learn about some of its characteristics.

Its function as a metadata repository is at the core of any data catalog. For open table formats, this repository stores specialized metadata about table assets, including their current version, schema definitions, and file locations, without containing the data itself. The metadata is typically structured hierarchically, organized into elements such as databases or schemas, tables, and columns. In open table catalogs, this structure often includes additional technical metadata like partition specifications, file manifests, and statistics that enable efficient query planning. This structure provides a single source of truth about all table assets within an organization, allowing query engines to understand what data exists and how to access it efficiently.

Data catalogs for open table formats provide mechanisms for data discovery and classification tailored to analytical workloads. While they may include automated discovery tools that scan data sources, these catalogs often work with explicit table registrations rather than general-purpose crawlers. In open table ecosystems, the focus shifts from general classification to tracking specific table properties like format version, optimization status, and transaction history. Search functionality still allows users to find relevant tables across the organization, but with additional technical context specific to analytical processing needs.

Schema management is particularly critical for open table format catalogs. They track current schemas and maintain a history of schema evolution, enabling time travel queries and ensuring backward compatibility. Open table catalogs specialize in handling the complex schema evolution patterns supported by formats like Iceberg, Delta Lake, and Hudi, including column additions, removals, and type changes. While these catalogs primarily focus on structured data in columnar formats like Parquet and ORC, they provide sophisticated versioning capabilities that ensure consistent data access even as schemas evolve.

Security remains paramount in open table format catalogs, which implement access control mechanisms for analytical workloads. They define and enforce permission models that determine who can access specific tables and operations, often integrating with data governance frameworks like Apache Ranger or AWS Lake Formation. For open table formats, these controls may extend to specific operations like compaction, schema evolution, or time travel queries. These integrate with organizational identity systems to ensure consistent security policies across the data lakehouse ecosystem.

Open table format catalogs maintain specialized lineage information, tracking data origins and the sequence of transactions and operations performed on tables. This capability is essential for features like time travel, where users need to access specific historical versions of data. For open table formats, governance features focus on ensuring data integrity, tracking optimization operations, and maintaining audit trails of schema and data changes. This technical lineage information complements broader data governance initiatives by providing a detailed operational history.

A key strength of open table format catalogs is their ability to integrate with diverse query engines and processing frameworks. They provide consistent interfaces that allow engines like Spark, Flink, Trino, and ClickHouse to access the same table definitions and metadata. This engine-agnostic approach ensures that different tools can work with the same data without inconsistencies. Open table catalogs typically offer standardized APIs based on specifications like the Iceberg API or Delta Protocol, enabling interoperability across the modern data stack. This integration capability is essential for building flexible data architectures that avoid vendor lock-in.

Several data catalog implementations have emerged to support open table formats, each with different strengths, integration capabilities, and deployment models. Below, we describe some of the most common data catalogs.

The Apache Hive Metastore is the original and most widely supported catalog for big data ecosystems. Despite being developed before modern open table formats, it remains a common choice due to its universal compatibility with data processing engines. However, it lacks native features specifically designed for the advanced capabilities of open table formats.

AWS Glue Data Catalog is a fully managed, serverless catalog service that has added native support for open table formats. It integrates deeply with AWS services like Athena, EMR, and Redshift, providing a zero-maintenance option for organizations already operating in the AWS ecosystem.

Unity Catalog is Databricks' metadata management solution designed primarily for Delta Lake but expanding to support other formats. It offers unified governance across data, AI models, and notebooks with multi-cloud support spanning AWS, Azure, and GCP.

Project Nessie is an open-source catalog specifically designed for open table formats. It focuses on Git-like versioning semantics and enables Git-inspired operations like branches and commits for data, making it particularly suitable for collaborative data engineering workflows.

ClickHouse offers seamless integration with popular data catalogs, allowing you to directly leverage existing metadata repositories and query open table formats. This capability transforms ClickHouse from a standalone analytical database into a powerful query engine that can work with your existing data lake infrastructure.

ClickHouse can connect directly to Unity Catalog, enabling you to query Delta Lake and Iceberg tables without data duplication. For Delta Lake tables, the configuration is straightforward:

1CREATE DATABASE unity
2ENGINE = DataLakeCatalog('https://.cloud.databricks.com/api/2.1/unity-catalog')
3SETTINGS   
4  warehouse = 'CATALOG_NAME',   
5  catalog_credential = '',   
6  catalog_type = 'unity';

ClickHouse also supports Iceberg tables through Unity Catalog's REST interface, offering more flexibility with OAuth authentication:

1CREATE DATABASE unity
2ENGINE = DataLakeCatalog('https://.cloud.databricks.com/api/2.1/unity-catalog/iceberg')
3SETTINGS   
4  catalog_type = 'rest',   
5  catalog_credential = ':',   
6  warehouse = 'workspace', 
7  oauth_server_uri = 'https://.cloud.databricks.com/oidc/v1/token',   
8  auth_scope = 'all-apis,sql';

Once configured, you can query the underlying tables using standard SQL syntax, just as you would with native ClickHouse tables. The catalog abstraction makes the experience seamless:

1SELECT count(*)
2FROM `uniform.delta_hits`;

Behind the scenes, ClickHouse leverages Unity Catalog's metadata to locate the data files, optimize query planning, and apply predicate pushdown where possible, delivering impressive performance even on large datasets.

ClickHouse integrates with the AWS Glue Catalog to provide direct access to tables defined in your AWS environment. Setting up the connection requires AWS credentials and region information:

1CREATE DATABASE glue
2ENGINE = DataLakeCatalog
3SETTINGS
4   catalog_type = 'glue',
5   region = 'us-west-2',
6   aws_access_key_id = '',
7   aws_secret_access_key = '';

You can also use IAM roles instead of hardcoded credentials for enhanced security in production environments, particularly when running ClickHouse on EC2 instances or EKS clusters with appropriate IAM permissions.

After establishing the connection, you can query Iceberg tables registered in the Glue Catalog:

1SELECT count(*) FROM `iceberg-benchmark.hitsiceberg`;

This integration is compelling for AWS-centric architectures, allowing you to maintain a single source of truth in the Glue Catalog while enabling ClickHouse to serve as a high-performance query engine for interactive analytics.