Avro vs Parquet

Choose Apache Avro when your application exchanges or replays complete records under a producer-consumer schema contract. Choose Apache Parquet when you store files for analytical queries that repeatedly read selected columns. The main difference is how data is organized and consumed. Performance depends on the reader and workload.

Avro vs Parquet at a glance #

For a Kafka consumer processing one event at a time, Avro addresses the record contract. For a SQL query aggregating two columns from a wide file, Parquet's layout addresses the read pattern. If your job consumes every field, measure that job before deciding on performance. The Avro specification and Parquet file layout explain these different units of storage.

What is Avro? #

Apache Avro is a schema-based serialization framework. Its binary encoding represents values according to a writer's schema, without repeating field names in each record. It supports individual encoded records and object container files, so the packaging matters when comparing event delivery with file storage.

An Avro object container file stores its schema in a header, followed by blocks containing multiple records. Blocks can be compressed and have synchronization markers.

An Avro message uses the packaging required by the messaging system. For example, Confluent's schema-registry serialization can carry a schema identifier instead of embedding a complete schema in each message. That envelope and registry compatibility policy belong to the integration. Confirm what your producer emits and what the consumer expects.

What is Parquet? #

Apache Parquet is a column-oriented file format for analytical data. Within each row group, values for each column are stored in column chunks. Footer metadata identifies their locations. This lets a capable reader fetch the columns a query needs while avoiding unrelated column data, rather than assembling every complete record.

Parquet also represents nested structures; nested data alone is not a reason to choose Avro. Definition and repetition levels preserve the structure around column values. Readers reconstruct objects when an application requests them, which has its own cost.

The Parquet layout trades buffering and file organization at write time for opportunities at read time. Row-group size, page size and dictionary choices affect memory, compression and read efficiency. A file with thousands of tiny row groups behaves differently from one designed for larger scans.

Why selected-column reads differ #

Parquet can avoid reading unrelated column chunks. Avro can avoid constructing unused field values, but that does not give it the same on-disk separation. In a compressed Avro container block, the reader still encounters a stream of interleaved records and must process the block to reach the required fields.

Consider an event with event_id, event_time, account_id, amount_cents, and a large payload. A query summing amount_cents by account_id does not need the payload. Parquet can make those columns independently addressable. An Avro reader can skip the payload value, reducing work, while still traversing the record layout.

Column projection selects columns. Predicate pruning uses statistics or indexes to eliminate groups or pages that cannot match a filter. Parquet's page index is optional, and readers differ in how they use it. Sorting data can make statistics more useful; a shuffled file can defeat that advantage. Evaluate projection and pruning separately.

How schema evolution actually works #

Avro's schema evolution follows rules for resolving a writer's schema against a reader's schema. Parquet files carry schemas too, but reading a collection of differently shaped files requires a defined mapping policy. A table format can supply that policy and track changes across files. These are separate responsibilities.

Suppose old Avro records contain name and age, and a new reader also expects email.

A default helps read older data; it does not make an arbitrary producer payload valid. Backward compatibility and forward compatibility also ask different questions: can a new consumer read old data, and can an old consumer read new data? Test the directions your deployment and replay policy require. A registry can enforce a chosen policy, but the policy still needs to match the application.

For Parquet, a newly added field in a dataset might be filled with nulls for older files by one reader, while another reader requires explicit mapping. Renames are particularly important: matching by field name and matching by a stable field ID can produce different behavior. Iceberg's schema evolution uses field IDs to manage changes at the table layer. Independent .parquet files still need a policy for mapping schema changes.

Compression, memory and the reader matter #

Parquet combines column encodings with compression, while Avro container codecs compress record blocks. Repeated categories, sorted values, random identifiers and large opaque payloads respond differently to those choices. Compare the same records, row order and codec before attributing a size difference to row versus column storage.

Even then, a matched codec is only one control. Dictionary encoding and block/page sizes remain part of each format's layout. A smaller file can require more CPU to decode; a faster reader can use more memory or parallelism. Read latency, CPU time, peak memory and file size answer different operational questions.

A reproducible file-reading example #

In this file-reading example, Parquet returned far fewer target-file bytes when selecting two columns. Avro used less CPU and memory than its all-column read, while its traced file-read volume stayed unchanged.

We used ClickHouse 26.8.32.16 to write the same 500,000 synthetic events and 20 fields to both formats. We read event_time and latency_ms, then all 20 fields, directly with file() and FORMAT Null. Full-record and selected-column RowBinary hashes matched the generator in separate correctness checks.

These are target-file pread64 bytes from one separate untimed trace per case, including metadata and read buffering. The all-column Parquet trace covered only part of the 61,650,454-byte file; it does not show why the remaining ranges were skipped. The Avro file was 70,484,749 bytes.

Supporting timings used seven interleaved repetitions per case in an ARM64 OrbStack container capped at two CPUs and 3 GiB RAM. Every run started a fresh local process with warm OS page cache. End-to-end wall time includes process startup and shutdown; query elapsed is the engine's printed time. CPU is total user plus system time; RSS is peak process memory. The table shows medians.

Each format reused one file for both projections. Both files used Zstd, but their compression levels may differ. The results therefore reflect the reader and compression settings as well as format layout. We used common pread reads, max_threads=1 and disabled parallel format parsing; helper concurrency still differs.

The timed runs reported zero guest block-read bytes. These results describe cached file reading with one engine and workload; they do not measure cold-storage or object-store latency. The traced file-byte totals count cached reads, not physical storage I/O. Parquet's all-column read also used more process memory in this setup.

You or an agent can use clickhousectl to set up the example, run it and validate the results.

What changed in the ecosystem? #

A smaller JVM dependency footprint is now an option. Gunnar Morling designated Hardwood 1.0 production-ready on June 25, 2026. Its Java 21+ Parquet reader offers row and column APIs without mandatory dependencies; Snappy, Zstd, LZ4 and Brotli need codec libraries. This gives JVM applications another way to consume Parquet without adopting a large dependency stack. Writer support is in the 1.1 beta line, including October 12's Beta2.

Parquet has a typed representation for heterogeneous values. Variant was introduced in August 2025 and explained by Apache in February 2026. Optional shredding extracts selected paths into typed columns while retaining other values. For logs or event attributes, this broadens the analytical-file option. Check the exact reader's support and full-object reconstruction behavior.

Avro upgrades need language-specific regression checks. Avro 1.12.2, released August 12, 2026, added allocation/decompression bounds and cross-language correctness repairs, including Python enum defaults and Java timestamp conversion. Test old records against the SDK version you plan to deploy, especially if you depend on enum defaults or timestamp conversion.

When to choose Avro #

Avro fits an event processor consuming complete records or an archive replaying those records. Validate compatibility across the producer and consumer versions you need to support, including older schemas retained in the archive.

• Your consumer handles events individually and benefits from the existing registry integration.
• Replay needs the original record structure and writer schema.
• Most reads need complete records, and your measured SDK/workload meets the resource budget.

Our Python Avro reading guide demonstrates file inspection and record access.

When to choose Parquet #

Parquet fits an object-store dataset queried for a few measures and dimensions. Verify logical types and layout with each engine that will read it, especially when enabling newer features.

• Queries repeatedly select a subset of a wider schema.
• Batching records into files fits your freshness requirements.
• Your writer can produce useful row groups, and your reader uses the relevant metadata.

If repeated selected-column queries are the main requirement, choose Parquet and validate the layout against those queries. Our Avro-to-Parquet conversion guide covers converting an existing archive and checking the result. If delivery and complete-record replay are the main requirement, choose Avro and test the consumer contract.