An Avro file is a row-oriented, binary data format that stores its own schema, written as JSON, in the file header. Each file is self-describing: the schema travels with the data, so any reader can decode the records without an external definition. Avro was built for schema evolution, which makes it a common choice for streaming and data-pipeline payloads where the shape of a record changes over time.
| Property | Avro |
|---|
| Layout | Row-oriented (records stored whole, one after another) |
| Schema | Embedded in the file header as JSON (self-describing) |
| Compression | Per-block: deflate, snappy, zstd (codec recorded in the header) |
| Schema evolution | First-class: add, remove or rename fields with defaults/aliases |
| Typical use | Streaming, message queues, data-pipeline records, RPC |
| Open standard | Apache Avro; readable by most data tools |
An Avro object container file has three parts:
- Header: the four magic bytes
Obj\x01, then file metadata as key/value pairs. Two keys matter: avro.schema (the full record schema as JSON) and avro.codec (the block compression codec). A 16-byte sync marker closes the header.
- Data blocks: the records, grouped into blocks. Each block stores its row count, its compressed size, the encoded rows, then the sync marker. Within a block, every record is serialised field-by-field in schema order.
- Sync markers: the random 16-byte value repeated between blocks. A reader that seeks into the middle of a large file scans forward to the next marker to find a block boundary, which makes Avro splittable for parallel processing.
The defining trait is the embedded schema. Describing it is one thing; reading it out of a real file is more convincing. So let's write an Avro file, then crack open its header.
clickhouse local can do both. It's part of ClickHouse; install it with clickhousectl, the ClickHouse CLI for local and cloud:
Write a small event file, events_v1.avro:
Because the schema is embedded, you can read the file straight back with no structure argument. DESCRIBE reads the header and reports the columns it found:
That column list came from the JSON schema sitting in the file header. We can pull it out verbatim. Read the raw bytes, match the avro.schema record JSON with a regular expression, and pretty-print it:
This is the actual definition stored in the file, not an inference. The Avro types (long, int, string, double) are what ClickHouse mapped to Int64, Int32, String and Float64 in the DESCRIBE above. Every Avro file carries this; it is what makes the format self-describing.
Compression and encoding
Avro compresses per data block, and records the codec in the header under avro.codec (the file above uses snappy by default). Because compression is applied to whole rows rather than to columns of one type, an Avro file typically does not pack as tightly as a columnar format. The trade is that a row-oriented layout is cheap to append to and cheap to read one whole record at a time, which suits streaming producers and consumers.
Querying the file is plain SQL. Read it directly and aggregate:
On a larger 3,000,000-row file (42 MB on disk), scanning and aggregating by country runs in 0.270s (best of three, warm cache, Apple M4 Pro, 24 GB RAM, macOS; the figure may be nudged by other work on the machine).
A record's shape changes over time. Avro is designed for that: a writer can add a field, and a reader using a newer schema with a default still decodes older records. To see the schema actually change, write a v2 file that adds a channel field, then read its embedded schema:
The header now lists five fields. Each Avro file is read against its own embedded schema, so the new file decodes cleanly with the extra column:
One practical note when reading many Avro files at once: a query that references channel across both v1 and v2 files will fail on the older file, because Avro decodes each file strictly against the schema in its own header. That strictness is the point. Evolution is handled at write time, by giving new fields defaults or aliases, not by guessing at read time.
Avro and Parquet are both binary, self-describing and open, and they are often used together: Avro for the row-by-row pipeline, Parquet for the columnar store the rows land in. The difference is layout. Avro is row-oriented, so it writes and reads whole records cheaply, which suits streaming. Parquet is columnar, so an analytical query reads only the columns it needs and skips the rest, which makes it faster to scan and smaller on disk for the same data. Choose Avro for moving records; choose Parquet for querying them at rest. When you need to move from one to the other, see convert Avro to Parquet.
For other row-oriented binary formats, see what is BSON and what is MessagePack.
You don't need a server or an import step to work with Avro. clickhouse local reads an Avro file in place from the command line, using the schema in the file's own header. ClickHouse uses the same SQL whether you point it at a file on your laptop, a server, or ClickHouse Cloud, so a query you write against a local Avro file scales up unchanged.
For the full walkthrough, covering globbing many files, projecting columns and casting types, see how to read an Avro file.
Run it yourself: the data generator and every command above live in local-analytics/what-is-an-avro-file in the ClickHouse examples repo.
Prefer Python? → Read an Avro file in Python.
