---
title: "What is an Avro file?"
date: "2026-06-15"
author: "Al Brown"
category: "Engineering"
excerpt: "An Avro file is a row-oriented, binary data format that embeds its own JSON schema in the file header, built for schema evolution. This page explains its structure and proves it by reading the embedded schema with clickhouse local."
---

# What is an Avro file?

An **Avro file** is a row-oriented, binary data format that stores its own schema, written as JSON, in the file header. Each file is self-describing: the schema travels with the data, so any reader can decode the records without an external definition. Avro was built for schema evolution, which makes it a common choice for streaming and data-pipeline payloads where the shape of a record changes over time.

## At a glance

| Property | Avro |
|---|---|
| Layout | Row-oriented (records stored whole, one after another) |
| Schema | Embedded in the file header as JSON (self-describing) |
| Compression | Per-block: deflate, snappy, zstd (codec recorded in the header) |
| Schema evolution | First-class: add, remove or rename fields with defaults/aliases |
| Typical use | Streaming, message queues, data-pipeline records, RPC |
| Open standard | Apache Avro; readable by most data tools |

## Internal structure

An Avro object container file has three parts:

- **Header**: the four magic bytes `Obj\x01`, then file metadata as key/value pairs. Two keys matter: `avro.schema` (the full record schema as JSON) and `avro.codec` (the block compression codec). A 16-byte sync marker closes the header.
- **Data blocks**: the records, grouped into blocks. Each block stores its row count, its compressed size, the encoded rows, then the sync marker. Within a block, every record is serialised field-by-field in schema order.
- **Sync markers**: the random 16-byte value repeated between blocks. A reader that seeks into the middle of a large file scans forward to the next marker to find a block boundary, which makes Avro splittable for parallel processing.

The defining trait is the embedded schema. Describing it is one thing; reading it out of a real file is more convincing. So let's write an Avro file, then crack open its header.

[`clickhouse local`](/resources/engineering/what-is-clickhouse-local) can do both. It's part of ClickHouse; install it with `clickhousectl`, the ClickHouse CLI for local and cloud:


```bash
curl https://clickhouse.com/cli | sh   # install clickhousectl
clickhousectl local use latest         # download ClickHouse and put it on your PATH
```

Write a small event file, `events_v1.avro`:


```bash
clickhouse local -q "
SELECT
    number AS id,
    toDateTime('2026-01-01 00:00:00') + toIntervalMinute(number) AS event_time,
    ['GB','AU','IN','US','DE'][(number % 5) + 1] AS country,
    round(randUniform(1, 500), 2) AS revenue
FROM numbers(1000)
INTO OUTFILE 'events_v1.avro'
FORMAT Avro"
```

Because the schema is embedded, you can read the file straight back with no structure argument. `DESCRIBE` reads the header and reports the columns it found:


```bash
clickhouse local -q "DESCRIBE file('events_v1.avro')"
```

```response
id	Int64
event_time	Int32
country	String
revenue	Float64
```

## The embedded schema, read from the header

That column list came from the JSON schema sitting in the file header. We can pull it out verbatim. Read the raw bytes, match the `avro.schema` record JSON with a regular expression, and pretty-print it:


```bash
clickhouse local -q "
SELECT extractGroups(raw, '(\{\"type\":\"record\".*?\}\]\})')[1] AS avro_schema
FROM file('events_v1.avro', RawBLOB, 'raw String')
FORMAT TSVRaw" | python3 -m json.tool
```

```response
{
    "type": "record",
    "name": "row",
    "fields": [
        {
            "name": "id",
            "type": "long"
        },
        {
            "name": "event_time",
            "type": "int"
        },
        {
            "name": "country",
            "type": "string"
        },
        {
            "name": "revenue",
            "type": "double"
        }
    ]
}
```

This is the actual definition stored in the file, not an inference. The Avro types (`long`, `int`, `string`, `double`) are what ClickHouse mapped to `Int64`, `Int32`, `String` and `Float64` in the `DESCRIBE` above. Every Avro file carries this; it is what makes the format self-describing.

## Compression and encoding

Avro compresses per data block, and records the codec in the header under `avro.codec` (the file above uses snappy by default). Because compression is applied to whole rows rather than to columns of one type, an Avro file typically does not pack as tightly as a columnar format. The trade is that a row-oriented layout is cheap to append to and cheap to read one whole record at a time, which suits streaming producers and consumers.

Querying the file is plain SQL. Read it directly and aggregate:


```bash
clickhouse local -q "
SELECT country, count() AS events, round(sum(revenue)) AS revenue
FROM file('events_v1.avro')
GROUP BY country
ORDER BY revenue DESC
FORMAT Pretty"
```

```response
   ┏━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
   ┃ country ┃ events ┃ revenue ┃
   ┡━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
1. │ DE      │    200 │   53729 │
   ├─────────┼────────┼─────────┤
2. │ GB      │    200 │   51090 │
   ├─────────┼────────┼─────────┤
3. │ AU      │    200 │   50394 │
   ├─────────┼────────┼─────────┤
4. │ IN      │    200 │   50181 │
   ├─────────┼────────┼─────────┤
5. │ US      │    200 │   49363 │
   └─────────┴────────┴─────────┘
```

On a larger 3,000,000-row file (42 MB on disk), scanning and aggregating by country runs in 0.270s (best of three, warm cache, Apple M4 Pro, 24 GB RAM, macOS; the figure may be nudged by other work on the machine).

## Schema evolution, the reason Avro exists

A record's shape changes over time. Avro is designed for that: a writer can add a field, and a reader using a newer schema with a default still decodes older records. To see the schema actually change, write a `v2` file that adds a `channel` field, then read its embedded schema:


```bash
clickhouse local -q "
SELECT extractGroups(raw, '(\{\"type\":\"record\".*?\}\]\})')[1] AS avro_schema
FROM file('events_v2.avro', RawBLOB, 'raw String')
FORMAT TSVRaw" | python3 -m json.tool
```

```response
{
    "type": "record",
    "name": "row",
    "fields": [
        {
            "name": "id",
            "type": "long"
        },
        {
            "name": "event_time",
            "type": "int"
        },
        {
            "name": "country",
            "type": "string"
        },
        {
            "name": "revenue",
            "type": "double"
        },
        {
            "name": "channel",
            "type": "string"
        }
    ]
}
```

The header now lists five fields. Each Avro file is read against its own embedded schema, so the new file decodes cleanly with the extra column:


```response
   ┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
   ┃ channel  ┃ events ┃ revenue ┃
   ┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
1. │ organic  │    334 │   83430 │
   ├──────────┼────────┼─────────┤
2. │ paid     │    333 │   85806 │
   ├──────────┼────────┼─────────┤
3. │ referral │    333 │   81942 │
   └──────────┴────────┴─────────┘
```

One practical note when reading many Avro files at once: a query that references `channel` across both `v1` and `v2` files will fail on the older file, because Avro decodes each file strictly against the schema in its own header. That strictness is the point. Evolution is handled at write time, by giving new fields defaults or aliases, not by guessing at read time.

## How it compares to Parquet

Avro and Parquet are both binary, self-describing and open, and they are often used together: Avro for the row-by-row pipeline, Parquet for the columnar store the rows land in. The difference is layout. Avro is row-oriented, so it writes and reads whole records cheaply, which suits streaming. [Parquet](/resources/engineering/what-is-parquet-file) is columnar, so an analytical query reads only the columns it needs and skips the rest, which makes it faster to scan and smaller on disk for the same data. Choose Avro for moving records; choose Parquet for querying them at rest. When you need to move from one to the other, see [convert Avro to Parquet](/resources/engineering/convert-avro-to-parquet).

For other row-oriented binary formats, see [what is BSON](/resources/engineering/what-is-bson) and [what is MessagePack](/resources/engineering/what-is-messagepack).

## Read one yourself

You don't need a server or an import step to work with Avro. [`clickhouse local`](/resources/engineering/what-is-clickhouse-local) reads an Avro file in place from the command line, using the schema in the file's own header. ClickHouse uses the same SQL whether you point it at a file on your laptop, a server, or [ClickHouse Cloud](https://clickhouse.com/cloud), so a query you write against a local Avro file scales up unchanged.

For the full walkthrough, covering globbing many files, projecting columns and casting types, see [how to read an Avro file](/resources/engineering/read-avro-file).

**Run it yourself:** the data generator and every command above live in [`local-analytics/what-is-an-avro-file`](https://github.com/ClickHouse/examples/tree/main/local-analytics/what-is-an-avro-file) in the ClickHouse examples repo.

Prefer Python? → [Read an Avro file in Python](/resources/engineering/read-avro-file-python).