--- title: "How to read an Avro file in Python" date: "2026-06-15" author: "Al Brown" category: "Engineering" excerpt: "Read an Avro file into a DataFrame with chDB and work with it using the pandas API you already know. The embedded schema is read for you, with no struct declarations and no reader loop." --- # How to read an Avro file in Python To read an Avro file in Python, use [chDB](/resources/engineering/what-is-chdb) to load it into a DataFrame and work with it using the pandas API you already know: ```python from chdb.datastore import DataStore df = DataStore.from_file("data/events.avro", format="Avro") ``` `df` supports the pandas API you already use (indexing, filtering, `groupby`, `merge`), but the work runs on ClickHouse's engine instead of pandas, so it stays fast as files grow. There is no server to start and no separate load step. ```bash pip install chdb ``` ## Read an Avro file into a DataFrame `DataStore.from_file` reads the schema from the Avro file header and infers every column's type automatically. Point it at a path and print the result: ```python from chdb.datastore import DataStore df = DataStore.from_file("data/events.avro", format="Avro") print(df) print(df.dtypes) ``` ```text event_id country event_type amount user 0 0 GB purchase 50.98 [100, gold] 1 1 AU view 30.73 [101, silver] 2 2 IN purchase 81.54 [102, bronze] 3 3 GB view 28.92 [103, gold] 4 4 AU purchase 4.15 [104, silver] event_id int64 country str event_type str amount float64 user object dtype: object ``` chDB read the types from the Avro schema embedded in the file header: `event_id` is an integer, `amount` is a float, and the nested `user` record (an Avro record-within-a-record) lands as a struct-valued column. You did not declare a single type. `df` is not the whole file loaded into memory. It is a lazy, ClickHouse-backed object that records what you asked for and runs it on ClickHouse's engine when you materialize a result, whether you print it, take `len()`, or call `.to_pandas()`. That is why it stays fast on large files: it only computes the rows and columns you actually use. ## Filter and aggregate the way you already do The pandas you write does not change. Filter with a boolean mask, group, and aggregate: ```python from chdb.datastore import DataStore df = DataStore.from_file("data/events.avro", format="Avro") revenue = df[df["event_type"] == "purchase"].groupby("country")["amount"].sum() print(revenue) ``` ```text country AU 4.15 GB 50.98 IN 81.54 Name: amount, dtype: float64 ``` Same syntax as pandas, same result. chDB compiles the whole chain into one optimized query and only scans the columns the aggregation touches, rather than materializing the full file first. Joins, computed columns with `assign`, the `.str` and `.dt` accessors, and `merge` across files all work the same way. ## Hand off to real pandas when you need it When a library downstream needs an actual pandas DataFrame (scikit-learn, a plotting call, anything that mutates in place), call `.to_pandas()` to materialize one: ```python df = DataStore.from_file("data/events.avro", format="Avro") pdf = df.to_pandas() # a real pandas.DataFrame, in memory ``` The usual pattern is to do the heavy filtering and aggregation on the DataStore object, where it is fast, then `.to_pandas()` the small result and continue in ordinary pandas. ## Is it faster than a fastavro reader loop? On anything sizeable, yes. The standard Python way to read Avro is [fastavro](https://github.com/fastavro/fastavro). For a filter-and-aggregate, you open the file, iterate every record, reach into nested keys by hand, and accumulate into a dict. Every record is decoded into a Python dict before you can touch it, and every filter and group key is code you write and maintain. chDB decodes in compiled ClickHouse code and never builds a Python object per row. The same aggregation on a 3M-row Avro file (~19 MB compressed) on an Apple M4 Pro (14 cores, 24 GB RAM, macOS); chDB 4.1.8, Python 3.14; best-of-3, warm: ```text fastavro (manual loop): 1.642s import chdb.datastore as pd: 0.288s speedup: 5.7x ``` About 5.7x here. The back-to-back ratio is the robust claim; absolute times shift with cache state and concurrent load. The gap widens as files grow, because the fastavro loop must decode every record into a Python dict before it can filter anything, while chDB streams and aggregates in one pass. The small case can cut the other way: on a tiny Avro file (a few thousand rows) fastavro is competitive, because chDB pays a fixed per-query startup cost that dominates when there is barely any data. Reach for chDB when the file is large. ## The codec note Avro compresses its data blocks. ClickHouse writes Avro with `snappy` by default. `fastavro` can read `snappy` only if you also `pip install cramjam`; without it you get a `snappy codec is supported but you need to install one of the following libraries: ('cramjam',)` error. Writing with the `deflate` codec sidesteps that, because `deflate` decodes with the Python standard library. chDB reads `snappy`, `deflate`, and uncompressed Avro out of the box; the codec only matters for the other reader. ## Works in Jupyter Printing a DataStore object renders it as a table in a notebook, and `.to_pandas()` feeds straight into `.plot()`, joins, or any pandas code. The companion folder ships a `run.ipynb` you can open and run cell by cell. ## Run it yourself The complete, runnable example is here, with `generate.sh` to create the sample Avro files plus `run.ipynb` and a `run.py` mirror containing the exact code above: [github.com/ClickHouse/examples/tree/main/local-analytics/chdb-avro](https://github.com/ClickHouse/examples/tree/main/local-analytics/chdb-avro) New to the format? See [what is an Avro file](/resources/engineering/what-is-an-avro-file). Working with other formats? [Read a Parquet file in Python](/resources/engineering/read-parquet-file-python) and [read a JSON file in Python](/resources/engineering/read-json-file-python) use the same `import chdb.datastore as pd` pattern; only the reader changes.