--- title: "How to read an Arrow file in Python (faster than pandas)" date: "2026-06-15" author: "Al Brown" category: "Engineering" excerpt: "Read an Arrow IPC file into a DataFrame with chDB, a drop-in replacement for pandas. Change one import line and your existing pandas code runs on ClickHouse's engine, so it stays fast as files grow." --- # How to read an Arrow file in Python (faster than pandas) To read an Arrow file in Python, use [chDB](/resources/engineering/what-is-chdb) as a drop-in replacement for pandas. Change one import line and the code you already write keeps working: ```python import chdb.datastore as pd df = pd.read_feather("data/events.arrow") # Arrow IPC files are read by the feather reader ``` `df` supports the pandas API you already use (indexing, filtering, `groupby`, `merge`), but the work runs on ClickHouse's engine instead of pandas, so it stays fast as files grow. There is no server to start and no separate load step. ```bash pip install chdb ``` ## Read an Arrow file into a DataFrame Arrow IPC files (`.arrow`) share an on-disk layout with the Feather format, so `read_feather` is the right reader for both. Pass the file path and print the result: ```python import chdb.datastore as pd df = pd.read_feather("data/events.arrow") print(df.head(8)) print(df.dtypes) ``` ```text id country event_type quantity amount event_time 0 0 GB cart 8 350.98 1768110393 1 1 IN view 6 830.73 1772107899 2 2 DE cart 10 981.54 1771606592 3 3 DE view 6 528.92 1772725868 4 4 US cart 5 804.15 1774390759 5 5 GB view 9 363.26 1772159477 6 6 IN purchase 10 23.42 1769334173 7 7 US view 10 747.28 1773645610 id uint64 country object event_type object quantity uint16 amount float64 event_time uint32 dtype: object ``` chDB reads the column names and types from the Arrow schema. `id` comes back as `uint64`, `amount` as `float64`, and `event_time` as `uint32`. The types are already embedded in the file, so there is no guessing or declaration step. `df` is not the whole file loaded into memory. It is a lazy, ClickHouse-backed object that records what you asked for and runs it on ClickHouse's engine when you materialize a result. You trigger that by calling `.head()`, taking `len()`, or calling `.to_pandas()`. That is why it stays fast on large files: it only computes the rows and columns you actually use. ## Filter and aggregate the way you already do The pandas you write does not change. Filter with a boolean mask, group, and aggregate: ```python import chdb.datastore as pd df = pd.read_feather("data/events.arrow") counts = (df[df["event_type"] == "purchase"] .groupby("country")["id"].count() .sort_values(ascending=False)) print(counts) ``` ```text country GB 200304 DE 200026 AU 199997 US 199997 IN 199631 Name: id, dtype: int64 ``` Same syntax as pandas, same result. chDB compiles the whole chain into one optimized query and only scans the columns the aggregation touches, rather than materializing the full file first. Joins, computed columns with `assign`, the `.str` and `.dt` accessors, and `merge` across files all work the same way. ## Hand off to real pandas when you need it When a library downstream needs an actual pandas DataFrame (scikit-learn, a plotting call, anything that mutates in place), call `.to_pandas()` to materialize one: ```python df = pd.read_feather("data/events.arrow") pdf = df.to_pandas() # a real pandas.DataFrame, in memory ``` The usual pattern is to do the heavy filtering and aggregation on the chDB object, where it is fast, then `.to_pandas()` the small result and continue in ordinary pandas. ## Is it faster than pyarrow + pandas? Arrow plays to pyarrow's strengths. It is pyarrow's native on-disk format, so reading it is fast. chDB still edges ahead because it only touches the columns and rows the query needs, while the pyarrow path decodes the whole table into a pandas frame before grouping. The same filter and aggregate above on a 3M-row (~69 MB) Arrow file, best-of-3 with a warm page cache on an Apple M4 Pro (14 cores, 24 GB RAM, macOS); chDB 4.1.8, Python 3.14; best-of-3, warm: ```text pyarrow + pandas: 0.052s import chdb.datastore as pd: 0.036s speedup: 1.5x ``` About 1.5x here, from the same code with one import changed. The back-to-back ratio is the robust claim; absolute times shift with cache state and concurrent load. The gap widens as files grow, because chDB never has to build the full frame just to answer a grouped query. The small case can cut the other way: on a tiny Arrow file (a few thousand rows) pyarrow is as fast or faster, because chDB pays a fixed per-query startup cost that dominates when there is barely any data to scan. Reach for chDB when the file is large, or when you want one import to cover both. ## Works in Jupyter Printing a chDB object renders it as a table in a notebook, and `.to_pandas()` feeds straight into `.plot()`, joins, or any pandas code. The companion folder ships a `run.ipynb` you can open and run cell by cell. ## Run it yourself The complete, runnable example is here, with `generate.sh` to create the sample Arrow file plus `run.ipynb` and a `run.py` mirror containing the exact code above: [github.com/ClickHouse/examples/tree/main/local-analytics/chdb-arrow](https://github.com/ClickHouse/examples/tree/main/local-analytics/chdb-arrow) Working with other formats? [Read a Parquet file in Python](/resources/engineering/read-parquet-file-python) and [read a CSV file in Python](/resources/engineering/read-csv-file-python) use the same drop-in `import chdb.datastore as pd` pattern; only the reader changes. The same code scales from a file on your laptop to a server to [ClickHouse Cloud](https://clickhouse.com/cloud) with no rewrite, so there is nothing to re-learn when the data outgrows one machine.