48.5k
To read an NPY file in Python, use chDB to load the array into a DataFrame and work with it using the pandas API you already know:
from chdb.datastore import DataStore
df = DataStore.from_file("data/readings.npy", format="Npy")df supports the pandas API you already use (indexing, filtering, groupby, merge), but the work runs on ClickHouse's engine instead of pandas, so it stays fast as files grow. There is no server to start and no separate load step.
pip install chdbRead a .npy file into a DataFrame
DataStore.from_file reads the array into a lazy, ClickHouse-backed object. Point it at the file with format="Npy" and print the result:
from chdb.datastore import DataStore
df = DataStore.from_file("data/readings.npy", format="Npy")
print(df)
print(df.dtypes)array
0 50.00
1 63.09
2 74.73
3 83.66
4 88.88
5 89.82
6 86.37
7 78.92
array float64
dtype: objectOne thing is worth knowing about the shape: a .npy file stores exactly one numeric array and carries no column names, so chDB exposes a single column called array. The dtype is inferred from the array's element type: float64 here because the file holds 64-bit floats.
Like the CSV and JSON readers, df is not the whole array loaded into memory. It is a lazy object that runs on ClickHouse's engine when you materialize a result by printing, calling len(), or invoking .to_pandas().
Filter and aggregate the way you already do
Use the pandas API you already know to aggregate the column:
from chdb.datastore import DataStore
df = DataStore.from_file("data/readings.npy", format="Npy")
mean_val = df["array"].mean()
print(f"mean: {float(mean_val):.2f}")
max_val = df["array"].max()
min_val = df["array"].min()
print(f"max: {float(max_val):.2f} min: {float(min_val):.2f}")mean: 76.93
max: 89.82 min: 50.00The same approach works for boolean indexing after you materialize to real pandas with .to_pandas():
pdf = df.to_pandas()
good = pdf[pdf["array"] > 75]
print(f"readings above 75: {len(good)}")readings above 75: 5A 2-D .npy reads as one row per outer element
A .npy can hold a matrix as well as a flat array. Save one with np.save("matrix.npy", np.arange(12).reshape(4, 3)) and chDB reads each outer element as a row in the array column:
mat = DataStore.from_file("data/matrix.npy", format="Npy")
print(mat)array
0 [0, 1, 2]
1 [3, 4, 5]
2 [6, 7, 8]
3 [9, 10, 11]Each element in the array column is a list. Call .to_pandas() to hand it off to pandas or NumPy for further processing.
Join two arrays by position
NumPy workflows often keep parallel arrays in separate files: readings in one .npy, a quality flag in another, aligned by position. Load each file, hand both off to real pandas, and join on the index:
readings_pdf = (
DataStore.from_file("data/readings.npy", format="Npy")
.to_pandas()
.rename(columns={"array": "reading"})
)
flags_pdf = (
DataStore.from_file("data/flags.npy", format="Npy")
.to_pandas()
.rename(columns={"array": "ok"})
)
combined = readings_pdf.join(flags_pdf)
print(combined)reading ok
0 50.00 0
1 63.09 1
2 74.73 1
3 83.66 1
4 88.88 0
5 89.82 1
6 86.37 1
7 78.92 1From there, filter to only the rows where ok == 1 and compute the mean in pandas:
mean_ok = combined[combined["ok"] == 1]["reading"].mean()
print(f"mean of flagged readings: {mean_ok:.2f}")Hand off to real pandas when you need it
When a library downstream needs an actual pandas DataFrame (scikit-learn, a plotting call, anything that mutates in place), call .to_pandas() to materialize one:
df = DataStore.from_file("data/readings.npy", format="Npy")
pdf = df.to_pandas() # a real pandas.DataFrame, in memory<class 'pandas.DataFrame'>The usual pattern is to do the heavy filtering and aggregation on the chDB object, where it is fast on large files, then .to_pandas() the small result and continue in ordinary pandas.
Works in Jupyter
Printing a chDB DataStore object renders it as a table in a notebook, and .to_pandas() feeds straight into .plot(), joins, or any pandas code. The companion folder ships a run.ipynb you can open and run cell by cell.
Run it yourself
The complete, runnable example is here, with generate.sh to create the sample .npy files plus run.ipynb and a run.py mirror containing the exact code above:
github.com/ClickHouse/examples/tree/main/local-analytics/chdb-npy
Working with other formats? Read a JSON file in Python and read a Parquet file in Python use the same chDB datastore pattern; only the reader changes. The same code scales from a file on your laptop to ClickHouse Cloud with no rewrite.
