Engineering Resources / Avro vs Parquet

Avro vs Parquet

The emergence of Apache Hadoop in the mid-2000s marked the beginning of the big data era, fundamentally changing how organizations store and process massive datasets. This transformation created a need for specialized file formats designed to handle data analysis and cross-system data transfer at scale efficiently.

Two notable file formats emerged from this evolution: Apache Avro (2009) and Apache Parquet (2013). While both formats serve essential roles in the big data ecosystem, each brings distinct advantages to different use cases.

In this article, we'll explore the key differences between Avro and Parquet, helping you understand when and why to use each format. We'll examine their strengths, limitations, and ideal applications to help you make informed decisions for your data architecture.

What is Avro?

Apache Avro is a row-oriented data serialization framework that emerged from the Apache Hadoop project. It provides a compact, fast, and efficient way to handle data serialization, making it particularly valuable for data streaming and message-based systems.

Avro’s key features

  • Schema-based serialization system
  • Rich schema definition and evolution capabilities
  • Compact binary data format
  • Integration with schema registries
  • Built-in support for data compression

One of Avro's standout features is its sophisticated schema management. While the schema must be present during data writing and reading, organizations can leverage schema registries to optimize this process. Instead of transmitting the complete schema with each data exchange, systems can simply reference a schema ID, significantly reducing overhead and improving performance.

Avro's architecture and use cases

  • Primarily used in streaming data scenarios
  • Strong integration with Apache Kafka for message publishing
  • Optimized for wire transfer of data
  • Typically handles smaller data units, often one record per file
  • Ideal for real-time data processing and messaging systems

The format gained prominence in the streaming data community due to its efficient serialization capabilities and schema evolution features. Its design makes it particularly well-suited for systems where data structures may need to evolve without breaking existing consumers.

When should I use Avro?

Apache Avro excels in data transfer scenarios, particularly with streaming data pipelines and message-based systems. Its design makes it ideal for several data streaming and messaging systems.

It has the following advantages compared to JSON for data transfer:

  1. Schema enforcement
    1. Strict schema definition ensures data consistency
    2. Clear contract between producers and consumers
    3. Reduces data quality issues and runtime errors
  2. Schema evolution
    1. Supports forward and backward compatibility
    2. Allows systems to evolve without breaking existing applications
    3. Clean handling of schema changes over time
  3. Performance benefits
    1. More compact binary format than JSON
    2. Reduced network bandwidth usage
    3. Faster serialization and deserialization
    4. Lower storage requirements

By choosing Avro over text-based formats like JSON, organizations can ensure better data governance, improved performance, and more reliable data integration across their systems. Combining schema validation and efficient binary serialization makes Avro particularly valuable in enterprise-scale data operations, where data consistency and performance are crucial.

Working with Avro in Python

Let’s look at how to work with Avro in Python. fastavro is a popular library for doing this. We're going to use the faker library to create a 1 million row Avro file:

1pip install fastavro faker
1import fastavro
2from faker import Faker
3
4fake = Faker()
5
6schema = {
7    "type": "record",
8    "name": "User",
9    "fields": [
10        {"name": "name", "type": "string"},
11        {"name": "age", "type": "int"},
12        {"name": "email", "type": "string"}
13    ]
14}
15
16records = [
17  {
18    "name": fake.name(),
19    "age": fake.random_int(min=18, max=80),
20    "email": fake.email()
21  }
22  for _ in range(1_000_000)
23]
24
25with open('users.avro', 'wb') as out:
26    fastavro.writer(out, schema, records)

When reading from this file, we need to first install the cramjam library:

1pip install cramjam
1with open('users.avro', 'rb') as file:
2    reader = fastavro.reader(file)
3    for record in reader:
4        print(record)
{'key': '_HYtg3FZj:77pUB1!x:?0', 'number': 55093572, 'value1': 70, 'value2': 199, 'value3': 10.167093690048159}
{'key': 'lSRL3Tk_1ZcY*F-3#\\4?', 'number': 55093573, 'value1': 29, 'value2': 193, 'value3': 27.204761505913336}
^C{'key': '~_=66N0);]JxeF>P74j', 'number': 55093577, 'value1': 93, 'value2': 198, 'value3': 5.050937769552736}
{'key': '\'<;2zu_>c"', 'number': 55093578, 'value1': 94, 'value2': 189, 'value3': 6.70526603589251}
{'key': '*Jl0ur=rJRDNe', 'number': 55093579, 'value1': 74, 'value2': 200, 'value3': 26.631595740252777}
{'key': "gX:DBO.'g+0g", 'number': 55093580, 'value1': 88, 'value2': 210, 'value3': 2.3440855043518796}

What is Parquet?

Apache Parquet is a columnar storage format that has emerged as the industry standard for data lake storage. This sophisticated format revolutionized big data analytics by optimizing how data is stored and queried at scale.

Parquet’s key features

  • Column-oriented storage format
  • Optimized for large-scale data processing
  • Commonly deployed on cloud object stores (Amazon S3, Google Cloud Storage)
  • Foundation for modern data lake implementations
  • Underlying format for Apache Iceberg and other table formats

One of Parquet's standout features is its intelligent data organization. The format uses row groups, column chunks, and metadata filtering to enable efficient querying of large datasets. With its sophisticated compression algorithms tailored to specific data types, Parquet significantly reduces storage costs while maintaining high query performance.

Parquet's architecture and use cases

  • The primary choice for data lake storage, especially on cloud platforms (S3, GCS)
  • Fundamental storage layer for modern table formats like Apache Iceberg
  • Optimized for analytical queries and big data processing
  • Ideal for data warehousing and business intelligence applications
  • Commonly used with query engines like Apache Spark, Presto, and Athena

The format gained prominence during the data lake movement due to its ability to handle large-scale analytical workloads efficiently. Its design makes it particularly well-suited for systems where query performance and storage optimization are critical requirements.

When should I use Parquet?

Apache Parquet is the ideal choice when working with large-scale analytical data that can be broken down into distinct fields. It excels in a couple of key scenarios:

Data Storage and Analytics

  • Large-scale data warehousing and data lakes
  • Long-term data storage ("data at rest")
  • Business intelligence and analytical processing
  • Complex queries requiring column-specific access

ETL and Batch Processing

  • Batch processing of records in ETL workflows
  • Aggregation of multiple records into single Parquet files
  • Kafka Connect used for time-based or record-count-based aggregation of streaming data

The format's columnar structure and compression capabilities make it particularly efficient for organizations that need to store and analyze large datasets cost-effectively.

Working with Parquet in Python

When working with the Parquet file format in Python, we can use the popular pandas library with either fastparquet or pyarrow. We can write the following code to create a file with 1 million rows:

1pip install pandas fastparquet
1import pandas as pd
2from faker import Faker
3
4fake = Faker()
5
6records = [
7  {
8    "name": fake.name(),
9    "age": fake.random_int(min=18, max=80),
10    "email": fake.email()
11  }
12  for _ in range(1_000_000)
13]
14
15df = pd.DataFrame(records)df.to_parquet('users.parquet', index=False)

And then to query the file:

1import pandas as pd
2
3df = pd.read_parquet('users.parquet')
4print(df.head())
                        key    number  value1  value2     value3
0      g1.NHQ(0%-JrY*}jxbCS  46752066      39     196  19.823986
1                jIY(eL!_Hu  46752067      37     213  12.839577
2  x}0I__1$MKvIQ+V@z00`*ol;  46752068      23     214  10.135543
3                   4Yv))Dt  46752069      58     194   5.582311
4   H7AP#bi_}4{p7\^O5`^Cpu:  46752070      34     183   5.432857

Exporting data from ClickHouse to Avro and Parquet

Next, we will generate Avro and Parquet files from ClickHouse. We’ll first create a table called events:

1CREATE TABLE events (
2  key String,
3  number UInt64,
4  value1 Int32,
5  value2 UInt64,
6  value3 Float64
7)
8ENGINE = MergeTree
9ORDER BY number;

Now, we’ll ingest 100 million records into this table:

1INSERT INTO events
2SELECT
3    randomPrintableASCII(randUniform(5, 25)) AS key,
4    number,
5    randUniform(18, 100)::Int AS value1,
6    randPoisson(200) AS value2,
7    randExponential(1/10) AS value3
8FROM numbers(100_000_000);
Ok.

0 rows in set. Elapsed: 14.013 sec. Processed 100.00 million rows, 800.00 MB (7.14 million rows/s., 57.09 MB/s.)
Peak memory usage: 91.82 MiB.

We can use the INTO OUTFILE clause to output data, so let’s do that first into a file with the Avro format:

1SELECT *
2FROM events
3INTO OUTFILE 'users.avro' TRUNCATE
4FORMAT Avro;
100000000 rows in set. Elapsed: 16.757 sec. Processed 99.61 million rows, 5.13 GB (5.94 million rows/s., 306.12 MB/s.)
Peak memory usage: 29.83 MiB.

And now the same thing for Parquet:

1SELECT *
2FROM events
3INTO OUTFILE 'users.parquet' TRUNCATE
4FORMAT Parquet;
100000000 rows in set. Elapsed: 3.161 sec. Processed 94.68 million rows, 4.88 GB (29.95 million rows/s., 1.54 GB/s.)
Peak memory usage: 1.07 GiB.

Querying Avro and Parquet with ClickHouse

ClickHouse supports direct querying of both Avro and Parquet files. Let’s start by querying the Avro version of the data:

1SELECT
2    count(),
3    quantiles(0.5, 0.9, 0.99)(value1) AS quantV1,
4    quantiles(0.5, 0.9, 0.99)(value2) AS quantV2,
5    arrayMap(x -> round(x, 2), quantiles(0.5, 0.9, 0.99)(value3)) AS quantV3
6FROM file('users.avro');
   ┌───count()─┬─quantV1────┬─quantV2───────┬─quantV3────────────┐
1.100000000 │ [58,91,99] │ [200,219,233] │ [6.85,22.46,46.01] │
   └───────────┴────────────┴───────────────┴────────────────────┘

1 row in set. Elapsed: 20.318 sec. Processed 99.68 million rows, 3.09 GB (4.91 million rows/s., 152.03 MB/s.)
Peak memory usage: 8.75 MiB.

Now, let’s do the same thing to the Parquet version of the data:

1SELECT
2    count(),
3    quantiles(0.5, 0.9, 0.99)(value1) AS quantV1,
4    quantiles(0.5, 0.9, 0.99)(value2) AS quantV2,
5    arrayMap(x -> round(x, 2), quantiles(0.5, 0.9, 0.99)(value3)) AS quantV3
6FROM file('users.parquet');
   ┌───count()─┬─quantV1────┬─quantV2───────┬─quantV3────────────┐
1.100000000 │ [58,92,99] │ [200,218,234] │ [6.96,22.54,44.25] │
   └───────────┴────────────┴───────────────┴────────────────────┘

1 row in set. Elapsed: 0.482 sec. Processed 85.27 million rows, 2.09 GB (177.09 million rows/s., 4.34 GB/s.)
Peak memory usage: 231.83 MiB.
Share this resource
Follow us
X imageSlack imageGitHub image
Telegram imageMeetup imageRss image