Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB)

[ben-freist]

Describe the bug
I can write a table with a column whose entries together are larger than 2**31-1.
Reading them fails with an overflow error.

To Reproduce

from pathlib import Path
import pyarrow
import tempfile
import arro3.io

schema = pyarrow.schema([pyarrow.field("html", pyarrow.binary())])
sizes = [5068563] * 500

new_table = pyarrow.Table.from_pylist([{"html": b"0" * size} for size in sizes], schema)

with tempfile.TemporaryDirectory() as tmp_dir:
    tmp = Path(tmp_dir) / "evil.parquet"
    arro3.io.write_parquet(new_table, tmp)
    print(len(arro3.io.read_parquet(tmp).read_all()))

This throws the following error:

Traceback (most recent call last):
  File "overflow.py", line 87, in <module>
    print(len(arro3.io.read_parquet(tmp).read_all()))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: C Data interface error: Parquet argument error: Parquet error: index overflow decoding byte array

Expected behavior
Should print 500.

Additional context
arro3 is just a thin wrapper around arro-rs, which I'm using for convenience.
I'm using arro3-io version 0.5.1 and python 3.12.
arro3 uses arrow-rs 54.1.0 and our internal library uses arrow-rs 55.1.0. The problem is present with both.
pyarrow can read the file just fine, that's what makes me think that this might be an error in arrow-rs.

[ben-freist]

For convenience here's the file: https://limewire.com/d/oaah6#xuomvd1yxZ
I've used compression='gzip(2)' to make it a bit smaller.

[jonded94]

Just for reference, we're seeing the same error even with a arrow-rs built from the most recent git commit 291e6e5 (dev commit after Arrow 55.2.0)

[alamb]

I've used compression='gzip(2)' to make it a bit smaller.

Here is the file hosted on gihub for convenience: evil.zip

[alamb]

The failure is here: https://github.com/apache/arrow-rs/blob/9c0cb9a56f0099e7d39087826d7e409ce0f1bf5f/parquet/src/arrow/buffer/offset_buffer.rs#L78-L77

Which is related to creating StringArray where the array has more than 2GB of data. With the default batch_size of 8k, rows, that means the average string length in each batch is restricted to 2GB / 8k = 256k

Your data file has a length of 5MB per row, which means you can only fit ~409 rows in each batch before exceeding the 2GB limit

The real fix for this bug is probably for the parquet decoder to internally create smaller batches when the data can't fit into the target batch size and data type

Workaround 1: use a smaller batch size

1. Set the batch size to be smaller (it works for me with a batch size of 100)

I am able to reproduce this using datafusion like

datafusion-cli -c "select length(html) from 'evil.parquet';"
DataFusion CLI v49.0.0

thread 'main' panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-55.2.0/src/builder/generic_bytes_builder.rs:86:57:
byte array offset overflow
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Setting the batch size to 100 lets it go:

andrewlamb@Andrews-MacBook-Pro-3 Downloads %  datafusion-cli -c "set datafusion.execution.batch_size=100; select avg(length(html)) from 'evil.parquet';"
DataFusion CLI v49.0.0
0 row(s) fetched.
Elapsed 0.000 seconds.

+------------------------------------------+
| avg(character_length(evil.parquet.html)) |
+------------------------------------------+
| 5068563.0                                |
+------------------------------------------+
1 row(s) fetched.
Elapsed 0.528 seconds.

Workaround 2: use a different arrow type:

You can potentially override the schema to use DataType::LargeUt8 rather than Utf8DataType::Utf8: https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderOptions.html#method.with_schema

[jonded94]

@alamb setting DataType::LargeUt8 (which is large_string() in pyarrow) does fix it, but this still is a arrow-rs bug AFAIS, since both pyarrow and duckdb can read the file just fine without changing the schema. I assume they do some internal splitting into smaller batches to not run into integer overflows.

[alamb]

@alamb setting DataType::LargeUt8 (which is large_string() in pyarrow) does fix it, but this still is a arrow-rs bug AFAIS, since both pyarrow and duckdb can read the file just fine without changing the schema. I assume they do some internal splitting into smaller batches to not run into integer overflows.

Yes I agree this is a bug in arrow-rs (specifically I think it is a bug in the parquet reader which is part of this repo)

What I think should happen is

1. when reading data into a Utf8 column,
2. if batch_size records can't be read into the target StringArray without overflowing the offesets
3. (aka more than 2GB) fewer than batch_size records should be read.

[JigaoLuo]

I see the same error during my-own parquet rewriting today with arrow-array-56.1.0

 thread '<unnamed>' panicked at /home/jluo/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-56.1.0/src/builder/generic_bytes_builder.rs:86:57:
 byte array offset overflow
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 
 thread '<unnamed>' panicked at /home/jluo/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-56.1.0/src/builder/generic_bytes_builder.rs:86:57:
 byte array offset overflow
 

(the same error pattern then repeats)

[jonded94]

Bumping this issue again, appears with public data too (https://huggingface.co/datasets/HuggingFaceTB/dclm-edu).

>>> ParquetFile(".../dclm-edu/data/000_00001.parquet").read_row_group(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Arrow read/write error: Parquet argument error: Parquet error: index overflow decoding byte array
>>> ParquetFile(".../dclm-edu/data/000_00002.parquet").read_row_group(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Arrow read/write error: Parquet argument error: Parquet error: index overflow decoding byte array

(using our own PyO3 wrapper layer around arrow-rs here)

[alamb]

Thanks @jonded94

I think the workaround is to read the data as a LargeStringArray or StringView (DataType::LargeUtf8 or DataType::StringView) or crank down the batch size

The real solution would be to teach the parquet decoder how to emit batches smaller than batch_size when it hit such an issue

(using our own PyO3 wrapper layer around arrow-rs here)

Can you point me at that? Maybe we should change the default type read by arrow-rs for Strings to StringView 🤔

[jonded94]

(using our own PyO3 wrapper layer around arrow-rs here)

Can you point me at that?

Sadly no, since that is a company-internal library 😅 But it's in principle very similar to arro3 (https://github.com/kylebarron/arro3) or even directly inspired by it in parts.

I think the workaround is to read the data as a LargeStringArray or StringView (DataType::LargeUtf8 or DataType::StringView)
[...]
Maybe we should change the default type read by arrow-rs for Strings to StringView 🤔

I get how LargeUtf8 would solve the issue, but could you briefly explain how StringView works and how it's able to solve the 32-bit overflow issue? I'm familiar with the concept of string views in C++, but I don't quite get what it would do in Arrow.

Slightly tangential: In this particular issue, I think pyarrow.ParquetFile(...).read_row_group(0) wouldn't crash because it's returning a Table which internally is representing the columns as ChunkedArray instead of Array, i.e. it's circumventing the issue that a single StringArray has to point to more than 2**32 bytes of String data by just splitting the arrays up.

As arrow-rs does not have the concept of "ChunkedArray" or "Table", we can only ever read as a single RecordBatch here. Would a single StringView RecordBatch be able to make more than 2**32 bytes of string data available? Wouldn't the string view still need a pointer larger than 32-bit to access all data?

Also another stupid question, sorry: How/where can you set with which type a specific column shall be read in arrow-rs? We have some schema upcasting logic in our internal library, but we apply that after the data was read from arrow-rs, so it actually didn't help us here since we already error out from stuff happening in arrow-rs first.

EDIT: Just as a datapoint from https://huggingface.co/datasets/HuggingFaceTB/dclm-edu/blob/main/data/000_00000.parquet, the parquet file I listed above, all rows are in a single row group, with the column text being the culprit, since it appears to be larger than 2**32 bytes:
row_groups.0.columns.0.meta_data.total_uncompressed_size is 4,714,628,206

[alamb]

I get how LargeUtf8 would solve the issue, but could you briefly explain how StringView works and how it's able to solve the 32-bit overflow issue? I'm familiar with the concept of string views in C++, but I don't quite get what it would do in Arrow.

I think this blog does a pretty good explaining what is going on: https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1 (basically strings are stored as a pointer / offset with multiple buffers)

As arrow-rs does not have the concept of "ChunkedArray" or "Table", we can only ever read as a single RecordBatch here.

I think Vec<RecordBatch> is the equivalent -- you certainly don't have to read the entire dataset into a single RecordBatch

Would a single StringView RecordBatch be able to make more than 2**32 bytes of string data available? Wouldn't the string view still need a pointer larger than 32-bit to access all data?

Yes -- it can do more than 2^32 bytes

The way it does so is that it has multiple "buffers" (which can each be 2GB) and then stores the pointer as a buffer index (i32) and buffer offset (i32)

You can read more here

• https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout