[C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file

[TomAugspurger]

Describe the bug, including details regarding any error messages, version, and platform.

Something about this parquet file form https://github.com/Parquet/parquet-compatibility/ causes an exception while reading with pyarrow 22.0.0:

import urllib.request
import pathlib
import pyarrow.parquet as pq

p = pathlib.Path("nation.impala.parquet")
if not p.exists():
    urllib.request.urlretrieve(
        "https://github.com/Parquet/parquet-compatibility/raw/master/parquet-testdata/impala/1.1.1-NONE/nation.impala.parquet",
        p
    )

pq.read_table(p)

which raises with

Traceback (most recent call last):
  File "/Users/toaugspurger/gh/dask/dask/bug.py", line 12, in <module>
    pq.read_table(p)
  File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/parquet/core.py", line 1899, in read_table
    return dataset.read(columns=columns, use_threads=use_threads,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/parquet/core.py", line 1538, in read
    table = self._dataset.to_table(
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_dataset.pyx", line 589, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 3969, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Invalid number of indices: 0

pyarrow 21.0.0 was able to read that file.

Component(s)

Parquet, Python

[raulcd]

I've been able to bisect and find the first commit that produces the failure for the above parquet file.
The commit: 33d1f32 fails reading the file.
The previous commit: 353a0a2 does not fail reading the file.

This was fixing some undefined behavior found on fuzzing, we should investigate further the file to understand what is the problem but this will give us a hint. It might probably tell us that there's a problem with the file generation.

CC @pitrou

[pitrou]

Ok, I looked at this.

These are extremely old Parquet files that were generated while the Parquet format was still being designed.

One of the columns, of length 25, has RLE_BIT_PACKED-encoded dictionary integers over a bit width of 5. The RLE_BIT_PACKED data is 17 bytes and contains a single bit-packed run with count=4, which is really 32 values. But 17 bytes (of which 1 byte header) cannot encode 32 5-bit values (only 25). So, while the RLE_BIT_PACKED data can technically encode the 25 integers, it lacks the padding implicit in the spec.

Now, we could relax this a bit in our decoder, but the question is whether more recent files share this property.

[TomAugspurger]

FWIW, this just came up in one of Dask's unit tests. I suspect it's not critical, especially if this file is arguably invalid.