[Python] `pyarrow` releases are built without Substrait in version >=19.0.0

[ingomueller-net]

Describe the bug, including details regarding any error messages, version, and platform.

The releases on PyPI seem to be compiled without the Substrait component. On all versions including and above 19.0.0, I observe the following problem (in a venv with pyarrow installed):

$ python -c "import pyarrow.substrait"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import pyarrow.substrait
  File "/usr/local/google/home/ingomueller/.venv/substrait-mlir/lib/python3.13/site-packages/pyarrow/substrait.py", line 30, in <module>
    raise ImportError(
    ...<2 lines>...
    ) from None
ImportError: The pyarrow installation is not built with support for 'substrait' (No module named 'substrait.proto')

In the several versions before, the same command succeeds (and has no output).

I have looked a bit into the build options and scripts but did not find any suspicious change. Another possible reason for the breakage is that something related to the protobuf files has changed but I haven't investigated that yet.

Component(s)

Python

[raulcd]

What wheel are you installing? (OS, version, etcetera so I can validate the specific one)
I don't seem to have issues:

$ python3 -m venv test-substrait
$ source test-substrait/bin/activate
(test-substrait) $ pip install pyarrow numpy
Collecting pyarrow
  Using cached pyarrow-21.0.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.3 kB)
Collecting numpy
  Using cached numpy-2.3.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (62 kB)
Using cached pyarrow-21.0.0-cp312-cp312-manylinux_2_28_x86_64.whl (42.8 MB)
Using cached numpy-2.3.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.6 MB)
Installing collected packages: pyarrow, numpy
Successfully installed numpy-2.3.2 pyarrow-21.0.0
(test-substrait) $ python -c "import pyarrow.substrait"
(test-substrait) $

I've tested 19.0.0 with the same result.

[ingomueller-net]

Ha, that's odd! We seem to be testing with the same wheel but get different results:

$ pip install pyarrow==21.0.0
Collecting pyarrow==21.0.0
  Using cached pyarrow-21.0.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.3 kB)
Using cached pyarrow-21.0.0-cp312-cp312-manylinux_2_28_x86_64.whl (42.8 MB)
Installing collected packages: pyarrow
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 15.0.2
    Uninstalling pyarrow-15.0.2:
      Successfully uninstalled pyarrow-15.0.2
Successfully installed pyarrow-21.0.0
$ python -c "import pyarrow.substrait"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "XXX/lib/python3.12/site-packages/pyarrow/substrait.py", line 30, in <module>
    raise ImportError(
ImportError: The pyarrow installation is not built with support for 'substrait' (No module named 'substrait.proto')

[ingomueller-net]

I investigated a bit more. pyarrow is built with Substrait support alright. That's how we can get an error from pyarrow/substrait.py. The problem is rather (as the error message says) that some package from with substrait (standalone!) cannot be found. The difference in our environments explains why it fails/failed for me: I used substrait==0.12.1; starting with substrait==0.15, it works. Presumably, that version starts to include the protobuf definitions (at least, under that name). The proper fix, thus, seems to be to require that version in some setup file (or test for that version in pyarrow/substrait.py).