[Packaging] Support abi3 wheels for pyarrow

[mroeschke]

Describe the enhancement requested

It would be great to have pyarrow ab3 (non-freethreading) wheels to reduce the amount of wheels shipped with a pyarrow release.

From an agent driven analysis, it appears this would require:

• Change (AFAICT) internals to use more Limited API calls or equivalent APIs
• Building with Numpy >= 2.0
• Setting an abi3 floor to 3.11 (0x030B0000) to support buffer protocol features

Happy to open pull requests (generally in that order of the list) to work towards this enhancement

Component(s)

Packaging

[raulcd]

There was a discussion about this on Zulip when we released 24.0.0 as we had to request a quota increase on PyPI.
I don't think we ended up opening an issue, thanks for raising this one @mroeschke .
@jorisvandenbossche @pitrou were part of the discussion.

I agree that this is an important enhancement to investigate.

[pitrou]

I expect this will depend on the maturity of Cython support for the stable ABI.

[raulcd]

Copy&pasting the link @jorisvandenbossche shared about Cython support on Zulip:
https://cython.readthedocs.io/en/latest/src/userguide/limited_api.html