[Python] Pandas is deprecating dataframe interchange protocol

[AlenkaF]

Describe the enhancement requested

New warnings are appearing with pytest and doctest:

  /opt/conda/envs/arrow/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.py:113: Pandas4Warning: The Dataframe Interchange Protocol is deprecated.
  For dataframe-agnostic code, you may want to look into:
  - Arrow PyCapsule Interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html
  - Narwhals: https://github.com/narwhals-dev/narwhals
  
    return _from_dataframe(df.__dataframe__(allow_copy=allow_copy),

due to Pandas moving the dataframe interchange protocol out of the repo in the next major release (4.0.0), see:

• warnings in doctest: https://github.com/apache/arrow/actions/runs/23557627928/job/68588688057?pr=49377#step:6:4433
• warning in pytest: https://github.com/apache/arrow/actions/runs/23557627928/job/68588688057?pr=49377#step:6:4392
• pandas deprecation issue: Move interchange protocol implementation into a separate project pandas-dev/pandas#56732

Two things that come to mind:

• for the time being, which library to use in the conversion tests and documentation examples? Maybe Polars? @jorisvandenbossche I am not sure if there is a plan to have a separate pandas interchange package or that was just an initial idea and pandas plans to completely move away from the protocol?
• How long to keep the implementation of the protocol in PyArrow. I think we should maintain it for a while. There has been not much of maintainance burden so far. The only problem is that feature requests are stuck due to the dataframe interchange protocol project being idle.

Component(s)

Python

[jorisvandenbossche]

for the time being, which library to use in the conversion tests and documentation examples? Maybe Polars?

For the tests, I would just keep those the same for now, and just suppress the warning. By the time pandas actually removes support, we can re-evaluate (maybe by then pyarrow has also deprecated it)

For the docs, it seems polars is already used there as an example.

@jorisvandenbossche I am not sure if there is a plan to have a separate pandas interchange package or that was just an initial idea and pandas plans to completely move away from the protocol?

The linked issue first discussed moving it to a separate package, but then got closed by deprecating it instead. So I don't think there is anyone planning to actually maintain this outside of pandas (at least I am not aware of anyone)

How long to keep the implementation of the protocol in PyArrow. I think we should maintain it for a while. There has been not much of maintainance burden so far.

I would be inclined to deprecate it as well, but indeed if not much of a burden to keep for now, we can also wait until the upstream protocol / consortium comes with a clearer message about its status.

[raulcd]

Maybe we can suppress the warnings, update the docs on a sub-issue and open a second sub-issue to deprecate the protocol on PyArrow, which we can hold a little until there's a clearer message?

[AlenkaF]

Yes, I agree with both 👍