[Docs][Python] Doctest fails on pandas 3.0

[rok]

Describe the bug, including details regarding any error messages, version, and platform.

It seems pandas 3 changed default string type when going from python types?

_________________ [doctest] pyarrow.lib.RecordBatch.add_column _________________
2861 >>> import pyarrow as pa
2862 >>> import pandas as pd
2863 >>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
2864 ...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
2865 >>> batch = pa.RecordBatch.from_pandas(df)
2866 
2867 Add column:
2868 
2869 >>> year = [2021, 2022, 2019, 2021]
2870 >>> batch.add_column(0,"year", year)
Differences (unified diff with -expected +actual):
    @@ -2,5 +2,5 @@
     year: int64
     n_legs: int64
    -animals: string
    +animals: large_string
     ----
     year: [2021,2022,2019,2021]

Here's the same check before and after the upgrade:
pandas 2.3.3
pandas 3.0.0

Component(s)

Python

[tadeja]

take

[h-vetinari]

It's not just doc tests. In conda-forge, I'm seeing the following kind of test failures for arrow v23 (& v22) with pandas v3

FAILED pyarrow/tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="a") are different

Attribute "dtype" are different
[left]:  <StringDtype(na_value=nan)>
[right]: object
FAILED pyarrow/tests/test_pandas.py::test_timestamp_as_object_non_nanosecond[dt0-America/New_York-s] - ValueError: fromutc: dt.tzinfo is not self
FAILED pyarrow/tests/test_pandas.py::test_timestamp_as_object_non_nanosecond[dt0-America/New_York-ms] - ValueError: fromutc: dt.tzinfo is not self
FAILED pyarrow/tests/test_pandas.py::test_timestamp_as_object_non_nanosecond[dt0-America/New_York-us] - ValueError: fromutc: dt.tzinfo is not self
FAILED pyarrow/tests/test_pandas.py::test_timestamp_as_object_non_nanosecond[dt1-America/New_York-s] - ValueError: fromutc: dt.tzinfo is not self
FAILED pyarrow/tests/test_pandas.py::test_timestamp_as_object_non_nanosecond[dt1-America/New_York-ms] - ValueError: fromutc: dt.tzinfo is not self
FAILED pyarrow/tests/test_pandas.py::test_timestamp_as_object_non_nanosecond[dt1-America/New_York-us] - ValueError: fromutc: dt.tzinfo is not self

Except for the fastparquet one, the failures always look the same

_______ test_timestamp_as_object_non_nanosecond[dt1-America/New_York-us] _______

resolution = 'us', tz = 'America/New_York'
dt = datetime.datetime(2020, 1, 1, 0, 0)

    @pytest.mark.parametrize("resolution", ["s", "ms", "us"])
    @pytest.mark.parametrize("tz", [None, "America/New_York"])
    # One datetime outside nanosecond range, one inside nanosecond range:
    @pytest.mark.parametrize("dt", [datetime(1553, 1, 1), datetime(2020, 1, 1)])
    def test_timestamp_as_object_non_nanosecond(resolution, tz, dt):
        # Timestamps can be converted Arrow and reloaded into Pandas with no loss
        # of information if the timestamp_as_object option is True.
        arr = pa.array([dt], type=pa.timestamp(resolution, tz=tz))
        table = pa.table({'a': arr})
    
        for result in [
            arr.to_pandas(timestamp_as_object=True),
            table.to_pandas(timestamp_as_object=True)['a']
        ]:
            assert result.dtype == object
            assert isinstance(result[0], datetime)
            if tz:
                assert result[0].tzinfo is not None
>               expected = result[0].tzinfo.fromutc(dt)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E               ValueError: fromutc: dt.tzinfo is not self

pyarrow/tests/test_pandas.py:4970: ValueError

The errors go away if I pin pandas <3.

[tadeja]

For the new non-docs failures raised by @h-vetinari I've created #48978, replicated failures at my end (with fastparquet and without pytz installed as that is not the case on CI) and suggested fixes in pr #48979