[CI][Python] Pandas nightly tests failing with wrong datetime unit (pandas 3.0 compat)

[AlenkaF]

Describe the bug, including details regarding any error messages, version, and platform.

Windows wheels seem to be failing due to failing test_column_index_names_with_tz and test_mismatch_metadata_schema tests for a while now with:

>           tm.assert_frame_equal(result, expected)
E           AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="datetime") are different
E           
E           Attribute "dtype" are different
E           [left]:  datetime64[ns, UTC]
E           [right]: datetime64[us, UTC]

See: https://github.com/ursacomputing/crossbow/actions/runs/19850201124/job/56875287582#step:9:949

Component(s)

Continuous Integration, Python

[AlenkaF]

cc @raulcd

[AlenkaF]

Same thing is happening on:

• test-conda-python-3.13-pandas-nightly-numpy-nightly
• test-conda-python-3.13-pandas-upstream_devel-numpy-nightly

[pitrou]

cc @jorisvandenbossche @raulcd

[raulcd]

All those jobs have in common that use either pandas nightly builds or the latest development commit. There has been several commits on the last 4-5 days (which is when this started failing) on pandas related to timestamp resolutions, so it seems possibly a regression on pandas:
pandas-dev/pandas@1b5b02c...8813faf

A thing we could do. that spawns from this, is modify our wheels being built to stop using pandas from the nightlies channel as from what I can see on PyPI pandas now provides free-threaded wheels for aarch64:

arrow/ci/docker/python-free-threaded-wheel-manylinux-test-unittests.dockerfile

Lines 44 to 50 in 79695c2

	# pandas doesn't provide wheels for aarch64 yet, so we have to install nightly Cython
	# along with the rest of pandas' build dependencies and disable build isolation
	RUN python -m pip install \
	--pre \
	--prefer-binary \
	--extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
	Cython numpy

[raulcd]

@WillAyd FYI

[jorisvandenbossche]

Pandas has changed the default unit for datetime64/timedelta64 from ns to us, and indeed there were a bunch of additional changes for that in the last days (eg pd.date_range() will now typically return a us resolution).
So my assumption is that those failures are "expected" and probably need a small update on the pyarrow side to match this changed behaviour in pandas (of course it could also signal a bug in the pandas changes). I can take a closer look later today.

[AlenkaF]

I am happy to submit a PR for the update needed on PyArrow side 👍