Regression with pandas 2.0 & how to handle for older versions?

[h-vetinari]

Just saw that pandas 2.0 broke the CI here (see #1015, but also #899), and that upstream patched this.

What do we want to do for older versions here? I think I'll try to backport this for arrow 11 and maybe 10, but it seems we might want to add a constraint pandas <2 to arrow 8 & 9? Or is that too big a hammer for 6 failing tests out of ~7000?

CC @jorisvandenbossche @kou @conda-forge/arrow-cpp

[h-vetinari]

Unfortunately, backporting apache/arrow#34878 only solved 3 out of the 6 failures. The rest:

=========================== short test summary info ============================
FAILED test_compute.py::test_extract_datetime_components - assert False
 +  where False = <built-in method equals of pyarrow.lib.Int64Array object at 0x7fc7079506a0>(<pyarrow.lib.Int32Array object at 0x7fc707950a60>\n[\n  1970,\n  2000,\n  2033,\n  2020,\n  2019,\n  2019,\n  2009,\n  2010,\n  2010,\n  2010,\n  2006,\n  2005,\n  2008,\n  2008,\n  2012\n])
 +    where <built-in method equals of pyarrow.lib.Int64Array object at 0x7fc7079506a0> = <pyarrow.lib.Int64Array object at 0x7fc7079506a0>\n[\n  1970,\n  2000,\n  2033,\n  2020,\n  2019,\n  2019,\n  2009,\n  2010,\n  2010,\n  2010,\n  2006,\n  2005,\n  2008,\n  2008,\n  2012\n].equals
 +      where <pyarrow.lib.Int64Array object at 0x7fc7079506a0>\n[\n  1970,\n  2000,\n  2033,\n  2020,\n  2019,\n  2019,\n  2009,\n  2010,\n  2010,\n  2010,\n  2006,\n  2005,\n  2008,\n  2008,\n  2012\n] = <function year at 0x7fc7298a51c0>(<pyarrow.lib.TimestampArray object at 0x7fc707950580>\n[\n  1970-01-01 00:00:59.123456789,\n  2000-02-29 23:23:23.999999999,\n  2033-05-18 03:33:20.000000000,\n  2020-01-01 01:05:05.001000000,\n  2019-12-31 02:10:10.002000000,\n  2019-12-30 03:15:15.003000000,\n  2009-12-31 04:20:20.004132000,\n  2010-01-01 05:25:25.005321000,\n  2010-01-03 06:30:30.006163000,\n  2010-01-04 07:35:35.000000000,\n  2006-01-01 08:40:40.000000000,\n  2005-12-31 09:45:45.000000000,\n  2008-12-28 00:00:00.000000000,\n  2008-12-29 00:00:00.000000000,\n  2012-01-01 01:02:03.000000000\n])
 +        where <function year at 0x7fc7298a51c0> = pc.year
 +    and   <pyarrow.lib.Int32Array object at 0x7fc707950a60>\n[\n  1970,\n  2000,\n  2033,\n  2020,\n  2019,\n  2019,\n  2009,\n  2010,\n  2010,\n  2010,\n  2006,\n  2005,\n  2008,\n  2008,\n  2012\n] = <built-in function array>(1970-01-01 00:00:59.123456789    1970\n2000-02-29 23:23:23.999999999    2000\n2033-05-18 03:33:20.000000000    2033\n2020-01-01 01:05:05.001000000    2020\n2019-12-31 02:10:10.002000000    2019\n2019-12-30 03:15:15.003000000    2019\n2009-12-31 04:20:20.004132000    2009\n2010-01-01 05:25:25.005321000    2010\n2010-01-03 06:30:30.006163000    2010\n2010-01-04 07:35:35.000000000    2010\n2006-01-01 08:40:40.000000000    2006\n2005-12-31 09:45:45.000000000    2005\n2008-12-28 00:00:00.000000000    2008\n2008-12-29 00:00:00.000000000    2008\n2012-01-01 01:02:03.000000000    2012\ndtype: int32)
 +      where <built-in function array> = pa.array
 +      and   1970-01-01 00:00:59.123456789    1970\n2000-02-29 23:23:23.999999999    2000\n2033-05-18 03:33:20.000000000    2033\n2020-01-01 01:05:05.001000000    2020\n2019-12-31 02:10:10.002000000    2019\n2019-12-30 03:15:15.003000000    2019\n2009-12-31 04:20:20.004132000    2009\n2010-01-01 05:25:25.005321000    2010\n2010-01-03 06:30:30.006163000    2010\n2010-01-04 07:35:35.000000000    2010\n2006-01-01 08:40:40.000000000    2006\n2005-12-31 09:45:45.000000000    2005\n2008-12-28 00:00:00.000000000    2008\n2008-12-29 00:00:00.000000000    2008\n2012-01-01 01:02:03.000000000    2012\ndtype: int32 = <pandas.core.indexes.accessors.DatetimeProperties object at 0x7fc707f92090>.year
 +        where <pandas.core.indexes.accessors.DatetimeProperties object at 0x7fc707f92090> = 1970-01-01 00:00:59.123456789   1970-01-01 00:00:59.123456789\n2000-02-29 23:23:23.999999999   2000-02-29 23:23:23.999999999\n2033-05-18 03:33:20.000000000   2033-05-18 03:33:20.000000000\n2020-01-01 01:05:05.001000000   2020-01-01 01:05:05.001000000\n2019-12-31 02:10:10.002000000   2019-12-31 02:10:10.002000000\n2019-12-30 03:15:15.003000000   2019-12-30 03:15:15.003000000\n2009-12-31 04:20:20.004132000   2009-12-31 04:20:20.004132000\n2010-01-01 05:25:25.005321000   2010-01-01 05:25:25.005321000\n2010-01-03 06:30:30.006163000   2010-01-03 06:30:30.006163000\n2010-01-04 07:35:35.000000000   2010-01-04 07:35:35.000000000\n2006-01-01 08:40:40.000000000   2006-01-01 08:40:40.000000000\n2005-12-31 09:45:45.000000000   2005-12-31 09:45:45.000000000\n2008-12-28 00:00:00.000000000   2008-12-28 00:00:00.000000000\n2008-12-29 00:00:00.000000000   2008-12-29 00:00:00.000000000\n2012-01-01 01:02:03.000000000   2012-01-01 01:02:03.000000000\ndtype: datetime64[ns].dt
FAILED test_pandas.py::test_table_from_pandas_schema_index_columns - AssertionError: DataFrame.index are different

Attribute "dtype" are different
[left]:  int32
[right]: int64
FAILED parquet/test_dataset.py::test_read_partitioned_directory[False] - AssertionError: Attributes of DataFrame.iloc[:, 2] (column name="foo") are different

Attribute "dtype" are different
[left]:  CategoricalDtype(categories=[0, 1], ordered=False)
[right]: CategoricalDtype(categories=[0, 1], ordered=False)

[h-vetinari]

Actually, on windows there were more than 6 failing tests initially:

=========================== short test summary info ===========================
FAILED test_compute.py::test_extract_datetime_components - assert False
FAILED test_pandas.py::TestConvertMetadata::test_column_index_names_with_tz
FAILED test_pandas.py::TestConvertMetadata::test_binary_column_name - NotImpl...
FAILED test_pandas.py::TestZeroCopyConversion::test_zero_copy_dictionaries - ...
FAILED test_pandas.py::test_roundtrip_with_bytes_unicode[columns0] - NotImple...
FAILED test_pandas.py::test_table_from_pandas_schema_index_columns - Assertio...
FAILED parquet/test_dataset.py::test_read_partitioned_directory[False] - Asse...
FAILED parquet/test_pandas.py::test_dataset_read_pandas_common_metadata[_metadata-False-True]
FAILED parquet/test_pandas.py::test_dataset_read_pandas_common_metadata[_metadata-False-False]
FAILED parquet/test_pandas.py::test_dataset_read_pandas_common_metadata[_common_metadata-False-True]
FAILED parquet/test_pandas.py::test_dataset_read_pandas_common_metadata[_common_metadata-False-False]

will see how many are left after apache/arrow#34878.

Interestingly, all these tests don't fail on arrow main, i.e. apache/arrow#34652 didn't need to add extra skips. So from that POV, I think it would also be OK to just skip the failing tests here, as arrow 12 is around the corner and that'll be the version for people to upgrade to who want to have newest everything (including pandas).

[h-vetinari]

Found two more relevant patches (apache/arrow#34498, apache/arrow#34881), will check if that solves things.

[h-vetinari]

The patches seem to work on arrow 11. arrow 10 has 36 failed tests on unix however, so I think the best approach there is to go with a run-constraint pandas <2.

[jorisvandenbossche]

I would just use pandas<2 for the tests (which can be removed for upcoming arrow 12.0), this are just some minor testing failures not worth putting too much time in.

[h-vetinari]

I would just use pandas<2 for the tests

Just for tests, or for the published artefact as well? So far, I've gone¹ with fixing it comprehensively on arrow 11, and forbidding pandas 2.0 for arrow <11.

Footnotes

1. PRs in flight, nothing merged yet. ↩

[jorisvandenbossche]

I would only do it for tests, not forbidding to install latest pandas and pyarrow together (the test failures are were mostly actual "test" failures (i.e. just something in the test we needed to update, not something user-facing that broke, or just accounting for a user-facing change that pandas made) or either small corner cases that had to be fixed (that 99.9% of our users won't run into)

[h-vetinari]

Sounds good, thanks for the input!

[h-vetinari]

Fixed by #899, as well as #1017, #1018, #1019