Skip to content

DEPR: Remove special date-datetime64 case in indexing #62232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

DEPR: Remove special date-datetime64 case in indexing #62232

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
bda37e8
Updated _infer_matches and _maybe_downcast_for_indexing
arthurlw Sep 2, 2025
7522828
Merge with main
arthurlw Sep 2, 2025
2712fd7
Deprecated tests
arthurlw Sep 5, 2025
8b1ab37
Deprecate categorical datetime coercion test
arthurlw Sep 10, 2025
6e1af00
Merge branch 'main' into Deprecate_indexing_dates_and_datetime64
arthurlw Sep 10, 2025
418dc4a
Deprecate get_indexer date coercion test
arthurlw Sep 10, 2025
b60c744
Update timestamp values in test_categorical_coerces_timestamp
arthurlw Sep 10, 2025
2e11d26
Updated deprecated test behavior
arthurlw Sep 14, 2025
b727250
precommit
arthurlw Sep 14, 2025
9083e84
Updated tests according to feedback
arthurlw Oct 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pandas/core/arrays/datetimes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ class DatetimeArray(dtl.TimelikeOps, dtl.DatelikeOps):
_is_recognized_dtype: Callable[[DtypeObj], bool] = lambda x: lib.is_np_dtype(
x, "M"
) or isinstance(x, DatetimeTZDtype)
_infer_matches = ("datetime", "datetime64", "date")
_infer_matches = ("datetime", "datetime64")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this might be worth doing, but the motivating issue I opened was only about the check in indexes.base

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my testing after removing the check in indexes.base, the example in the original issue still fails because of _infer_matches called under _validate_listlike

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, looking at the other places where _infer_matches is used, im OK with that change, but will need to make sure we have appropriate testing and whatsnew note for e.g. equals


@property
def _scalar_type(self) -> type[Timestamp]:
Expand Down
6 changes: 0 additions & 6 deletions pandas/core/indexes/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@
no_default,
)
from pandas._libs.tslibs import (
OutOfBoundsDatetime,
Timestamp,
tz_compare,
)
Expand Down Expand Up @@ -6216,11 +6215,6 @@ def _maybe_downcast_for_indexing(self, other: Index) -> tuple[Index, Index]:
# standardize on UTC
return self.tz_convert("UTC"), other.tz_convert("UTC")

elif self.inferred_type == "date" and isinstance(other, ABCDatetimeIndex):
try:
return type(other)(self), other
except OutOfBoundsDatetime:
return self, other
elif self.inferred_type == "timedelta" and isinstance(other, ABCTimedeltaIndex):
# TODO: we dont have tests that get here
return type(other)(self), other
Expand Down
12 changes: 6 additions & 6 deletions pandas/tests/frame/methods/test_asfreq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,15 +186,15 @@ def test_asfreq_fillvalue(self):
tm.assert_series_equal(expected_series, actual_series)

def test_asfreq_with_date_object_index(self, frame_or_series):
# GH#62158 date objects lose indexing special case
rng = date_range("1/1/2000", periods=20)
ts = frame_or_series(np.random.default_rng(2).standard_normal(20), index=rng)
ts.index = [x.date() for x in ts.index]

ts2 = ts.copy()
ts2.index = [x.date() for x in ts2.index]

result = ts2.asfreq("4h", method="ffill")
expected = ts.asfreq("4h", method="ffill")
tm.assert_equal(result, expected)
with pytest.raises(
TypeError, match="Cannot compare Timestamp with datetime.date"
):
ts.asfreq("4h", method="ffill")

def test_asfreq_with_unsorted_index(self, frame_or_series):
# GH#39805
Expand Down
23 changes: 15 additions & 8 deletions pandas/tests/indexes/datetimes/test_indexing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,10 +535,11 @@ def test_get_indexer_pyarrow(self, as_td):
tm.assert_numpy_array_equal(result2, expected)

def test_get_indexer_date_objs(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of removing the test entirely, test the new behavior, with a comment about the history/change

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(havent looked closely, but this likely applies to tests below)

# GH#62158 date objects lose indexing special case
rng = date_range("1/1/2000", periods=20)

result = rng.get_indexer(rng.map(lambda x: x.date()))
expected = rng.get_indexer(rng)

expected = np.full(len(rng), -1, dtype=np.intp)
tm.assert_numpy_array_equal(result, expected)

def test_get_indexer(self):
Expand Down Expand Up @@ -583,17 +584,23 @@ def test_get_indexer(self):
idx.get_indexer(idx[[0]], method="nearest", tolerance="foo")

@pytest.mark.parametrize(
"target",
"target, expected",
[
[date(2020, 1, 1), Timestamp("2020-01-02")],
[Timestamp("2020-01-01"), date(2020, 1, 2)],
(
[date(2020, 1, 1), Timestamp("2020-01-02")],
np.array([-1, 1], dtype=np.intp),
),
(
[Timestamp("2020-01-01"), date(2020, 1, 2)],
np.array([0, -1], dtype=np.intp),
),
],
)
def test_get_indexer_mixed_dtypes(self, target):
# https://github.com/pandas-dev/pandas/issues/33741
def test_get_indexer_mixed_dtypes(self, target, expected):
# GH#33741 regression test: mixed dtypes should not error
# GH#62158 date objects lose indexing special case
values = DatetimeIndex([Timestamp("2020-01-01"), Timestamp("2020-01-02")])
result = values.get_indexer(target)
expected = np.array([0, 1], dtype=np.intp)
tm.assert_numpy_array_equal(result, expected)

@pytest.mark.parametrize(
Expand Down
18 changes: 3 additions & 15 deletions pandas/tests/io/parser/dtypes/test_categorical.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,24 +272,12 @@ def test_categorical_coerces_numeric(all_parsers):
tm.assert_frame_equal(result, expected)


def test_categorical_coerces_datetime(all_parsers):
parser = all_parsers
dti = pd.DatetimeIndex(["2017-01-01", "2018-01-01", "2019-01-01"], freq=None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this removed?

dtype = {"b": CategoricalDtype(dti)}

data = "b\n2017-01-01\n2018-01-01\n2019-01-01"
expected = DataFrame({"b": Categorical(dtype["b"].categories)})

result = parser.read_csv(StringIO(data), dtype=dtype)
tm.assert_frame_equal(result, expected)


def test_categorical_coerces_timestamp(all_parsers):
parser = all_parsers
dtype = {"b": CategoricalDtype([Timestamp("2014")])}
dtype = {"b": CategoricalDtype([Timestamp("2014-01-01 12:00:00")])}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this change needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when read_csv reads "b\n2014-01-01\n2014-01-01" it converts the data into date objects, while the Timestamps are in datetime64. With the deprecation in this PR, this is no longer supported. Updating the data with the time "b\n2014-01-01 12:00:00\n2014-01-01 12:00:00"ensures read_csv produces datetime64 values that match the Timestamp categories

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we want to continue testing the original input data


data = "b\n2014-01-01\n2014-01-01"
expected = DataFrame({"b": Categorical([Timestamp("2014")] * 2)})
data = "b\n2014-01-01 12:00:00\n2014-01-01 12:00:00"
expected = DataFrame({"b": Categorical([Timestamp("2014-01-01 12:00:00")] * 2)})

result = parser.read_csv(StringIO(data), dtype=dtype)
tm.assert_frame_equal(result, expected)
Expand Down
7 changes: 3 additions & 4 deletions pandas/tests/series/test_arithmetic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -757,20 +757,19 @@ def test_datetime_understood(self, unit):
expected = Series(exp_dti)
tm.assert_series_equal(result, expected)

def test_align_date_objects_with_datetimeindex(self):
def test_align_date_objects_with_datetimeindex_coerced(self):
# GH#62158: datetime.date objects no longer auto-align with Timestamps
rng = date_range("1/1/2000", periods=20)
ts = Series(np.random.default_rng(2).standard_normal(20), index=rng)

ts_slice = ts[5:]
ts2 = ts_slice.copy()
ts2.index = [x.date() for x in ts2.index]
ts2.index = pd.to_datetime([x.date() for x in ts2.index])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of this, test the new behavior with the old index


result = ts + ts2
result2 = ts2 + ts
expected = ts + ts[5:]
expected.index = expected.index._with_freq(None)
tm.assert_series_equal(result, expected)
tm.assert_series_equal(result2, expected)


class TestNamePreservation:
Expand Down
Loading