Merge remote-tracking branch 'upstream/main' into test/sleep

Author: mroeschke

doc/source/whatsnew/v2.3.2.rst

@@ -22,8 +22,6 @@ become the default string dtype in pandas 3.0. See
 
 Bug fixes
 ^^^^^^^^^
-- Fix :meth:`~Series.str.isdigit` to correctly recognize unicode superscript
-  characters as digits for :class:`StringDtype` backed by PyArrow (:issue:`61466`)
 - Fix :meth:`~DataFrame.to_json` with ``orient="table"`` to correctly use the
   "string" type in the JSON Table Schema for :class:`StringDtype` columns
   (:issue:`61889`)
@@ -39,4 +37,4 @@ Bug fixes
 Contributors
 ~~~~~~~~~~~~
 
-.. contributors:: v2.3.1..v2.3.2|HEAD
+.. contributors:: v2.3.1..v2.3.2

doc/source/whatsnew/v2.3.3.rst

@@ -1,14 +1,14 @@
 .. _whatsnew_233:
 
-What's new in 2.3.3 (September XX, 2025)
+What's new in 2.3.3 (September 29, 2025)
 ----------------------------------------
 
 These are the changes in pandas 2.3.3. See :ref:`release` for a full changelog
 including other versions of pandas.
 
 {{ header }}
 
-.. _whatsnew_220.py14_compat:
+.. _whatsnew_233.py14_compat:
 
 Pandas 2.3.3 is now compatible with Python 3.14
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -37,27 +37,23 @@ Improvements
   specifying ``include=["object"]`` for backwards compatibility. In a future
   release, this will be deprecated and code for pandas 3+ should be updated to
   do ``include=["str"]`` (:issue:`61916`)
-
+- Support the ``/`` operation between a ``pathlib.Path`` object and a :class:`StringDtype`
+  Series, similarly as it works for object-dtype Series (:issue:`61940`)
 
 .. _whatsnew_233.string_fixes.bugs:
 
 Bug fixes
 ^^^^^^^^^
 - Fix bug in :meth:`Series.str.replace` using named capture groups (e.g., ``\g<name>``) with the Arrow-backed dtype would raise an error (:issue:`57636`)
-- Fix regression in ``~Series.str.contains``, ``~Series.str.match`` and ``~Series.str.fullmatch``
+- Fix regression in :meth:`Series.str.contains`, :meth:`~Series.str.match` and :meth:`~Series.str.fullmatch`
   with a compiled regex and custom flags (:issue:`62240`)
-- Fix :meth:`Series.str.match` and :meth:`Series.str.fullmatch` not matching patterns with groups correctly for the Arrow-backed string dtype (:issue:`61072`)
+- Fix :meth:`Series.str.match` and :meth:`~Series.str.fullmatch` not matching patterns with groups correctly for the Arrow-backed string dtype (:issue:`61072`)
+- Fix bug in :meth:`~DataFrame.groupby` with ``sum()`` and unobserved categories resulting in ``0`` instead of the empty string ``""`` (:issue:`61909`)
+- Fix :meth:`Series.str.isdigit` to correctly recognize unicode superscript
+  characters as digits for :class:`StringDtype` backed by PyArrow (:issue:`61466`)
 - Fix comparing a :class:`StringDtype` Series with mixed objects raising an error (:issue:`60228`)
 - Fix error being raised when using a numpy ufunc with a Python-backed string array (:issue:`40800`)
 
-Improvements and fixes for Copy-on-Write
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Bug fixes
-^^^^^^^^^
-
-- The :meth:`DataFrame.iloc` now works correctly with ``copy_on_write`` option when assigning values after subsetting the columns of a homogeneous DataFrame (:issue:`60309`)
-
 Other changes
 ~~~~~~~~~~~~~
 
@@ -66,9 +62,17 @@ Other changes
   Resampling with a :class:`PeriodIndex` is supported again, but a subset of
   methods that return incorrect results will raise an error in pandas 3.0 (:issue:`57033`)
 
+Other bug fixes
+~~~~~~~~~~~~~~~~
+
+- Fix memory leak in :meth:`DataFrame.to_json` with datetime columns (:issue:`62204`)
+- Fixed regression in :meth:`DataFrame.from_records` not initializing subclasses properly (:issue:`57008`)
+- The :meth:`DataFrame.iloc` now works correctly with ``copy_on_write`` option when assigning values after subsetting the columns of a homogeneous DataFrame (:issue:`60309`)
 
 .. ---------------------------------------------------------------------------
 .. _whatsnew_233.contributors:
 
 Contributors
 ~~~~~~~~~~~~
+
+.. contributors:: v2.3.2..v2.3.3|HEAD

doc/source/whatsnew/v3.0.0.rst

@@ -1054,6 +1054,8 @@ MultiIndex
 I/O
 ^^^
 - Bug in :class:`DataFrame` and :class:`Series` ``repr`` of :py:class:`collections.abc.Mapping` elements. (:issue:`57915`)
+- Fix bug in ``on_bad_lines`` callable when returning too many fields: now emits
+  ``ParserWarning`` and truncates extra fields regardless of ``index_col`` (:issue:`61837`)
 - Bug in :meth:`.DataFrame.to_json` when ``"index"`` was a value in the :attr:`DataFrame.column` and :attr:`Index.name` was ``None``. Now, this will fail with a ``ValueError`` (:issue:`58925`)
 - Bug in :meth:`.io.common.is_fsspec_url` not recognizing chained fsspec URLs (:issue:`48978`)
 - Bug in :meth:`DataFrame._repr_html_` which ignored the ``"display.float_format"`` option (:issue:`59876`)
@@ -1217,10 +1219,11 @@ Other
 - Bug in printing a :class:`DataFrame` with a :class:`DataFrame` stored in :attr:`DataFrame.attrs` raised a ``ValueError`` (:issue:`60455`)
 - Bug in printing a :class:`Series` with a :class:`DataFrame` stored in :attr:`Series.attrs` raised a ``ValueError`` (:issue:`60568`)
 - Deprecated the keyword ``check_datetimelike_compat`` in :meth:`testing.assert_frame_equal` and :meth:`testing.assert_series_equal` (:issue:`55638`)
+- Fixed bug in :meth:`Series.replace` and :meth:`DataFrame.replace` when trying to replace :class:`NA` values in a :class:`Float64Dtype` object with ``np.nan``; this now works with ``pd.set_option("mode.nan_is_na", False)`` and is irrelevant otherwise (:issue:`55127`)
+- Fixed bug in :meth:`Series.replace` and :meth:`DataFrame.replace` when trying to replace :class:`np.nan` values in a :class:`Int64Dtype` object with :class:`NA`; this is now a no-op with ``pd.set_option("mode.nan_is_na", False)`` and is irrelevant otherwise (:issue:`51237`)
 - Fixed bug in the :meth:`Series.rank` with object dtype and extremely small float values (:issue:`62036`)
 - Fixed bug where the :class:`DataFrame` constructor misclassified array-like objects with a ``.name`` attribute as :class:`Series` or :class:`Index` (:issue:`61443`)
 - Fixed regression in :meth:`DataFrame.from_records` not initializing subclasses properly (:issue:`57008`)
--
 
 .. ***DO NOT USE THIS SECTION***
 

pandas/core/arrays/masked.py

@@ -312,7 +312,9 @@ def __setitem__(self, key, value) -> None:
         key = check_array_indexer(self, key)
 
         if is_scalar(value):
-            if is_valid_na_for_dtype(value, self.dtype):
+            if is_valid_na_for_dtype(value, self.dtype) and not (
+                lib.is_float(value) and not is_nan_na()
+            ):
                 self._mask[key] = True
             else:
                 value = self._validate_setitem_value(value)

pandas/core/missing.py

@@ -15,6 +15,8 @@
 
 import numpy as np
 
+from pandas._config import is_nan_na
+
 from pandas._libs import (
     NaT,
     algos,
@@ -37,7 +39,11 @@
     is_object_dtype,
     needs_i8_conversion,
 )
-from pandas.core.dtypes.dtypes import DatetimeTZDtype
+from pandas.core.dtypes.dtypes import (
+    ArrowDtype,
+    BaseMaskedDtype,
+    DatetimeTZDtype,
+)
 from pandas.core.dtypes.missing import (
     is_valid_na_for_dtype,
     isna,
@@ -86,6 +92,31 @@ def mask_missing(arr: ArrayLike, value) -> npt.NDArray[np.bool_]:
     """
     dtype, value = infer_dtype_from(value)
 
+    if (
+        isinstance(arr.dtype, (BaseMaskedDtype, ArrowDtype))
+        and lib.is_float(value)
+        and np.isnan(value)
+        and not is_nan_na()
+    ):
+        # TODO: this should be done in an EA method?
+        if arr.dtype.kind == "f":
+            # GH#55127
+            if isinstance(arr.dtype, BaseMaskedDtype):
+                # error: "ExtensionArray" has no attribute "_data"  [attr-defined]
+                mask = np.isnan(arr._data) & ~arr.isna()  # type: ignore[attr-defined,operator]
+                return mask
+            else:
+                # error: "ExtensionArray" has no attribute "_pa_array"  [attr-defined]
+                import pyarrow.compute as pc
+
+                mask = pc.is_nan(arr._pa_array).fill_null(False).to_numpy()  # type: ignore[attr-defined]
+                return mask
+
+        elif arr.dtype.kind in "iu":
+            # GH#51237
+            mask = np.zeros(arr.shape, dtype=bool)
+            return mask
+
     if isna(value):
         return isna(arr)
 

pandas/io/parsers/python_parser.py

@@ -21,6 +21,7 @@
 import numpy as np
 
 from pandas._libs import lib
+from pandas._typing import Scalar
 from pandas.errors import (
     EmptyDataError,
     ParserError,
@@ -77,7 +78,6 @@
         ArrayLike,
         DtypeObj,
         ReadCsvBuffer,
-        Scalar,
         T,
     )
 
@@ -954,7 +954,9 @@ def _alert_malformed(self, msg: str, row_num: int) -> None:
         """
         if self.on_bad_lines == self.BadLineHandleMethod.ERROR:
             raise ParserError(msg)
-        if self.on_bad_lines == self.BadLineHandleMethod.WARN:
+        if self.on_bad_lines == self.BadLineHandleMethod.WARN or callable(
+            self.on_bad_lines
+        ):
             warnings.warn(
                 f"Skipping line {row_num}: {msg}\n",
                 ParserWarning,
@@ -1189,29 +1191,35 @@ def _rows_to_cols(self, content: list[list[Scalar]]) -> list[np.ndarray]:
 
             for i, _content in iter_content:
                 actual_len = len(_content)
-
                 if actual_len > col_len:
                     if callable(self.on_bad_lines):
                         new_l = self.on_bad_lines(_content)
                         if new_l is not None:
-                            content.append(new_l)  # pyright: ignore[reportArgumentType]
+                            new_l = cast(list[Scalar], new_l)
+                            if len(new_l) > col_len:
+                                row_num = self.pos - (content_len - i + footers)
+                                bad_lines.append((row_num, len(new_l), "callable"))
+                                new_l = new_l[:col_len]
+                            content.append(new_l)
+
                     elif self.on_bad_lines in (
                         self.BadLineHandleMethod.ERROR,
                         self.BadLineHandleMethod.WARN,
                     ):
                         row_num = self.pos - (content_len - i + footers)
-                        bad_lines.append((row_num, actual_len))
-
+                        bad_lines.append((row_num, actual_len, "normal"))
                         if self.on_bad_lines == self.BadLineHandleMethod.ERROR:
                             break
                 else:
                     content.append(_content)
 
-            for row_num, actual_len in bad_lines:
+            for row_num, actual_len, source in bad_lines:
                 msg = (
                     f"Expected {col_len} fields in line {row_num + 1}, saw {actual_len}"
                 )
-                if (
+                if source == "callable":
+                    msg += " from bad_lines callable"
+                elif (
                     self.delimiter
                     and len(self.delimiter) > 1
                     and self.quoting != csv.QUOTE_NONE

pandas/tests/arithmetic/test_string.py

@@ -0,0 +1,114 @@
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from pandas.errors import Pandas4Warning
+
+from pandas import (
+    NA,
+    ArrowDtype,
+    Series,
+    StringDtype,
+)
+import pandas._testing as tm
+
+
+def test_reversed_logical_ops(any_string_dtype):
+    # GH#60234
+    dtype = any_string_dtype
+    warn = None if dtype == object else Pandas4Warning
+    left = Series([True, False, False, True])
+    right = Series(["", "", "b", "c"], dtype=dtype)
+
+    msg = "operations between boolean dtype and"
+    with tm.assert_produces_warning(warn, match=msg):
+        result = left | right
+    expected = left | right.astype(bool)
+    tm.assert_series_equal(result, expected)
+
+    with tm.assert_produces_warning(warn, match=msg):
+        result = left & right
+    expected = left & right.astype(bool)
+    tm.assert_series_equal(result, expected)
+
+    with tm.assert_produces_warning(warn, match=msg):
+        result = left ^ right
+    expected = left ^ right.astype(bool)
+    tm.assert_series_equal(result, expected)
+
+
+def test_pathlib_path_division(any_string_dtype, request):
+    # GH#61940
+    if any_string_dtype == object:
+        mark = pytest.mark.xfail(
+            reason="with NA present we go through _masked_arith_op which "
+            "raises TypeError bc Path is not recognized by lib.is_scalar."
+        )
+        request.applymarker(mark)
+
+    item = Path("/Users/Irv/")
+    ser = Series(["A", "B", NA], dtype=any_string_dtype)
+
+    result = item / ser
+    expected = Series([item / "A", item / "B", ser.dtype.na_value], dtype=object)
+    tm.assert_series_equal(result, expected)
+
+    result = ser / item
+    expected = Series(["A" / item, "B" / item, ser.dtype.na_value], dtype=object)
+    tm.assert_series_equal(result, expected)
+
+
+def test_mixed_object_comparison(any_string_dtype):
+    # GH#60228
+    dtype = any_string_dtype
+    ser = Series(["a", "b"], dtype=dtype)
+
+    mixed = Series([1, "b"], dtype=object)
+
+    result = ser == mixed
+    expected = Series([False, True], dtype=bool)
+    if dtype == object:
+        pass
+    elif dtype.storage == "python" and dtype.na_value is NA:
+        expected = expected.astype("boolean")
+    elif dtype.storage == "pyarrow" and dtype.na_value is NA:
+        expected = expected.astype("bool[pyarrow]")
+
+    tm.assert_series_equal(result, expected)
+
+
+def test_pyarrow_numpy_string_invalid():
+    # GH#56008
+    pa = pytest.importorskip("pyarrow")
+    ser = Series([False, True])
+    ser2 = Series(["a", "b"], dtype=StringDtype(na_value=np.nan))
+    result = ser == ser2
+    expected_eq = Series(False, index=ser.index)
+    tm.assert_series_equal(result, expected_eq)
+
+    result = ser != ser2
+    expected_ne = Series(True, index=ser.index)
+    tm.assert_series_equal(result, expected_ne)
+
+    with pytest.raises(TypeError, match="Invalid comparison"):
+        ser > ser2
+
+    # GH#59505
+    ser3 = ser2.astype("string[pyarrow]")
+    result3_eq = ser3 == ser
+    tm.assert_series_equal(result3_eq, expected_eq.astype("bool[pyarrow]"))
+    result3_ne = ser3 != ser
+    tm.assert_series_equal(result3_ne, expected_ne.astype("bool[pyarrow]"))
+
+    with pytest.raises(TypeError, match="Invalid comparison"):
+        ser > ser3
+
+    ser4 = ser2.astype(ArrowDtype(pa.string()))
+    result4_eq = ser4 == ser
+    tm.assert_series_equal(result4_eq, expected_eq.astype("bool[pyarrow]"))
+    result4_ne = ser4 != ser
+    tm.assert_series_equal(result4_ne, expected_ne.astype("bool[pyarrow]"))
+
+    with pytest.raises(TypeError, match="Invalid comparison"):
+        ser > ser4