Merge remote-tracking branch 'upstream/main' into series-round-object-dtype

Author: sharkipelago

README.md

@@ -19,9 +19,9 @@
 **pandas** is a Python package that provides fast, flexible, and expressive data
 structures designed to make working with "relational" or "labeled" data both
 easy and intuitive. It aims to be the fundamental high-level building block for
-doing practical, **real world** data analysis in Python. Additionally, it has
-the broader goal of becoming **the most powerful and flexible open source data
-analysis / manipulation tool available in any language**. It is already well on
+doing practical, **real-world** data analysis in Python. Additionally, it has
+the broader goal of becoming **the most powerful and flexible open-source data
+analysis/manipulation tool available in any language**. It is already well on
 its way towards this goal.
 
 ## Table of Contents
@@ -64,7 +64,7 @@ Here are just a few of the things that pandas does well:
     data sets
   - [**Hierarchical**][mi] labeling of axes (possible to have multiple
     labels per tick)
-  - Robust IO tools for loading data from [**flat files**][flat-files]
+  - Robust I/O tools for loading data from [**flat files**][flat-files]
     (CSV and delimited), [**Excel files**][excel], [**databases**][db],
     and saving/loading data from the ultrafast [**HDF5 format**][hdfstore]
   - [**Time series**][timeseries]-specific functionality: date range
@@ -138,7 +138,7 @@ or for installing in [development mode](https://pip.pypa.io/en/latest/cli/pip_in
 
 
 ```sh
-python -m pip install -ve . --no-build-isolation -Ceditable-verbose=true
+python -m pip install -ve . --no-build-isolation --config-settings editable-verbose=true
 ```
 
 See the full instructions for [installing from source](https://pandas.pydata.org/docs/dev/development/contributing_environment.html).
@@ -155,7 +155,7 @@ has been under active development since then.
 
 ## Getting Help
 
-For usage questions, the best place to go to is [StackOverflow](https://stackoverflow.com/questions/tagged/pandas).
+For usage questions, the best place to go to is [Stack Overflow](https://stackoverflow.com/questions/tagged/pandas).
 Further, general questions and discussions can also take place on the [pydata mailing list](https://groups.google.com/forum/?fromgroups#!forum/pydata).
 
 ## Discussion and Development

doc/redirects.csv

@@ -643,7 +643,6 @@ generated/pandas.Index.get_slice_bound,../reference/api/pandas.Index.get_slice_b
 generated/pandas.Index.groupby,../reference/api/pandas.Index.groupby
 generated/pandas.Index.has_duplicates,../reference/api/pandas.Index.has_duplicates
 generated/pandas.Index.hasnans,../reference/api/pandas.Index.hasnans
-generated/pandas.Index.holds_integer,../reference/api/pandas.Index.holds_integer
 generated/pandas.Index,../reference/api/pandas.Index
 generated/pandas.Index.identical,../reference/api/pandas.Index.identical
 generated/pandas.Index.inferred_type,../reference/api/pandas.Index.inferred_type

doc/source/index.rst.template

@@ -113,7 +113,7 @@ programming language.
     :titlesonly:
 
     {{ single_doc[:-4] }}
-{% elif single_doc and single_doc.count('.') <= 1 %}
+{% elif single_doc and ((single_doc.count('.') <= 1) or ('tseries' in single_doc)) -%}
 .. autosummary::
     :toctree: reference/api/
 

doc/source/reference/general_functions.rst

@@ -71,6 +71,7 @@ Top-level evaluation
 .. autosummary::
    :toctree: api/
 
+   col
    eval
 
 Datetime formats

doc/source/user_guide/dsintro.rst

@@ -553,6 +553,12 @@ a function of one argument to be evaluated on the DataFrame being assigned to.
 
    iris.assign(sepal_ratio=lambda x: (x["SepalWidth"] / x["SepalLength"])).head()
 
+or, using :meth:`pandas.col`:
+
+.. ipython:: python
+
+   iris.assign(sepal_ratio=pd.col("SepalWidth") / pd.col("SepalLength")).head()
+
 :meth:`~pandas.DataFrame.assign` **always** returns a copy of the data, leaving the original
 DataFrame untouched.
 

doc/source/user_guide/migration-3-strings.rst

@@ -188,6 +188,14 @@ let pandas do the inference. But if you want to be specific, you can specify the
 This is actually compatible with pandas 2.x as well, since in pandas < 3,
 ``dtype="str"`` was essentially treated as an alias for object dtype.
 
+.. attention::
+
+   While using ``dtype="str"`` in constructors is compatible with pandas 2.x,
+   specifying it as the dtype in :meth:`~Series.astype` runs into the issue
+   of also stringifying missing values in pandas 2.x. See the section
+   :ref:`string_migration_guide-astype_str` for more details.
+
+
 The missing value sentinel is now always NaN
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -310,52 +318,69 @@ case.
 Notable bug fixes
 ~~~~~~~~~~~~~~~~~
 
+.. _string_migration_guide-astype_str:
+
 ``astype(str)`` preserving missing values
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-This is a long standing "bug" or misfeature, as discussed in https://github.com/pandas-dev/pandas/issues/25353.
+The stringifying of missing values is a long standing "bug" or misfeature, as
+discussed in https://github.com/pandas-dev/pandas/issues/25353, but fixing it
+introduces a significant behaviour change.
 
-With pandas < 3, when using ``astype(str)`` (using the built-in :func:`str`, not
-``astype("str")``!), the operation would convert every element to a string,
-including the missing values:
+With pandas < 3, when using ``astype(str)`` or ``astype("str")``, the operation
+would convert every element to a string, including the missing values:
 
 .. code-block:: python
 
    # OLD behavior in pandas < 3
-   >>> ser = pd.Series(["a", np.nan], dtype=object)
+   >>> ser = pd.Series([1.5, np.nan])
    >>> ser
-   0      a
+   0    1.5
    1    NaN
-   dtype: object
-   >>> ser.astype(str)
-   0      a
+   dtype: float64
+   >>> ser.astype("str")
+   0    1.5
    1    nan
    dtype: object
-   >>> ser.astype(str).to_numpy()
-   array(['a', 'nan'], dtype=object)
+   >>> ser.astype("str").to_numpy()
+   array(['1.5', 'nan'], dtype=object)
 
 Note how ``NaN`` (``np.nan``) was converted to the string ``"nan"``. This was
 not the intended behavior, and it was inconsistent with how other dtypes handled
 missing values.
 
-With pandas 3, this behavior has been fixed, and now ``astype(str)`` is an alias
-for ``astype("str")``, i.e. casting to the new string dtype, which will preserve
-the missing values:
+With pandas 3, this behavior has been fixed, and now ``astype("str")`` will cast
+to the new string dtype, which preserves the missing values:
 
 .. code-block:: python
 
    # NEW behavior in pandas 3
    >>> pd.options.future.infer_string = True
-   >>> ser = pd.Series(["a", np.nan], dtype=object)
-   >>> ser.astype(str)
-   0      a
+   >>> ser = pd.Series([1.5, np.nan])
+   >>> ser.astype("str")
+   0    1.5
    1    NaN
    dtype: str
-   >>> ser.astype(str).values
-   array(['a', nan], dtype=object)
+   >>> ser.astype("str").to_numpy()
+   array(['1.5', nan], dtype=object)
 
 If you want to preserve the old behaviour of converting every object to a
-string, you can use ``ser.map(str)`` instead.
+string, you can use ``ser.map(str)`` instead. If you want do such conversion
+while preserving the missing values in a way that works with both pandas 2.x and
+3.x, you can use ``ser.map(str, na_action="ignore")`` (for pandas 3.x only, you
+can do ``ser.astype("str")``).
+
+If you want to convert to object or string dtype for pandas 2.x and 3.x,
+respectively, without needing to stringify each individual element, you will
+have to use a conditional check on the pandas version.
+For example, to convert a categorical Series with string categories to its
+dense non-categorical version with object or string dtype:
+
+.. code-block:: python
+
+   >>> import pandas as pd
+   >>> ser = pd.Series(["a", np.nan], dtype="category")
+   >>> ser.astype(object if pd.__version__ < "3" else "str")
 
 
 ``prod()`` raising for string data

doc/source/whatsnew/v2.3.1.rst

@@ -73,4 +73,4 @@ Bug fixes
 Contributors
 ~~~~~~~~~~~~
 
-.. contributors:: v2.3.0..v2.3.1|HEAD
+.. contributors:: v2.3.0..v2.3.1

doc/source/whatsnew/v2.3.2.rst

@@ -1,6 +1,6 @@
 .. _whatsnew_232:
 
-What's new in 2.3.2 (August XX, 2025)
+What's new in 2.3.2 (August 21, 2025)
 -------------------------------------
 
 These are the changes in pandas 2.3.2. See :ref:`release` for a full changelog
@@ -28,9 +28,13 @@ Bug fixes
 - Boolean operations (``|``, ``&``, ``^``) with bool-dtype objects on the left and :class:`StringDtype` objects on the right now cast the string to bool, with a deprecation warning (:issue:`60234`)
 - Fixed ``~Series.str.match``, ``~Series.str.fullmatch`` and ``~Series.str.contains``
   with compiled regex for the Arrow-backed string dtype (:issue:`61964`, :issue:`61942`)
+- Bug in :meth:`Series.replace` and :meth:`DataFrame.replace` inconsistently
+  replacing matching values when missing values are present for string dtypes (:issue:`56599`)
 
 .. ---------------------------------------------------------------------------
 .. _whatsnew_232.contributors:
 
 Contributors
 ~~~~~~~~~~~~
+
+.. contributors:: v2.3.1..v2.3.2|HEAD

doc/source/whatsnew/v3.0.0.rst

@@ -117,10 +117,28 @@ process in more detail.
 
     `PDEP-7: Consistent copy/view semantics in pandas with Copy-on-Write <https://pandas.pydata.org/pdeps/0007-copy-on-write.html>`__
 
-.. _whatsnew_300.enhancements.enhancement2:
+.. _whatsnew_300.enhancements.col:
 
-Enhancement2
-^^^^^^^^^^^^
+``pd.col`` syntax can now be used in :meth:`DataFrame.assign` and :meth:`DataFrame.loc`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can now use ``pd.col`` to create callables for use in dataframe methods which accept them. For example, if you have a dataframe
+
+.. ipython:: python
+
+    df = pd.DataFrame({'a': [1, 1, 2], 'b': [4, 5, 6]})
+
+and you want to create a new column ``'c'`` by summing ``'a'`` and ``'b'``, then instead of
+
+.. ipython:: python
+
+    df.assign(c = lambda df: df['a'] + df['b'])
+
+you can now write:
+
+.. ipython:: python
+
+    df.assign(c = pd.col('a') + pd.col('b'))
 
 New Deprecation Policy
 ^^^^^^^^^^^^^^^^^^^^^^
@@ -1094,7 +1112,6 @@ Other
 - Bug in :meth:`Series.isin` raising ``TypeError`` when series is large (>10**6) and ``values`` contains NA (:issue:`60678`)
 - Bug in :meth:`Series.mode` where an exception was raised when taking the mode with nullable types with no null values in the series. (:issue:`58926`)
 - Bug in :meth:`Series.rank` that doesn't preserve missing values for nullable integers when ``na_option='keep'``. (:issue:`56976`)
-- Bug in :meth:`Series.replace` and :meth:`DataFrame.replace` inconsistently replacing matching instances when ``regex=True`` and missing values are present. (:issue:`56599`)
 - Bug in :meth:`Series.replace` and :meth:`DataFrame.replace` throwing ``ValueError`` when ``regex=True`` and all NA values. (:issue:`60688`)
 - Bug in :meth:`Series.to_string` when series contains complex floats with exponents (:issue:`60405`)
 - Bug in :meth:`read_csv` where chained fsspec TAR file and ``compression="infer"`` fails with ``tarfile.ReadError`` (:issue:`60028`)

pandas/__init__.py

@@ -105,6 +105,7 @@
     Series,
     DataFrame,
 )
+from pandas.core.col import col
 
 from pandas.core.dtypes.dtypes import SparseDtype
 
@@ -281,6 +282,7 @@
     "array",
     "arrays",
     "bdate_range",
+    "col",
     "concat",
     "crosstab",
     "cut",