Skip to content

DOC : document pyarrow vs python string trade-offs #63505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
joddeepesh-cloud wants to merge 2 commits into
base: main
Choose a base branch
from
Open

DOC : document pyarrow vs python string trade-offs #63505

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a6e039e
DOC : document pyarrow vs python string trade-offs
joddeepesh-cloud Dec 29, 2025
1e7c52a
DOC: clarify note about optimized indexing methods
joddeepesh-cloud Dec 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions doc/source/user_guide/indexing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,17 @@ this area.

.. note::

The Python and NumPy indexing operators ``[]`` and attribute operator ``.``
provide quick and easy access to pandas data structures across a wide range
of use cases. This makes interactive work intuitive, as there's little new
to learn if you already know how to deal with Python dictionaries and NumPy
The Python and NumPy indexing operators ``[]`` and the attribute operator ``.``
provide quick and easy access to pandas data structures across a wide range of
use cases. This makes interactive work intuitive, as there's little new to
learn if you already know how to deal with Python dictionaries and NumPy
arrays. However, since the type of the data to be accessed isn't known in
advance, directly using standard operators has some optimization limits. For
production code, we recommended that you take advantage of the optimized
pandas data access methods exposed in this chapter.
advance, directly using these standard operators has some optimization limits.

For performance-critical or production code, we recommend using the optimized
pandas data access methods (such as ``.loc`` and ``.iloc``) described in this
chapter.

See the :ref:`MultiIndex / Advanced Indexing <advanced>` for ``MultiIndex`` and more advanced indexing documentation.

See the :ref:`cookbook<cookbook.selection>` for some advanced strategies.
Expand Down
35 changes: 35 additions & 0 deletions doc/source/user_guide/text.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,41 @@ See :ref:`text.four_string_variants` section below for details.

.. _text.string_methods:

String storage: pyarrow vs python
---------------------------------

Pandas supports different storage backends for string data.
Depending on the configuration and installed dependencies,
string data may be stored using either a Python object-based
implementation or a pyarrow-backed implementation.

In general, the pyarrow-backed string storage is recommended
for most users, as it provides better performance and a more
Copy link
Member

@rhshadrach rhshadrach Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyArrow is not more performant across the board, nor is it a more compact memory representation in all cases. Python will cache certain strings, leading to much less memory usage with Python storage. PyArrow does not do this.

size = 10
df = pd.DataFrame(
    {
        "a": [f"{e}" for e in range(100)] * size, 
        "b": np.random.randint(0, 100_000, size * 100)
    }
)
df["b"] = df["b"].map(str).astype("object")

gb = df.groupby("a")
gb.sum()  # Populate caches
%timeit gb.sum()
# 83.4 μs ± 435 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

df_arrow = df.convert_dtypes(dtype_backend="pyarrow")
gb = df_arrow.groupby("a")
gb.sum()  # Populate caches
%timeit gb.sum()
# 6.75 ms ± 30.3 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

compact memory representation.

**pyarrow-backed string storage**

- Pros:
- More compact memory footprint
- Faster vectorized string operations
- Cons:
- Strings are immutable; modifying values results in new arrays
- Some edge-case behavior differences compared to Python strings

**Python object string storage**

- Pros:
- Uses Python string objects (mutable at the array level)
- Behavior consistent with standard Python string semantics
- Cons:
- Higher memory usage
- Slower performance due to lack of vectorization

While pandas aims to provide identical results regardless of
the underlying string storage, some behavior differences may
exist in edge cases (for example, certain Unicode operations).
These differences are documented where relevant.

String methods
==============

Expand Down