Enh arrow json extension #61103
Merged
Enh arrow json extension #61103
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mroeschke
reviewed
Mar 11, 2025
pandas/core/dtypes/dtypes.py
Outdated
Comment on lines
2268
to
2271
| elif isinstance(pa_type, pa.ExtensionType): | ||
| return type(self)(pa_type.storage_type).type | ||
| elif isinstance(pa_type, pa.JsonType): | ||
| return str |
Member
There was a problem hiding this comment.
Suggested change
| elif isinstance(pa_type, pa.ExtensionType): | |
| return type(self)(pa_type.storage_type).type | |
| elif isinstance(pa_type, pa.JsonType): | |
| return str | |
| elif isinstance(pa_type, pa.BaseExtensionType): | |
| return type(self)(pa_type.storage_type).type |
mroeschke
reviewed
Mar 11, 2025
doc/source/whatsnew/v3.0.0.rst
Outdated
| - :class:`Rolling` and :class:`Expanding` now support ``nunique`` (:issue:`26958`) | ||
| - :class:`Rolling` and :class:`Expanding` now support aggregations ``first`` and ``last`` (:issue:`33155`) | ||
| - :func:`read_parquet` accepts ``to_pandas_kwargs`` which are forwarded to :meth:`pyarrow.Table.to_pandas` which enables passing additional keywords to customize the conversion to pandas, such as ``maps_as_pydicts`` to read the Parquet map data type as python dictionaries (:issue:`56842`) | ||
| - :meth: ``ArrowDtype.type`` now supports the pyarrow json data type (:issue:`60958`) |
Member
There was a problem hiding this comment.
Suggested change
| - :meth: ``ArrowDtype.type`` now supports the pyarrow json data type (:issue:`60958`) | |
| - :class:`ArrowDtype` now supports ``pyarrowJsonType`` (:issue:`60958`) |
mroeschke
reviewed
Mar 12, 2025
doc/source/whatsnew/v3.0.0.rst
Outdated
| - :meth:`Series.cummin` and :meth:`Series.cummax` now supports :class:`CategoricalDtype` (:issue:`52335`) | ||
| - :meth:`Series.plot` now correctly handle the ``ylabel`` parameter for pie charts, allowing for explicit control over the y-axis label (:issue:`58239`) | ||
| - :meth:`DataFrame.plot.scatter` argument ``c`` now accepts a column of strings, where rows with the same string are colored identically (:issue:`16827` and :issue:`16485`) | ||
| - :class:`ArrowDtype` now supports ``pyarrowJsonType`` (:issue:`60958`) |
Member
There was a problem hiding this comment.
Suggested change
| - :class:`ArrowDtype` now supports ``pyarrowJsonType`` (:issue:`60958`) | |
| - :class:`ArrowDtype` now supports ``pyarrow.JsonType`` (:issue:`60958`) |
mroeschke
reviewed
Mar 12, 2025
| tm.assert_series_equal(result, expected) | ||
|
|
||
|
|
||
| def test_arrow_json_type(): |
Member
There was a problem hiding this comment.
You'll need to @pytest.mark.skipif(pa_version_under19p0, reason=...) as pa.json_ is new as of Pyarrow 19
Added support for PyArrow's JSON extension type in ArrowDtype.type by mapping JsonType to str. Fixes pandas-dev#60958.
asharmalik19
force-pushed
the
enh-arrow-json-extension
branch
from
d2c4db1 to
dced6fa
Compare
Improved extension type handling by using BaseExtensionType for consistent storage type resolution across all PyArrow extension types, including JSON. Fixes pandas-dev#60958
asharmalik19
force-pushed
the
enh-arrow-json-extension
branch
from
dced6fa to
a945971
Compare
mroeschke
approved these changes
Mar 12, 2025
Member
|
Thanks @asharmalik19 |
Contributor
Author
|
Thanks @mroeschke for such quick feedback |
Contributor
|
Thank you so much for your help, @asharmalik19 and @mroeschke ! |
Contributor
|
@mroeschke Is there any chance this fix for arrow extension types could get back ported to pandas 2.x or do we need to wait for 3.x? |
Member
|
I don't think the team is planning on making another 2.x release. We ideally will be releasing a 3.0rc this week or next cc @jbrockmendel |
Member
|
@mroeschke There has been some new backported commits to |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
pa.json_in arrow extension type #60958doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.