BUG: OverflowError on DataFrame.dtypes.to_json#66361
Conversation
| if ( | ||
| default_handler is None | ||
| and isinstance(obj_to_write, ABCSeries) | ||
| and any(isinstance(obj, np.dtype) for obj in obj_to_write.array) |
There was a problem hiding this comment.
I don’t think we want to iterate in python like this. If nothing else, probably check dtype=object first
There was a problem hiding this comment.
Thanks, @jbrockmendel for the comment. I added the check dtype=object. Maybe we could use infer_dtype. What do you think? I am not sure about it because infer_dtype returns "unknown-array" in this case.
There was a problem hiding this comment.
i think the right place to do this is in objToJSON.c. Use PyArray_DescrCheck in Object_beginTypeContext. Also pls write tests for all the relevant cases, off the top of my head:
dts = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}).dtypes
dts.to_json(orient="split", index=False)
dts.to_json(orient="table")
dts.to_frame().to_json()
| - Bug in :meth:`DataFrame.to_json` and :meth:`Series.to_json` with ``orient="table"`` silently dropping the timezone of columns with a fixed-offset timezone (e.g. ``datetime.timezone(timedelta(hours=1))``), so the offset was lost on round-trip through :func:`read_json` (:issue:`39537`) | ||
| - Bug in :meth:`DataFrame.to_stata` raising ``KeyError`` when column names require renaming and ``convert_dates`` is specified for a different column (:issue:`60536`) | ||
| - Bug in :meth:`DataFrame.to_string` where ``formatters`` dict was applied to wrong columns when output was horizontally truncated via ``max_cols`` (:issue:`35410`) | ||
| - Bug in :meth:`Series.to_json` and :meth:`DataFrame.to_json` which raised an OverflowError when serializing ``DataFrame.dtypes`` or ``DataFrame.dtypes.to_frame`` (:issue:`61170`) |
There was a problem hiding this comment.
i think this is just about np.dtype or ExtensionDtype objects?
There was a problem hiding this comment.
actually i suspect this doesnt handle ExtensionDtypes well?
There was a problem hiding this comment.
i think this is just about np.dtype or ExtensionDtype objects?
Yes, it covers only np.dtype. I will update the whatsnew note.
There was a problem hiding this comment.
actually i suspect this doesnt handle ExtensionDtypes well?
In a way, it also handles ExtensionDtype objects. On main, the example below raises:
ValueError: Unable to serialize object to JSON: encountered an unsupported object type; convert the column to a supported type (e.g. str) before calling to_json.
ser = DataFrame(
{
"A": Series([1, 2, 3], dtype="Int64"),
"B": Series([4, 5, 6], dtype="Int64"),
}
).dtypes
result = ser.to_json()
With the current changes, it passes. I think we expect this example to raise an error. I am not sure how to handle this.
|
BTW if you're using AI to help write PRs please acknowledge it in the OP, ideally with a description of which one (more specific is better) |
Yes, sure. I added the name of the AI tool I used to the PR description. I hope using such tools is allowed. |
Absolutely. In the PR using AI is fine. Its only in the comments we ask for exclusively human interaction. |
2 participants
The example below previously raised
OverflowError: Maximum recursion level reachedI used AI to help me with this pull request, specifically GPT-5.6 Thinking