Skip to content

Improve the workarounds for handling pandas nullable dtypes in pandas<=2.1 #3596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 12, 2024
Merged

Improve the workarounds for handling pandas nullable dtypes in pandas<=2.1 #3596

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0aa73e6
Improve the workaround for handling pandas null dtypes in pandas<=2.1
seisman Dec 2, 2024
3f7eff2
Separate variable 'dtype' and 'numpy_dtype' for the input and result …
seisman Dec 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 25 additions & 12 deletions pygmt/clib/conversion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,19 +168,32 @@ def _to_numpy(data: Any) -> np.ndarray:
"date64[ms][pyarrow]": "datetime64[ms]",
}

# The dtype for the input object.
dtype = getattr(data, "dtype", getattr(data, "type", ""))
# The numpy dtype for the result numpy array, but can be None.
numpy_dtype = dtypes.get(str(dtype))

# pandas numeric dtypes were converted to np.object_ dtype prior pandas 2.2, and are
# converted to suitable NumPy dtypes since pandas 2.2. Refer to the following link
# for details: https://pandas.pydata.org/docs/whatsnew/v2.2.0.html#to-numpy-for-numpy-nullable-and-arrow-types-converts-to-suitable-numpy-dtype
#
# Workarounds for pandas < 2.2. Following SPEC 0, pandas 2.1 should be dropped in
# 2025 Q3, so it's likely we can remove the workaround in PyGMT v0.17.0.
if (
hasattr(data, "isna")
and data.isna().any()
and Version(pd.__version__) < Version("2.2")
):
# Workaround for dealing with pd.NA with pandas < 2.2.
# Bug report at: https://github.com/GenericMappingTools/pygmt/issues/2844
# Following SPEC0, pandas 2.1 will be dropped in 2025 Q3, so it's likely
# we can remove the workaround in PyGMT v0.17.0.
array = np.ascontiguousarray(data.astype(float))
else:
vec_dtype = str(getattr(data, "dtype", getattr(data, "type", "")))
array = np.ascontiguousarray(data, dtype=dtypes.get(vec_dtype))
Version(pd.__version__) < Version("2.2") # pandas < 2.2 only.
and hasattr(data, "dtype") # NumPy array or pandas objects only.
and hasattr(data.dtype, "numpy_dtype") # pandas dtypes only.
and data.dtype.kind in "iuf" # Numeric dtypes only.
): # pandas Series/Index with pandas nullable numeric dtypes.
# The numpy dtype of the result numpy array.
numpy_dtype = data.dtype.numpy_dtype
if getattr(data, "hasnans", False):
if data.dtype.kind in "iu":
# Integers with missing values are converted to float64.
numpy_dtype = np.float64
data = data.to_numpy(na_value=np.nan)

array = np.ascontiguousarray(data, dtype=numpy_dtype)

# Check if a np.object_ array can be converted to np.str_.
if array.dtype == np.object_:
Expand Down
Loading