You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
dtype="string[pyarrow_numpy]"# added in 2.1, deprecated in 2.3
36
38
dtype=pd.ArrowDtype(pa.string())
37
39
dtype=pd.ArrowDtype(pa.large_string())
38
40
```
39
41
40
-
``dtype="string"`` was the first truly new string implementation starting back in pandas 0.23.0, and it is a common pitfall for new users not to understand that there is a huge difference between that and ``dtype=str``. The pyarrow strings have trickled in in more recent memory, but also are very difficult to reason about. The fact that ``dtype="string[pyarrow]"`` is not the same as ``dtype=pd.ArrowDtype(pa.string()`` or ``dtype=pd.ArrowDtype(pa.large_string())`` was a surprise [to the author of this PDEP](https://github.com/pandas-dev/pandas/issues/58321).
42
+
``dtype="string"`` was the first truly new string implementation starting back in pandas 0.23.0, and it is a common pitfall for new users not to understand that there is a huge difference between that and ``dtype=str``. The pyarrow strings have trickled in in more recent releases, but also are very difficult to reason about. The fact that ``dtype="string[pyarrow]"`` is not the same as ``dtype=pd.ArrowDtype(pa.string()`` or ``dtype=pd.ArrowDtype(pa.large_string())`` was a surprise [to the author of this PDEP](https://github.com/pandas-dev/pandas/issues/58321).
41
43
42
-
While some of these are aliases, the main reason why we have so many different string dtypes is because we have historically used NumPy and created custom missing value solutions around the ``np.nan`` marker, which are incompatible with the ``pd.NA`` sentinel introduced a few years back. Our ``pd.StringDtype()`` uses the pd.NA sentinel, as do our pyarrow based solutions; bridging these into one unified solution has proven challenging.
44
+
While some of these are aliases, the main reason why we have so many different string dtypes is because we have historically used NumPy and created custom missing value solutions around the ``np.nan`` marker, which are incompatible with ``pd.NA``. Our ``pd.StringDtype()`` uses the pd.NA sentinel, as do our pyarrow based solutions; bridging these into one unified solution has proven challenging.
43
45
44
-
To try and smooth over the different missing value semantics and how they affect the underlying type system, the status quo has been to add another string dtype. ``string[pyarrow_numpy]`` was an attempt to use pyarrow strings but adhere to NumPy nullability semantics, under the assumption that the latter offers maximum backwards compatibility. However, being the exclusive data type that uses pyarrow for storage but NumPy for nullability handling, this data type just adds more inconsistency to how we handle missing data, a problem we have been attempting to solve back since discussions around pandas2. The name ``string[pyarrow_numpy]``is not descriptive to end users, and unless it is inferred requires users to explicitly ``.astype("string[pyarrow_numpy]")``, again putting a burden on end users to know what ``pyarrow_numpy`` means and to understand the missing value semantics of both systems.
46
+
To try and smooth over the different missing value semantics and how they affect the underlying type system, the status quo has always been to add another string dtype. With PDEP-14 we now have a "compatibility" string of ``pd.StringDtype("python|pyarrow", na_value=np.nan)``that makes a best effort to move users towards all the benefits of PyArrow strings (assuming pyarrow is installed) while retaining backwards-compatible missing value handling with ``np.nan``as the missing value marker. The usage of the ``pd.StringDtype`` in this manner is a good stepping stone towards the goals of this PDEP, although it is stuck in an "in-between" state without other types following suit.
45
47
46
-
PDEP-14 has been proposed to smooth over that and change our ``pd.StringDtype()``to be an alias for ``string[pyarrow_numpy]``. This would at least offer some abstraction to end users who just want strings, but on the flip side would be breaking behavior for users that have already opted into ``dtype="string"`` or ``dtype=pd.StringDtype()`` and the related pd.NA missing value marker for the prior 4 years of their existence.
48
+
For instance, if a user calls ``Series.value_counts()`` on the ``pd.StringDtype()``, the type of the returned Series can vary wildly, and in non-obvious ways:
47
49
48
-
A logical type system can help us abstract all of these issues. At the end of the day, this PDEP assumes a user wants a string data type. If they call ``Series.str.len()`` against a Series of that type with missing data, they should get back a Series with an integer data type.
It is also worth noting that different methods will return different data types. For a pyarrow-backed string with pd.NA, ``Series.value_counts()`` returns a ``int64[pyarrow]`` but ``Series.str.len()`` returns a ``pd.Int64Dtype()``.
62
+
63
+
A logical type system can help us abstract all of these issues. At the end of the day, this PDEP assumes a user wants a string data type. If they call ``Series.str.value_counts()`` against a Series of that type with missing data, they should get back a Series with an integer data type.
49
64
50
65
### Problem 2: Inconsistent Constructors
51
66
@@ -69,7 +84,7 @@ It would stand to reason in this approach that you could use a ``pd.DatetimeDtyp
69
84
70
85
### Problem 3: Lack of Clarity on Type Support
71
86
72
-
The third issue is that the extent to which pandas may support any given type is unclear. Issue [#58307](https://github.com/pandas-dev/pandas/issues/58307) highlights one example. It would stand to reason that you could interchangeably use a pandas datetime64 and a pyarrow timestamp, but that is not always true. Another common example is the use of NumPy fixed length strings, which users commonly try to use even though we claim no real support for them (see [#5764](https://github.com/pandas-dev/pandas/issues/57645)).
87
+
The third issue is that the extent to which pandas may support any given type is unclear. Issue [#58307](https://github.com/pandas-dev/pandas/issues/58307) highlights one example. It would stand to reason that you could interchangeably use a pandas datetime64 and a pyarrow timestamp, but that is not always true. Another example is the use of NumPy fixed length strings, which users commonly try to use even though we claim no real support for them (see [#5764](https://github.com/pandas-dev/pandas/issues/57645)).
73
88
74
89
## Assessing the Current Type System(s)
75
90
@@ -84,7 +99,7 @@ Derived from the hierarchical visual in the previous section, this PDEP proposes
84
99
- Signed Integer
85
100
- Unsigned Integer
86
101
- Floating Point
87
-
-Fixed Point
102
+
-Decimal
88
103
- Boolean
89
104
- Date
90
105
- Datetime
@@ -93,7 +108,7 @@ Derived from the hierarchical visual in the previous section, this PDEP proposes
93
108
- Period
94
109
- Binary
95
110
- String
96
-
-Map
111
+
-Dict
97
112
- List
98
113
- Struct
99
114
- Interval
@@ -120,10 +135,11 @@ To satisfy all of the types highlighted above, this would require the addition o
120
135
- pd.Duration()
121
136
- pd.CalendarInterval()
122
137
- pd.BinaryDtype()
123
-
- pd.MapDtype() # or pd.DictDtype()
138
+
- pd.DictDtype()
124
139
- pd.ListDtype()
125
140
- pd.StructDtype()
126
141
- pd.ObjectDtype()
142
+
- pd.NullDtype()
127
143
128
144
The storage / backend to each of these types is left as an implementation detail. The fact that ``pd.StringDtype()`` may be backed by Arrow while ``pd.PeriodDtype()`` continues to be a custom solution is of no concern to the end user. Over time this will allow us to adopt more Arrow behind the scenes without breaking the front end for our end users, but _still_ giving us the flexibility to produce data types that Arrow will not implement (e.g. ``pd.ObjectDtype()``).
129
145
@@ -137,43 +153,24 @@ The methods of each logical type are expected in turn to yield another logical t
137
153
138
154
The ``Series.dt.date`` example is worth an extra look - with a PDEP-13 logical type system in place we would theoretically have the ability to keep our default ``pd.DatetimeDtype()`` backed by our current NumPy-based array but leverage pyarrow for the ``Series.dt.date`` solution, rather than having to implement a DateArray ourselves.
139
155
140
-
While this PDEP proposes reusing existing extension types, it also necessitates extending those types with extra metadata:
156
+
To implement this PDEP, we expect all of the logical types to have at least the following metadata:
141
157
142
-
```python
143
-
classBaseType:
144
-
145
-
@property
146
-
def data_manager -> Literal["numpy", "pyarrow"]:
147
-
"""
148
-
Who manages the data buffer - NumPy or pyarrow
149
-
"""
150
-
...
151
-
152
-
@property
153
-
def physical_type:
154
-
"""
155
-
For logical types which may have different implementations, what is the
156
-
actual implementation? For pyarrow strings this may mean pa.string() versus
157
-
pa.large_string() versrus pa.string_view(); for NumPy this may mean object
158
-
or their 2.0 string implementation.
159
-
"""
160
-
...
161
-
162
-
@property
163
-
def na_marker -> pd.NA|np.nan|pd.NaT:
164
-
"""
165
-
Sentinel used to denote missing values
166
-
"""
167
-
...
168
-
```
158
+
* storage: Either "numpy" or "pyarrow". Describes the library used to create the data buffer
159
+
* physical_type: Can expose the physical type being used. As an example, StringDtype could return pa.string_view
160
+
* na_value: Either pd.NA, np.nan, or pd.NaT.
169
161
170
-
``na_marker`` is expected to be read-only (see next section). For advanced users that have a particular need for a storage type, they may be able to construct the data type via``pd.StringDtype(data_manager=np)`` to assert NumPy managed storage. While the PDEP allows constructing in this fashion, operations against that data make no guarantees that they will respect the storage backend and are free to convert to whichever storage the internals of pandas considers optimal (Arrow will typically be preferred).
162
+
While these attributes are exposed as construction arguments to end users, users are highly discouraged from trying to control them directly. Put explicitly, this PDEP allows a user to request a``pd.XXXDtype(storage="numpy")`` to request a NumPy-backed array, if possible. While pandas may respect that during construction, operations against that data make no guarantees that the storage backend will be persisted through, giving pandas the freedom to convert to whichever storage is internally optimal (Arrow will typically be preferred).
171
163
172
164
### Missing Value Handling
173
165
174
-
Missing value handling is a tricky area as developers are split between pd.NA semantics versus np.nan, and the transition path from one to the other is not always clear.
166
+
Missing value handling is a tricky area as developers are split between pd.NA semantics versus np.nan, and the transition path from one to the other is not always clear. This PDEP does not aim to "solve" that issue per se (for that discussion, please refer to PDEP-16), but aims to provide a go-forward path that strikes a reasonable balance between backwards compatibility and a consistent missing value approach in the future.
167
+
168
+
This PDEP proposes that the default missing value for logical types is ``pd.NA``. The reasoning is two-fold:
169
+
170
+
1. We are in many cases re-using extension types as logical types, which mostly use pd.NA (StrDtype and datetimes are the exception)
171
+
2. For new logical types that have nothing to do with NumPy, using np.nan as a missing value marker is an odd fit
175
172
176
-
Because this PDEP proposes reuse of the existing pandas extension type system, the default missing value marker will consistently be ``pd.NA``. However, to help with backwards compatibility for users that heavily rely on the equality semantics of np.nan, an option of ``pd.na_marker = "legacy"`` can be set. This would mean that the missing value indicator for logical types would be:
173
+
However, to help with backwards compatibility for users that heavily rely on the semantics of ``np.nan`` or ``pd.NaT``, an option of ``pd.na_value = "legacy"`` can be set. This would mean that the missing value indicator for logical types would be:
177
174
178
175
| Logical Type | Default Missing Value | Legacy Missing Value |
179
176
| pd.BooleanDtype() | pd.NA | np.nan |
@@ -182,17 +179,18 @@ Because this PDEP proposes reuse of the existing pandas extension type system, t
182
179
| pd.StringDtype() | pd.NA | np.nan |
183
180
| pd.DatetimeType() | pd.NA | pd.NaT |
184
181
185
-
However, all data types for which there is no legacy NumPy-backed equivalent will continue to use ``pd.NA``, even in "legacy" mode. Legacy is provided only for backwards compatibility, but pd.NA usage is encouraged going forward to give users one exclusive missing value indicator.
182
+
However, all data types for which there is no legacy NumPy-backed equivalent will continue to use ``pd.NA``, even in "legacy" mode. Legacy is provided only for backwards compatibility, but ``pd.NA`` usage is encouraged going forward to give users one exclusive missing value indicator and better align with the goals of PDEP-16.
186
183
187
184
### Transitioning from Current Constructors
188
185
189
-
To maintain a consistent path forward, _all_ constructors with the implementation of this PDEP are expected to map to the logical types. This means that providing ``np.int64`` as the data type argument makes no guarantee that you actually get a NumPy managed storage buffer; pandas reserves the right to optimize as it sees fit and may decide instead to just pyarrow.
186
+
To maintain a consistent path forward, _all_ constructors with the implementation of this PDEP are expected to map to the logical types. This means that providing ``np.int64`` as the data type argument makes no guarantee that you actually get a NumPy managed storage buffer; pandas reserves the right to optimize as it sees fit and may decide instead to use PyArrow.
190
187
191
188
The theory behind this is that the majority of users are not expecting anything particular from NumPy to happen when they say ``dtype=np.int64``. The expectation is that a user just wants _integer_ data, and the ``np.int64`` specification owes to the legacy of pandas' evolution.
192
189
193
-
This PDEP makes no guarantee that we will stay that way forever; it is certainly reasonable that a few years down the road we deprecate and fully stop support for backend-specifc constructors like ``np.int64`` or ``pd.ArrowDtype(pa.int64())``. However, for the execution of this PDEP, such an initiative is not in scope.
190
+
This PDEP makes no guarantee that we will stay that way forever; it is certainly reasonable that, in the future, we deprecate and fully stop support for backend-specifc constructors like ``np.int64`` or ``pd.ArrowDtype(pa.int64())``. However, for the execution of this PDEP, such an initiative is not in scope.
0 commit comments