You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PDEP proposes a logical type system for pandas to abstract underlying library differences from end users, clarify the scope of pandas type support, and give pandas developers more flexibility to manage the implementation of types.
11
+
This PDEP proposes a logical type system for pandas which will decouple user semantics (i.e. _this column should be an integer_) from the pandas implementation/internal (i.e. _we will use NumPy/pyarrow/X to store this array_).By decoupling these through a logical type system, the expectation is that this PDEP will:
12
+
13
+
* Abstract underlying library differences from end users
14
+
* More clearly define the types of data pandas supports
15
+
* Allow pandas developers more flexibility to manage type implementations
16
+
* Pave the way for continued adoption of Arrow in the pandas code base
12
17
13
18
## Background
14
19
15
20
When pandas was originally built, the data types that it exposed were a subset of the NumPy type system. Starting back in version 0.23.0, pandas introduced Extension types, which it also began to use internally as a way of creating arrays instead of exclusively relying upon NumPy. Over the course of the 1.x releases, pandas began using pyarrow for string storage and in version 1.5.0 introduced the high level ``pd.ArrowDtype`` wrapper.
16
21
17
-
While these new type systems have brought about many great features, they have surfaced three major problems. The first is that we put the onus on users to understand the differences of the physical type implementations. Consider the many ways pandas allows you to create a "string":
22
+
While these new type systems have brought about many great features, they have surfaced three major problems.
23
+
24
+
### Problem 1: Inconsistent Type Naming / Behavior
25
+
26
+
There is no better example of our current type system being problematic than strings. Let's assess the number of string iterations a user could create (this is a non-exhaustive list):
Keeping track of all of these iterations and their subtle differences is difficult even for [core maintainers](https://github.com/pandas-dev/pandas/issues/58321).
40
+
``dtype="string"`` was the first truly new string implementation starting back in pandas 0.23.0, and it is a common pitfall for new users not to understand that there is a huge difference between that and ``dtype=str``. The pyarrow strings have trickled in in more recent memory, but also are very difficult to reason about. The fact that ``dtype="string[pyarrow]"`` is not the same as ``dtype=pd.ArrowDtype(pa.string()`` or ``dtype=pd.ArrowDtype(pa.large_string())`` was a surprise [to the author of this PDEP](https://github.com/pandas-dev/pandas/issues/58321).
41
+
42
+
While some of these are aliases, the main reason why we have so many different string dtypes is because we have historically used NumPy and created custom missing value solutions around the ``np.nan`` marker, which are incompatible with the ``pd.NA`` sentinel introduced a few years back. Our ``pd.StringDtype()`` uses the pd.NA sentinel, as do our pyarrow based solutions; bridging these into one unified solution has proven challenging.
43
+
44
+
To try and smooth over the different missing value semantics and how they affect the underlying type system, the status quo has been to add another string dtype. ``string[pyarrow_numpy]`` was an attempt to use pyarrow strings but adhere to NumPy nullability semantics, under the assumption that the latter offers maximum backwards compatibility. However, being the exclusive data type that uses pyarrow for storage but NumPy for nullability handling, this data type just adds more inconsistency to how we handle missing data, a problem we have been attempting to solve back since discussions around pandas2. The name ``string[pyarrow_numpy]`` is not descriptive to end users, and unless it is inferred requires users to explicitly ``.astype("string[pyarrow_numpy]")``, again putting a burden on end users to know what ``pyarrow_numpy`` means and to understand the missing value semantics of both systems.
31
45
32
-
The second problem is that the conventions for constructing types from a given type backend are inconsistent. Let's review string aliases used to construct certain types:
46
+
PDEP-14 has been proposed to smooth over that and change our ``pd.StringDtype()`` to be an alias for ``string[pyarrow_numpy]``. This would at least offer some abstraction to end users who just want strings, but on the flip side would be breaking behavior for users that have already opted into ``dtype="string"`` or ``dtype=pd.StringDtype()`` and the related pd.NA missing value marker for the prior 4 years of their existence.
47
+
48
+
A logical type system can help us abstract all of these issues. At the end of the day, this PDEP assumes a user wants a string data type. If they call ``Series.str.len()`` against a Series of that type with missing data, they should get back a Series with an integer data type.
49
+
50
+
### Problem 2: Inconsistent Constructors
51
+
52
+
The second problem is that the conventions for constructing types from the various _backends_ are inconsistent. Let's review string aliases used to construct certain types:
"string[pyarrow]" is excluded from the above table because it is misleading; while "int64[pyarrow]" definitely gives you a pyarrow backed string, "string[pyarrow]" gives you a pandas extension array which itself then uses pyarrow, which can introduce behavior differences (see [issue 58321](https://github.com/pandas-dev/pandas/issues/58321)).
59
+
"string[pyarrow]" is excluded from the above table because it is misleading; while "int64[pyarrow]" definitely gives you a pyarrow backed string, "string[pyarrow]" gives you a pandas extension array which itself then uses pyarrow. Subtleties like this then lead to behavior differences (see [issue 58321](https://github.com/pandas-dev/pandas/issues/58321)).
40
60
41
61
If you wanted to try and be more explicit about using pyarrow, you could use the ``pd.ArrowDtype`` wrapper. But this unfortunately exposes gaps when trying to use that pattern across all backends:
42
62
@@ -47,6 +67,8 @@ If you wanted to try and be more explicit about using pyarrow, you could use the
47
67
48
68
It would stand to reason in this approach that you could use a ``pd.DatetimeDtype()`` but no such type exists (there is a ``pd.DatetimeTZDtype`` which requires a timezone).
49
69
70
+
### Problem 3: Lack of Clarity on Type Support
71
+
50
72
The third issue is that the extent to which pandas may support any given type is unclear. Issue [#58307](https://github.com/pandas-dev/pandas/issues/58307) highlights one example. It would stand to reason that you could interchangeably use a pandas datetime64 and a pyarrow timestamp, but that is not always true. Another common example is the use of NumPy fixed length strings, which users commonly try to use even though we claim no real support for them (see [#5764](https://github.com/pandas-dev/pandas/issues/57645)).
51
73
52
74
## Assessing the Current Type System(s)
@@ -55,6 +77,8 @@ A best effort at visualizing the current type system(s) with types that we curre
55
77
56
78
## Proposal
57
79
80
+
### Proposed Logical Types
81
+
58
82
Derived from the hierarchical visual in the previous section, this PDEP proposes that pandas supports at least all of the following _logical_ types, excluding any type widths for brevity:
59
83
60
84
- Signed Integer
@@ -65,97 +89,109 @@ Derived from the hierarchical visual in the previous section, this PDEP proposes
65
89
- Date
66
90
- Datetime
67
91
- Duration
68
-
-Interval
92
+
-CalendarInterval
69
93
- Period
70
94
- Binary
71
95
- String
72
-
-Dictionary
96
+
-Map
73
97
- List
74
98
- Struct
99
+
- Interval
100
+
- Object
101
+
102
+
One of the major problems this PDEP has tried to highlight is the historical tendency of our team to "create more types" to solve existing problems. To minimize the need for that, this PDEP proposes re-using our existing extension types where possible, and only adding new ones where they do not exist.
103
+
104
+
The existing extension types which will become our "logical types" are:
105
+
106
+
- pd.StringDtype()
107
+
- pd.IntXXDtype()
108
+
- pd.UIntXXDtype()
109
+
- pd.FloatXXDtype()
110
+
- pd.BooleanDtype()
111
+
- pd.PeriodDtype(freq)
112
+
- pd.IntervalDtype()
113
+
114
+
To satisfy all of the types highlighted above, this would require the addition of:
115
+
116
+
- pd.DecimalDtype()
117
+
- pd.DateDtype()
118
+
- pd.DatetimeDtype(unit, tz)
119
+
- pd.Duration()
120
+
- pd.CalendarInterval()
121
+
- pd.BinaryDtype()
122
+
- pd.MapDtype() # or pd.DictDtype()
123
+
- pd.ListDtype()
124
+
- pd.StructDtype()
125
+
- pd.ObjectDtype()
126
+
127
+
The storage / backend to each of these types is left as an implementation detail. The fact that ``pd.StringDtype()`` may be backed by Arrow while ``pd.PeriodDtype()`` continues to be a custom solution is of no concern to the end user. Over time this will allow us to adopt more Arrow behind the scenes without breaking the front end for our end users, but _still_ giving us the flexibility to produce data types that Arrow will not implement (e.g. ``pd.ObjectDtype()``).
75
128
76
-
To ensure we maintain all of the current functionality of our existing type system(s), a base type structure would need to look something like:
129
+
The methods of each logical type are expected in turn to yield another logical type. This can enable us to smooth over differences between the NumPy and Arrow world, while also leveraging the best of both backends. To illustrate, let's look at some methods where the return type today deviates for end users depending on if they are using NumPy-backed data types or Arrow-backed data types. The equivalent PDEP-13 logical data type is presented as the last column:
130
+
131
+
| Method | NumPy-backed result | Arrow Backed result type | PDEP-13 result type |
The ``Series.dt.date`` example is worth an extra look - with a PDEP-13 logical type system in place we would theoretically have the ability to keep our default ``pd.DatetimeDtype()`` backed by our current NumPy-based array but leverage pyarrow for the ``Series.dt.date`` solution, rather than having to implement a DateArray ourselves.
138
+
139
+
While this PDEP proposes reusing existing extension types, it also necessitates extending those types with extra metadata:
Library is responsible for the array implementation
147
+
Who manages the data buffer - NumPy or pyarrow
85
148
"""
86
149
...
87
150
88
151
@property
89
152
def physical_type:
90
153
"""
91
-
How does the backend physically implement this logical type? i.e. our
92
-
logical type may be a "string" and we are using pyarrow underneath -
93
-
is it a pa.string(), pa.large_string(), pa.string_view() or something else?
154
+
For logical types which may have different implementations, what is the
155
+
actual implementation? For pyarrow strings this may mean pa.string() versus
156
+
pa.large_string() versrus pa.string_view(); for NumPy this may mean object
157
+
or their 2.0 string implementation.
94
158
"""
95
159
...
96
160
97
161
@property
98
-
defmissing_value_marker-> pd.NA|np.nan:
162
+
defna_marker-> pd.NA|np.nan|pd.NaT:
99
163
"""
100
164
Sentinel used to denote missing values
101
165
"""
102
166
...
103
167
```
104
168
105
-
The theory behind this PDEP is that most users _should not_ care about the physical type that is being used. But if the abstraction our logical type provides is too much, a user could at least inspect and potentially configure which physical type to use.
169
+
``na_marker`` is expected to be read-only (see next section). For advanced users that have a particular need for a storage type, they may be able to construct the data type via ``pd.StringDtype(data_manager=np)`` to assert NumPy managed storage. While the PDEP allows constructing in this fashion, operations against that data make no guarantees that they will respect the storage backend and are free to convert to whichever storage the internals of pandas considers optimal (Arrow will typically be preferred).
106
170
107
-
With regards to how we may expose such types to end users, there are two currently recognized proposals. The first would use factory functions to create the logical type, i.e. something like:
108
-
109
-
```python
110
-
pd.Series(["foo", "bar", "baz"], dtype=pd.string() # assumed common case
Note that the class-based approach would reuse existing classes like ``pd.StringDtype`` but change their purpose, whereas the factory function would more explicitly be a new approach. This is an area that requires more discussion amongst the team.
123
-
124
-
## String Type Arguments
125
-
126
-
This PDEP proposes that we maintain only a small subset of string arguments that can be used to construct logical types. Those string arguments are:
127
-
128
-
- intXX
129
-
- uintXX
130
-
- floatXX
131
-
- string
132
-
- datetime64[unit]
133
-
- datetime64[unit, tz]
134
-
135
-
However, new code should be encouraged to use the logical constructors outlined previously. Particularly for aggregate types, trying to encode all of the information into a string can become unwieldy. Instead, keyword argument use should be encouraged:
136
-
137
-
```python
138
-
pd.Series(dtype=pd.list(value_type=pd.string()))
139
-
```
171
+
### Missing Value Handling
140
172
141
-
## Bridging Type Systems
173
+
Missing value handling is a tricky area as developers are split between pd.NA semantics versus np.nan, and the transition path from one to the other is not always clear.
142
174
143
-
An interesting question arises when a user constructs two logical types withdiffering physical types. If one is backed by NumPy andthe other is backed by pyarrow, what should happen?
175
+
Because this PDEP proposes reuse of the existing pandas extension type system, the default missing value marker will consistently be ``pd.NA``. However, to help with backwards compatibility for users that heavily rely on the equality semantics of np.nan, an option of ``pd.na_marker = "legacy"`` can be set. This would mean that the missing value indicator for logical types would be:
144
176
145
-
This PDEP proposes the following backends should be prioritized in the following order (1. is the highest priority):
177
+
| Logical Type | Default Missing Value | Legacy Missing Value |
178
+
| pd.BooleanDtype() | pd.NA | np.nan |
179
+
| pd.IntXXType() | pd.NA | np.nan |
180
+
| pd.FloatXXType() | pd.NA | np.nan |
181
+
| pd.StringDtype() | pd.NA | np.nan |
182
+
| pd.DatetimeType() | pd.NA | pd.NaT |
146
183
147
-
1. Arrow
148
-
2. pandas
149
-
3. NumPy
184
+
However, all data types for which there is no legacy NumPy-backed equivalent will continue to use ``pd.NA``, even in "legacy" mode. Legacy is provided only for backwards compatibility, but pd.NA usage is encouraged going forward to give users one exclusive missing value indicator.
150
185
151
-
One reason for this is that Arrow represents the most efficient and assumedly least-lossy physical representations. An obvious example comes when a pyarrow int64 array with missing data gets added to a NumPy int64 array; casting to the latter would lose data. Another reason is that Arrow represents the fastest growing ecosystem of tooling, and the PDEP author believes improving pandas's interoperability within that landscape is extremely important.
186
+
### Transitioning from Current Constructors
152
187
153
-
Aside from the backend, the C standard rules for [implicit conversion](https://en.cppreference.com/w/c/language/conversion) should apply to the data buffer, i.e. adding a pyarrow int8 array to a NumPy uint64 array should produce a pyarrow uint64 array.
188
+
To maintain a consistent path forward, _all_ constructors with the implementation of this PDEP are expected to map to the logical types. This means that providing ``np.int64`` as the data type argument makes no guarantee that you actually get a NumPy managed storage buffer; pandas reserves the right to optimize as it sees fit and may decide instead to just pyarrow.
154
189
155
-
For more expensive conversions, pandas retains the right to throw warnings or even error out when two of the same logical typewith differing physical types is added. For example, attempting to do string concatenation of string arrays backed by pyarrow and Python objects may throw a ``PerformanceWarning``, or maybe even a ``MemoryError``if such a conversion exhausts the available system resources.
190
+
The theory behind this is that the majority of users are not expecting anything particular from NumPy to happen when they say ``dtype=np.int64``. The expectation is that a user just wants _integer_ data, and the ``np.int64``specification owes to the legacy of pandas' evolution.
156
191
157
-
The ``BaseType`` proposed above also has a propertyforthe ``missing_value_marker``. Operations that use two logical types with different missing value markers should raise, as there is no clear way to prioritize between the various sentinels.
192
+
This PDEP makes no guarantee that we will stay that way forever; it is certainly reasonable that a few years down the road we deprecate and fully stop support for backend-specifc constructors like ``np.int64`` or ``pd.ArrowDtype(pa.int64())``. However, for the execution of this PDEP, such an initiative is not in scope.
0 commit comments