Skip to content

Commit 344d58b

Browse files
WillAydWillAyd
committed
PDEP-13: The pandas Logical Type System
1 parent 0d12b44 commit 344d58b

File tree

1 file changed

+161
-0
lines changed

1 file changed

+161
-0
lines changed

‎web/pandas/pdeps/0013-logical-type-system.md

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
# PDEP-13: The pandas Logical Type System
2+
3+
- Created: 27 Apr 2024
4+
- Status: Draft
5+
- Discussion: [#58141](https://github.com/pandas-dev/pandas/issues/58141)
6+
- Author: [Will Ayd](https://github.com/willayd),
7+
- Revision: 1
8+
9+
## Abstract
10+
11+
This PDEP proposes a logical type system for pandas to abstract underlying library differences from end users, clarify the scope of pandas type support, and give pandas developers more flexibility to manage the implementation of types.
12+
13+
## Background
14+
15+
When pandas was originally built, the data types that it exposed were a subset of the NumPy type system. Starting back in version 0.23.0, pandas introduced Extension types, which it also began to use internally as a way of creating arrays instead of exclusively relying upon NumPy. Over the course of the 1.x releases, pandas began using pyarrow for string storage and in version 1.5.0 introduced the high level ``pd.ArrowDtype`` wrapper.
16+
17+
While these new type systems have brought about many great features, they have surfaced three major problems. The first is that we put the onus on users to understand the differences of the physical type implementations. Consider the many ways pandas allows you to create a "string":
18+
19+
```python
20+
dtype=object
21+
dtype=str
22+
dtype="string"
23+
dtype=pd.StringDtype()
24+
dtype=pd.StringDtype("pyarrow")
25+
dtype="string[pyarrow]"
26+
dtype="string[pyarrow_numpy]"
27+
dtype=pd.ArrowDtype(pa.string())
28+
```
29+
30+
Keeping track of all of these iterations and their subtle differences is difficult even for [core maintainers](https://github.com/pandas-dev/pandas/issues/58321).
31+
32+
The second problem is that the conventions for constructing types from a given type backend are inconsistent. Let's review string aliases used to construct certain types:
33+
34+
| logical type | NumPy | pandas extension | pyarrow |
35+
| int | "int64" | "Int64" | "int64[pyarrow]" |
36+
| string | N/A | "string" | N/A |
37+
| datetime | N/A | "datetime64[us]" | "timestamp[us][pyarrow]" |
38+
39+
"string[pyarrow]" is excluded from the above table because it is misleading; while "int64[pyarrow]" definitely gives you a pyarrow backed string, "string[pyarrow]" gives you a pandas extension array which itself then uses pyarrow, which can introduce behavior differences (see [issue 58321](https://github.com/pandas-dev/pandas/issues/58321)).
40+
41+
If you wanted to try and be more explicit about using pyarrow, you could use the ``pd.ArrowDtype`` wrapper. But this unfortunately exposes gaps when trying to use that pattern across all backends:
42+
43+
| logical type | NumPy | pandas extension | pyarrow |
44+
| int | np.int64 | pd.Int64Dtype() | pd.ArrowDtype(pa.int64()) |
45+
| string | N/A | pd.StringDtype() | pd.ArrowDtype(pa.string()) |
46+
| datetime | N/A | ??? | pd.ArrowDtype(pa.timestamp("us")) |
47+
48+
It would stand to reason in this approach that you could use a ``pd.DatetimeDtype()`` but no such type exists (there is a ``pd.DatetimeTZDtype`` which requires a timezone).
49+
50+
The third issue is that the extent to which pandas may support any given type is unclear. Issue [#58307](https://github.com/pandas-dev/pandas/issues/58307) highlights one example. It would stand to reason that you could interchangeably use a pandas datetime64 and a pyarrow timestamp, but that is not always true. Another common example is the use of NumPy fixed length strings, which users commonly try to use even though we claim no real support for them (see [#5764](https://github.com/pandas-dev/pandas/issues/57645)).
51+
52+
## Assessing the Current Type System(s)
53+
54+
A best effort at visualizing the current type system(s) with types that we currently "support" or reasonably may want to is shown [in this comment](https://github.com/pandas-dev/pandas/issues/58141#issuecomment-2047763186). Note that this does not include the ["pyarrow_numpy"](https://github.com/pandas-dev/pandas/pull/58451) string data type or the string data type that uses the NumPy 2.0 variable length string data type (see [comment](https://github.com/pandas-dev/pandas/issues/57073#issuecomment-2080798081)) as they are under active discussion.
55+
56+
## Proposal
57+
58+
Derived from the hierarchical visual in the previous section, this PDEP proposes that pandas supports at least all of the following _logical_ types, excluding any type widths for brevity:
59+
60+
- Signed Integer
61+
- Unsigned Integer
62+
- Floating Point
63+
- Fixed Point
64+
- Boolean
65+
- Date
66+
- Datetime
67+
- Duration
68+
- Interval
69+
- Period
70+
- Binary
71+
- String
72+
- Dictionary
73+
- List
74+
- Struct
75+
76+
To ensure we maintain all of the current functionality of our existing type system(s), a base type structure would need to look something like:
77+
78+
```python
79+
class BaseType:
80+
81+
@property
82+
def dtype_backend -> Literal["pandas", "numpy", "pyarrow"]:
83+
"""
84+
Library is responsible for the array implementation
85+
"""
86+
...
87+
88+
@property
89+
def physical_type:
90+
"""
91+
How does the backend physically implement this logical type? i.e. our
92+
logical type may be a "string" and we are using pyarrow underneath -
93+
is it a pa.string(), pa.large_string(), pa.string_view() or something else?
94+
"""
95+
...
96+
97+
@property
98+
def missing_value_marker -> pd.NA|np.nan:
99+
"""
100+
Sentinel used to denote missing values
101+
"""
102+
...
103+
```
104+
105+
The theory behind this PDEP is that most users /should not care/ about the physical type that is being used. But if the abstraction our logical type provides is too much, a user could at least inspect and potentially configure which physical type to use.
106+
107+
With regards to how we may expose such types to end users, there are two currently recognized proposals. The first would use factory functions to create the logical type, i.e. something like:
108+
109+
```python
110+
pd.Series(["foo", "bar", "baz"], dtype=pd.string() # assumed common case
111+
pd.Series(["foo", "bar", "baz"], dtype=pd.string(missing_value_marker=np.nan)
112+
pd.Series(["foo", "bar", "baz"], dtype=pd.string(physical_type=pa.string_view())
113+
```
114+
115+
Another approach would be to classes:
116+
117+
```python
118+
pd.Series(["foo", "bar", "baz"], dtype=pd.StringDtype()
119+
pd.Series(["foo", "bar", "baz"], dtype=pd.StringDtype(missing_value_marker=np.nan)
120+
pd.Series(["foo", "bar", "baz"], dtype=pd.StringDtype(physical_type=pa.string_view())
121+
```
122+
Note that the class-based approach would reuse existing classes like ``pd.StringDtype`` but change their purpose, whereas the factory function would more explicitly be a new approach. This is an area that requires more discussion amongst the team.
123+
124+
## String Type Arguments
125+
126+
This PDEP proposes that we maintain only a small subset of string arguments that can be used to construct logical types. Those string arguments are:
127+
128+
- intXX
129+
- uintXX
130+
- floatXX
131+
- string
132+
- datetime64[unit]
133+
- datetime64[unit, tz]
134+
135+
However, new code should be encouraged to use the logical constructors outlined previously. Particularly for aggregate types, trying to encode all of the information into a string can become unwieldy. Instead, keyword argument use should be encouraged:
136+
137+
```python
138+
pd.Series(dtype=pd.list(value_type=pd.string()))
139+
```
140+
141+
## Bridging Type Systems
142+
143+
An interesting question arises when a user constructs two logical types with differing physical types. If one is backed by NumPy and the other is backed by pyarrow, what should happen?
144+
145+
This PDEP proposes the following backends should be prioritized in the following order (1. is the highest priority):
146+
147+
1. Arrow
148+
2. pandas
149+
3. NumPy
150+
151+
One reason for this is that Arrow represents the most efficient and assumedly least-lossy physical representations. An obvious example comes when a pyarrow int64 array with missing data gets added to a NumPy int64 array; casting to the latter would lose data. Another reason is that Arrow represents the fastest growing ecosystem of tooling, and the PDEP author believes improving pandas's interoperability within that landscape is extremely important.
152+
153+
Aside from the backend, the C standard rules for [implicit conversion](https://en.cppreference.com/w/c/language/conversion) should apply to the data buffer, i.e. adding a pyarrow int8 array to a NumPy uint64 array should produce a pyarrow uint64 array.
154+
155+
For more expensive conversions, pandas retains the right to throw warnings or even error out when two of the same logical type with differing physical types is added. For example, attempting to do string concatenation of string arrays backed by pyarrow and Python objects may throw a ``PerformanceWarning``, or maybe even a ``MemoryError`` if such a conversion exhausts the available system resources.
156+
157+
The ``BaseType`` proposed above also has a property for the ``missing_value_marker``. Operations that use two logical types with different missing value markers should raise, as there is no clear way to prioritize between the various sentinels.
158+
159+
## PDEP-11 History
160+
161+
- 27 April 2024: Initial version

0 commit comments

Comments
 (0)