API: Setting Arrow-backed dtypes by default

[datapythonista]

I've been using the new Arrow backed dtypes, and I'm a bit confused on how it is decided which backend is used. One example:

>>> with pandas.option_context("mode.dtype_backend", "pyarrow"):
...     pandas.Series([1, 2, 3, 4])
... 
0    1
1    2
2    3
3    4
dtype: int64

Why is setting the dtype_backend to pyarrow not enough to use Arrow in the Series constructor when no dtype is specified?

Also, when using for example read_csv:

>>> import pandas
>>> pandas.read_csv('test.csv').dtypes
name    object
age      int64
dtype: object
>>> pandas.read_csv('test.csv', use_nullable_dtypes=True).dtypes
name    string[python]
age              Int64
dtype: object
>>> with pandas.option_context("mode.dtype_backend", "pyarrow"):
...     pandas.read_csv('test.csv').dtypes
... 
name    object
age      int64
dtype: object
>>> with pandas.option_context("mode.dtype_backend", "pyarrow"):
...     pandas.read_csv('test.csv', use_nullable_dtypes=True).dtypes
... 
name    string[pyarrow]
age      int64[pyarrow]
dtype: object

Why again is not enough that the user set the backend to pyarrow to use Arrow dtypes, and needs to call use_nullable_dtypes? This s what we returned, which doesn't make sense to me:

	dtype_backend=None	dtype_backend=pyarrow
use_nullable_dtypes=False	NumPy	NumPy ???
use_nullable_dtypes=True	Arrow+NumPy nullables	Arrow

What I would expect:

	dtype_backend=None	dtype_backend=pyarrow
use_nullable_dtypes=False	NumPy	Arrow
use_nullable_dtypes=True	Arrow eventually, Arrow+Numpy nullables for now	Arrow

Sorry if I missed the discussion, maybe I'm just missing something. But I don't see what's the use case for a user to explicitly say they want Arrow types with the option, but still giving them NumPy backed series and dataframes... Is this something it was agreed, or we just didn't make the changes to have a more intuitive behavior?

CC: @mroeschke

[mroeschke]

Yeah it was discussed previously that this setting should only apply when use_nullable_dtype=True was set or df.convert_dtypes(). This was chosen instead of changing use_nullable_dtypes="pandas"|"pyarrow" for example

I think your first example should eventually return pyarrow backed types in the future for any method, but that would require a lot more changes.

[phofl]

Additionally, neither our own nullables nor pyarrow dtypes are yet implemented in our constructors

[datapythonista]

I see that for the constructor it's just that it's not implemented, I thought it was related to the IO readers decision. Sorry for mixing both things, I guess we agree on that.

For the readers, I've been reading the discussions, but I still couldn't find the reason why we want to ignore the user preference unless use_nullable_dtype is specified. The API seems quite cumbersome, and I see in the discussion in #50291 that @phofl seems to agree, and emphasizes that the behavior should be well documented, since it's not intuitive. For what I understand, seems like it was motivated by the original option being nullable_backend before being renamed to dtype_backend. Before the rename it made sense to use it if use_nullable_dtypes was True. I understand that decision in the historical context, but I think the final API after the rename is counter-intuitive for no reason.

Personally, I'd make the nullable_dtypes option and the use_nullable_dtype parameter just a value of dtype_backend (which could be a parameter too). Not proposing to change that now, but feels like we could have an option for numpy (now named pandas), another for custom-nullable (the mix of numpy nullables and the Arrow string type used now with use_nullable_dtype=True), and the last one pyarrow (which should be respected). That would probably simplify things IMO.

But regardless of that, I don't see a use case when a user says (with the option) "I want pyarrow dtypes", and we still give them numpy dtypes, because a mostly unrelated parameter use_nullable_dtypes is set. pyarrow dtypes can be nullable, but still that option is mostly unrelated, and having it to false would make users expect to have pyarrow dtypes without the NA mask, not numpy types.

Am I missing something? Is there an advantage on forcing the user to set the use_nullable_dtype even if it set the pyarrow backend option?

[mroeschke]

True, the coupling the use_nullable_dtypes keyword with the nullable_backend global option is heavily tied due to the historical renaming during development, inspired from the revelation of "pyarrow is an alternative nullable type implementation" to "pyarrow is generically a whole different type implementation".

I think @phofl expressed this before too, but an ideal end state is decoupling these two and only needing to set dtype_backend to 'numpy', 'pandas', 'pyarrow' (or similar) and getting numpy, pandas-nullable, and pyarrow-backed dtypes everywhere respectively and not needing use_nullable_dtypes at all

[datapythonista]

I see that parquet works as expected (as expected by me :)

>>> with pandas.option_context("mode.dtype_backend", "pyarrow"):
...     print(pandas.read_parquet('test.parquet').dtypes)
Date       string[pyarrow]
Open       double[pyarrow]
High       double[pyarrow]
Low        double[pyarrow]
Close      double[pyarrow]
Volume      int64[pyarrow]
OpenInt     int64[pyarrow]
dtype: object

But I test also ORC, and seems it's behaving like CSV.

Is there any objection to make all readers return parquet types if mode.dtype_backend=pyarrow (regardless of the parameter use_nullable_dtypes? @pandas-dev/pandas-core

[phofl]

I‘d prefer implementing this everywhere before creating an option that is not tied to a keyword. Right now it works in half or 2/3 of our methods/functions and it‘s kind of vodoo when it works and when it does not.

I think we used this approach to avoid having two keywords, e.g. use_nullable_dtypes and something like use_pyarrow_dtypes

[datapythonista]

Thanks for the feedback @phofl. I don't fully understand what you propose. What exactly works in half or two thirds of methods?

My understanding is that the dtype_backend option should affect mostly when data is created or loaded. For the constructor doesn't seem immediate to implement using it. But for the readers I think we can change the behavior to what read_parquet is doing relatively fast if there is agreement, no? And that would help avoid this magic of not understanding when it's working one way or another, and things would be better than the current state, no?

Or am I missing something?

[phofl]

If we don't tie the dtype_backend to a keyword, I would expect (as a user) that I'll get arrow-backed data everywhere. This is not the case right now, since constructors, among others, aren't implemented yet.

The alternative we considered was adding a dtype_backend keyword to every method where the option is supported, which is something I would prefer over the option magically working for some functions but not for others

[datapythonista]

I see your point, thanks for clarifying. Personally I think users will still expect that data is loaded with the Arrow backend, even if we create this (in my opinion arbitrary) link with a keyword. And it'd be better to give them as much of that even if we can't immediately use Arrow types for any data loaded into pandas.

I don't know what regular users would expect, but I don't think as a user I'd expect that the option will get me Arrow data "everywhere" in the sense that df['numpy_backed_column'] + 1 would return an Arrow backend column. I see the scope here being data loaded/imported into pandas, via I/O or from Python structures via constructors.

If you really think the current approach to make that option only relevant when use_nullable_dtypes is set to True (which as I said, I think it's a very bad idea from the user experience point of view), would you make read_parquet use NumPy types when dtype_backend='pyarrow' and use_nullable_dtype=False then?