BUG: read_csv with engine=pyarrow and numpy-nullable dtype

[jbrockmendel]

• closes BUG: read_csv loses precision when engine='pyarrow' and dtype Int64 #56136 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Also makes this code path robust to always-distinguish behavior in #62040

[mroeschke]

From the original issue, do you know where we are introducing float to lose precision when wanting the result type to be int?

[jbrockmendel]

From the original issue, do you know where we are introducing float to lose precision when wanting the result type to be int?

In arrow_table_to_pandas the pyarrow[int64] columns get converted to np.float64, then in finalize_pandas_output that gets cast back to Int64.

[mroeschke]

OK I see, it's pyarrow.Table.to_pandas casting the int to float when there's null.

What if in arrow_table_to_pandas, we always provide fallback type_mapper={pyarrow ints : pandas nullable ints} to avoid the lossy conversions, then afterwards we cast the pandas nullable ints to the appropriate type?

[jbrockmendel]

That’s basically what this is currently doing, just not in that function since it is also called from other places.

I’m out of town for a few days. If you feel strongly that this logic should live inside that function I’ll move it when I get back

[jbrockmendel]

Looking at this again, I'm skeptical of moving the logic into arrow_table_to_pandas. The trouble is that between the table.to_pandas() and the .astype conversions, we have to do a bunch of other csv-keyword-specific stuff like set_index and column renaming. (Just opened #62087 to clean that up a bit). Shoe-horning all of that into arrow_table_to_pandas would make it a really big function in a way that i think is a net negative.

[mroeschke]

Sorry in #62053 (comment), I meant for arrow_table_to_pandas to just have this change

diff --git a/pandas/io/_util.py b/pandas/io/_util.py
index 6827fbe9c9..2e15bd3749 100644
--- a/pandas/io/_util.py
+++ b/pandas/io/_util.py
@@ -85,7 +85,14 @@ def arrow_table_to_pandas(
         else:
             types_mapper = None
     elif dtype_backend is lib.no_default or dtype_backend == "numpy":
-        types_mapper = None
+        # Avoid lossy conversion to float64
+        # Caller is responsible for converting to numpy type if needed
+        types_mapper = {
+            pa.int8(): pd.Int8Dtype(),
+            pa.int16(): pd.Int16Dtype(),
+            pa.int32(): pd.Int32Dtype(),
+            pa.int64(): pd.Int64Dtype(),
+        }
     else:
         raise NotImplementedError

And then each IO parser is responsible for manipulating this result based on the IO arguments

[jbrockmendel]

And then each IO parser is responsible for manipulating this result based on the IO arguments

That would mean adding that logic to each of the 7 places where arrow_table_to_pandas is called, so we would almost-certainly be better off having it centralized.

If we get #62087 in then moving all the logic into arrow_table_to_pandas at least gets a little bit less bulky, so I can give it a try.