BUG: DataFrame with empty categorical column saved as parquet is not correcly read with pyarrow engine

[julienT-livejourney]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df = pd.DataFrame(columns=["cat_column"], dtype="category")
df.info()
df.to_parquet("test.parquet", engine="pyarrow")
pd.read_parquet("test.parquet", engine="pyarrow").info()

Issue Description

A dataframe with an empty categorical column is not correctly read with pyarrow engine. After reading it, the column cat_column is no longer a categorical (it's an object).
Writing and reading with fastparquet has the expected behavior (cat_column is still categorical). If the dataframe is saved with pyarrow and read with fastparquet it has also the correct behavior (still a categorical data).

Expected Behavior

import pandas as pd

df = pd.DataFrame(columns=["cat_column"], dtype="category")
df.info()
df.to_parquet("test.parquet", engine="pyarrow")
pd.read_parquet("test.parquet", engine="fastparquet").info()

Installed Versions

INSTALLED VERSIONS

commit : 87cfe4e
python : 3.10.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 126 Stepping 5, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : fr_FR.cp1252

pandas : 1.5.0
numpy : 1.23.1
pytz : 2022.1
dateutil : 2.8.2
setuptools : 58.1.0
pip : 22.2.2
Cython : None
pytest : 7.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : 0.8.3
fsspec : 2022.8.2
gcsfs : None
matplotlib : 3.5.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 9.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.1
snappy : None
sqlalchemy : 1.4.41
tables : None
tabulate : 0.8.10
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None

[topper-123]

I can confirm this issue. In addition, if the categorical has one element, the roundtrip works:

>>> import pandas as pd
>>> df = pd.DataFrame(["a"], columns=["cat_column"], dtype="category")
>>> df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1 entries, 0 to 0
Data columns (total 1 columns):
 #   Column      Non-Null Count  Dtype
---  ------      --------------  -----
 0   cat_column  1 non-null      category
dtypes: category(1)
memory usage: 245.0 bytes
>>> df.to_parquet("test.parquet", engine="pyarrow")
>>> pd.read_parquet("test.parquet", engine="pyarrow").info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1 entries, 0 to 0
Data columns (total 1 columns):
 #   Column      Non-Null Count  Dtype
---  ------      --------------  -----
 0   cat_column  1 non-null      category
dtypes: category(1)
memory usage: 245.0 bytes

[adampinky85]

We encountered this issue today, the inconsistent dtype failed attempting to use the .cat series accessor. It would be great if this could be resolve pandas, thanks!

[narukaze132]

Just verified that this issue still exists in the current stable Pandas and PyArrow. Additionally, I tried encoding with fastparquet and decoding with pyarrow and the issue persisted, which seems to confirm that the problem occurs within pyarrow's decoding specifically and occurs regardless of what engine was used to encode it.

I'm going to go report this on apache/arrow, since the issue appears to be on their end.