BUG: to_parquet produces ValueError exception

[johanrex]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
d = {"col1": [1, 2], "col2": [3, 4]}
df = pd.DataFrame(data=d)
bytes_data = df.to_parquet(engine="fastparquet")

Issue Description

From the documentation for to_parquet (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) we can read this behavior for the path argument (default value is None):

If None, the result is returned as bytes.

When I run the above I get this exception:

Traceback (most recent call last):
  File "c:\git\useful-snippets\cryptography\tmp.py", line 4, in <module>
    bytes_data = df.to_parquet(engine="fastparquet")
  File "C:\Users\serexj\Miniconda3\envs\cryptography\lib\site-packages\pandas\util\_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\serexj\Miniconda3\envs\cryptography\lib\site-packages\pandas\core\frame.py", line 2976, in to_parquet
    return to_parquet(
  File "C:\Users\serexj\Miniconda3\envs\cryptography\lib\site-packages\pandas\io\parquet.py", line 442, in to_parquet
    return path_or_buf.getvalue()
ValueError: I/O operation on closed file.

Expected Behavior

Instead of an exception, return the result as bytes, as the documentation states.

Installed Versions

INSTALLED VERSIONS

commit : 2e218d1
python : 3.10.9.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 141 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : Swedish_Sweden.1252

pandas : 1.5.3
numpy : 1.24.1
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 65.6.3
pip : 22.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : 2023.1.0
fsspec : 2023.1.0
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None
None

[phofl]

Hi, thanks for your report. I think this is an upstream issue, the buffer is closed when fastparquet returns

[msSarriman]

Is there any workaround until this is resolved?

[MSfinnet]

I am also suffering from this one.

@msSarriman

Is there any workaround until this is resolved?

You could of course persist the data by writing a parquet file and go from there. You could read the bytes from file subsequently. But I assume you are looking for a way without persisting (as I am myself, too).

[msSarriman]

@MSfinnet I am looking in memory operations, yes.

I have attempted something like the following, but didn't test extensively:

import fsspec

df.to_parquet("memory://temp.parquet")
with fsspec.open("memory://temp.parquet", "rb") as f:
   response = f.read()

return response

[MSfinnet]

@msSarriman

@MSfinnet I am looking in memory operations, yes.

I have attempted something like the following, but didn't test extensively:

import fsspec

df.to_parquet("memory://temp.parquet")
with fsspec.open("memory://temp.parquet", "rb") as f:
   response = f.read()

return response

That's pretty much what I meant. I am far from an expert on this topic, so maybe there is another or better way out there.

Hoping for a quick fix. 🙈

[msSarriman]

@MSfinnet

Note that using "memory://", you are actually creating a filesystem in memory, so you aren't utilizing actual files. The operations will not reach out of the memory heap of your application.

Still though, this is a workaround to the original issue.

[johanrex]

@msSarriman that fsspec library seems super interesting for many use cases.

[johanrex]

@phofl how should upstream bugs be handled?

[phofl]

probably best to report upstream

[johanrex]

@phofl is fastparquet the upstream project? I'm not sure how I would be able to create a useful bug report on that project though since I'm only using it indirectly. I haven't worked on pandas internals so I'm not sure how it's being used.

[phofl]

Yes fast parquet ist the upstream project. You'd have to dig into their documentation or our usage. I am not really familiar with that part of the code myself

[MSfinnet]

Ah, do I understand correctly: The problem is exclusive to fastparquet?
So if you are not tied to that specific engine, simply use another one, i.d. pyarrow.

Works for me :-)

[johanrex]

@MSfinnet and @msSarriman yes, the error is exclusive to fastparquet. Pyarrow works.

bytes_data= df.to_parquet(engine="pyarrow") # works
bytes_data= df.to_parquet(engine="fastparquet") # fails

I'll use pyarrow as a workaround.