QST: how can I save my sparse dataframe with indexes and columns to a format other than .pkl?

[michelkluger]

Research

• I have searched the [pandas] tag on StackOverflow for similar questions.

• I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://stackoverflow.com/questions/60956268/save-sparse-pandas-dataframe-to-different-file-types

Question about pandas

I want have sparse dataframe structure saved in one file (to avoid proliferation of files)

how can I write and read and get the same sparse df? any alternative to pickle?

[rhshadrach]

Is your goal just to save disk space? If you can convert the sparse DataFrame to dense in memory, then you can use parquet which is very efficient on sparse data. In the example that you gave in SO, I'm seeing the dense DataFrame take up 18 megabytes in memory but only 1.5 megabytes on disk.

[michelkluger]

Is your goal just to save disk space? If you can convert the sparse DataFrame to dense in memory, then you can use parquet which is very efficient on sparse data. In the example that you gave in SO, I'm seeing the dense DataFrame take up 18 megabytes in memory but only 1.5 megabytes on disk.

Ideally I would like a way / format, to_,format method that is easy to read end write from sparse. If my matrix is too big, converting to dense will cause memory errors

[rhshadrach]

Makes sense - it seems to me that hdf would be the most appropriate, and for that there is #42070. Closing in favor of that issue.

[michelkluger]

it seems to me that hdf would be the most appropriate,

why?

[rhshadrach]

I do not believe any other file format can support it.