Polars enables compression types including:
{‘lz4’, ‘uncompressed’, ‘snappy’, ‘gzip’, ‘lzo’, ‘brotli’, ‘zstd’}
Pandas enables compression types including:
None, ‘gzip’, ‘brotli’, ‘lz4’, ‘zstd’.
Pyspark enables compression types including:
(none, uncompressed, snappy, gzip, lzo, brotli, lz4, and zstd)
Tested and working:
- uncompressed
- lz4
- snappy
- gzip
- zstd
Tested and not working:
Untested:
I believe this is determined by the libraries in the base image. Using Ubuntu vs Alpine solved some but not all.