-
Notifications
You must be signed in to change notification settings - Fork 1k
Open
Description
I really appreciate how fread and fwrite support gzip compression with basically no effort on the users part. I think it would be awesome if zstandard was also supported. https://github.com/facebook/zstd
Various online benchmarks have shown compression and decompression to be faster than gzip and with better compression ratios. Another big benefit is that zstd, unlike gzip, is splittable. That means you could take these CSVs and read them with Spark in parallel.
Thanks for considering!