Write partial file portions?

[severo]

Would it be possible (and would it make sense to you @platypii?) to be able to generate only part of a file? Ie: if we do a local edit to a table, know which parts of the Parquet file will be modified? It would work well when using CDC and Xet (see https://huggingface.co/blog/parquet-cdc).

I guess it's not straightforward, since an edit can change the page, the row group metadata, and the file metadata, but, similarly to hyparquet letting us read a localized part of a Parquet file, it would help us write a Parquet file surgically.

[platypii]

A lot of the primitives you would need are already here in hyparquet-writer. I haven't heard of re-writing parts of a parquet file. In theory, parquet was made to support to appending row groups, the idea was that you write your new rowgroups and then append a new metadata at the footer of the file. But since cloud storage (S3) doesn't support append, I haven't seen this feature used much either. It is interesting from a Xet point of view though. Open to suggestions!