Support obstore as storage for df.to_parquet()

[machichima]

Currently, using obstore's AsyncFsspecStore for df.to_parquet() does not work (no error but also no files uploaded). It seems like we only need to add write() method in the BufferedFileSimple class and use the obstore.put() there.

obstore/obstore/python/obstore/fsspec.py

Lines 177 to 186 in b40d59b

	def _open(self, path, mode="rb", **kwargs):
	"""Return raw bytes-mode file-like from the file-system"""
	return BufferedFileSimple(self, path, mode, **kwargs)
	class BufferedFileSimple(fsspec.spec.AbstractBufferedFile):
	def __init__(self, fs, path, mode="rb", **kwargs):
	if mode != "rb":
	raise ValueError("Only 'rb' mode is currently supported")
	super().__init__(fs, path, mode, **kwargs)

I would like ask if there is any existing progress on it? If not, I can help working on this.

[kylebarron]

A PR for this would be great!

cc @martindurant who implemented it in #63

[martindurant]

Does obstore support multi-part uploads, or would whole files need to be buffered in memory before a call to put()?

[kylebarron]

It uses multipart uploads automatically (there's some mention of this in https://developmentseed.org/obstore/latest/api/put/) (unless use_multipart is False).

You can pass a sync or async iterator to put

[martindurant]

You can pass a sync or async iterator to put

That's not much use from the file API, I'm afraid - you need to be able to write, send some data and return; so the iterator would be the wrong way around.

[machichima]

Hi @kylebarron ,

I tried it out using the iterator. However, just like what @martindurant metioned, the whole file will be bufferd in the memory, which is not ideal. Probably the Rust part needs to be updated to support this.

[martindurant]

At the moment, obstore doesn't implement a file API at all, I guess this is in the name. As well as buffering in memory, you could also use fsspec's simplecache to buffer to local file and then upload that - also not entirely a perfect solution.

[kylebarron]

At the moment, obstore doesn't implement a file API at all

obstore implements a file API for reading but it's not yet implemented for writing. See https://developmentseed.org/obstore/v0.3.0/api/file/

I'm open to having a similar API for writing that wraps https://docs.rs/object_store/latest/object_store/buffered/struct.BufWriter.html

[kylebarron]

so the iterator would be the wrong way around.

I'm not sure I fully understand this use case, but if the iterator is the wrong way around, then maybe instead of a "push-based" iterator, you could pass in a "pull-based" file-like object? That's already supported.

[machichima]

I'm not sure I fully understand this use case, but if the iterator is the wrong way around, then maybe instead of a "push-based" iterator, you could pass in a "pull-based" file-like object? That's already supported.

Hi,
The problem here is that to_parquet() will call write() multiple times, each time it will provide a chunk of data (bytes). So the thing we can pass to obstore.put() is either bytes or iterator, and not file-like object.
However, when calling obstore.put() multiple times, each with different data chunk, it overwrites the original files. Hence, what I can do now is to collect data chunks get in write() into one and send to obstore.put() at once, which load all the data into memory.

Please let me know if I understand anything wrong. Thanks!

[kylebarron]

Ah that makes sense (sorry, going through my notifications while pretty tired).

I think the easiest way to solve this is to implement the writable file API that https://docs.rs/object_store/latest/object_store/buffered/struct.BufWriter.html implements. (It should be pretty similar to the existing readable file API)

[kylebarron]

#167 is a stub that adds writable file support. We basically just need constructors to create the WritableFile/AsyncWritableFile and to decide on the API.

[kylebarron]

Also, once we get this working, we should add an example to the docs that it works with pandas (I assume your df.to_parquet example is Pandas)