Explore use of obstore

[astrojuanlu]

Hi! I learned about Dataplug today at EuroSciPy from @macarronesc, looks very interesting.

One of the questions asked by the audience was "how to use this with non-AWS storage", and the answer was to write a custom plugin. I notice that Dataplug uses boto3, and I wonder if the project could benefit from obstore https://developmentseed.org/obstore/, which is cloud-agnostic, fast (thanks to Rust), and maybe fixes the pickle-ability issues?

dataplug/dataplug/storage/picklableS3.py

Lines 37 to 42 in 3f1eca1

	class PickleableS3ClientProxy:
	"""
	A Pickleable S3 client proxy that can be pickled and unpickled.
	Dataplug requires having an S3 client that can be pickled and sent remotely to workers, so that
	remote workers can access S3 objects. This class is a proxy to an S3 client that can be pickled.

See https://developmentseed.org/obstore/latest/alternatives/#obstore-vs-boto3