Support for cloud-based datastores? #23
Description
This looks like a super powerful tool, looking forward to using it! I'd love to implement an API abstraction for cloud datastores like BossDB or CloudVolume so that one could, in theory, generate peta-scale segmentation without having to download the data and reformat into n5/hdf.
These datastores tend to have client-side libraries that support numpy-like indexing: e.g:
# Import intern (pip install intern)
from intern import array
# Save a cutout to a numpy array in ZYX order:
em = array("bossdb://microns/minnie65_8x8x40/em")
data = em[19000:19016, 56298:57322, 79190:80214]My understanding is that this should be a simple drop-in replacement for the ws_path and ws_key if we had a class that looked something like this:
from intern import array
class BossDBAdapterFile:
def __init__(self, filepath: str):
self.array = array(filepath)
def __getitem__(self, groupname: str):
return self.array
...(I expect I've forgotten a few key APIs / organization, but the gist is this)
Is this something that you imagine is feasible? Desirable? My hypothesis is that this would be pretty straightforward and open up a ton of cloud-scale capability, but I may be misunderstanding. Maybe there's a better place to plug in here than "pretending" to be an n5 file?