Single store API for all backends, where the url chooses which backend

[kylebarron]

E.g. new_store(url) which could load from env, or new_store(url, kwargs)

[daviewales]

It would be amazing to be able to do something like:

import obstore as obs
from obstore.store import PrefixStore

store = PrefixStore.from_url('abfs://container@account.dfs.core.windows.net/path')
blobs = obs.list(store)
print(blobs)

i.e. Initialise a new PrefixStore from a URL in one line.
I've been hunting everywhere for the simplest method to list the blobs at an arbitrary location, but there's nothing as simple as this.

Even more amazing would be to simplify it at follows:

import obstore as obs
blobs = obs.list('abfs://container@account.dfs.core.windows.net/path')
print(blobs)

fsspec request something like the following:

from urllib.parse import urlparse
import fsspec

source_url = 'abfs://container@account.dfs.core.windows.net/path'
source = urlparse(source_url)
source_account = source.hostname.split(".", 1)[0] if source.hostname else None

fs = fsspec.filesystem(
    source.scheme,
    account_name=source_account,
    anon=False,
)
blobs = fs.ls(source_url)
print(blobs)

This is not terrible, but it requires me to manually extract all the components from the URL.

[kylebarron]

I spent a lot of time today on the API design here, and I have #185 and #179, which together should make it really easy to create a store by URL.

Here's an example with a test bucket of mine

new_store checks the scheme of the URL and passes it to the relevant class' from_url classmethod. And with #185 you won't need to use a PrefixStore anymore; the stores will automatically handle a prefix. And from_url automatically checks for any environment variables, so you can automatically load data even from a private bucket.

(feedback welcomed on the name of new_store)

[kylebarron]

Even more amazing would be to simplify it at follows:

import obstore as obs
blobs = obs.list('abfs://container@account.dfs.core.windows.net/path')
print(blobs)

I won't say never, but for the moment I think that's too much magic. Ideally something as simple as new_store(url) will be easy to do once and then pass into the list function. That also gives better typing as you can only pass the store object in the first parameter. Otherwise someone could get confused whether the first parameter is the store or the prefix to the store, or something like that. An overuse of strings is bad API design IMO and they should be confined to well-defined APIs like new_store.

[daviewales]

Thanks!