Opening via xarray backendentrypoint

[TomNicholas]

Some changes are needed in xarray to support using the xarray backend entrypoint system to open datasets from disk as ManifestArray-backed Variables just by passing a keyword arg to open_dataset/open_mfdataset. This requires dodging some internal array wrapping that occurs in the depths of xarray's backend machinery.

Originally posted by @TomNicholas in https://github.com/TomNicholas/VirtualiZarr/issues/14#issuecomment-1998720573

[TomNicholas]

This would be cool, but it's actually not needed for an MVP, because we have open_dataset_via_kerchunk.

[TomNicholas]

Trying to get this to work leads down rabbit holes like the one in pydata/xarray#8712

[TomNicholas]

Note that to get the syntax open_dataset(file, engine='virtualizarr', indexes={}) to work as a way of avoiding creating indexes does not actually require that indexes be added to xr.open_dataset upstream (xref pydata/xarray#6633). That's because if the kwarg is not recognized by xarray it should be passed on to our backend engine, which already does support an indexes kwarg.

[TomNicholas]

I'm no longer so sure that we actually need this feature?

It's a bit of a pain to implement because it likely requires finicky upstream changes to xarray, and the benefit in the end is purely syntactic sugar, not actually new capability.

It's also weird design-wise for a few reasons:

1. The set of kwargs to xr.open_dataset and virtualizarr.open_virtual_dataset are related, but there are some arguments that make sense on the former but not the latter (e.g. chunks, cache), and some that make sense on the latter but not the former (e.g. loadable_variables).
2. Mostly in xarray the engine kwarg refers to the type of file you're trying to open. But here it would refer to the wrapped array type that gets returned, and would be valid for a range of filetypes.
3. If we wanted to allow 3rd parties to create their own kerchunk-like readers for virtualizarr without them all having to live in this codebase, we could imagine having an entrypoint system for virtualizarr. But combining that would mean xarray's backendentrypoint called a virtualizarr engine, which then calls more entrypoints...

I'm curious what others think, especially @ayushnag, @betolink, and @sharkinsspatial.

[maxrjones]

#124 provides a supported proposal for opening datasets using in-memory ManifestArrays, so closing this as not-planned. Please re-open if needed.