Reading kerchunk references in using the Zarr v3 engine.

[betolink]

In the current documentation we have this snippet to read a kerchunk file:

backend_args = {"consolidated": False, "storage_options": {"fo": d, "remote_protocol": "s3","remote_options": {"anon": True}}}
print(xr.open_dataset("reference://", engine="zarr", backend_kwargs=backend_args))

This throws an exception ValueError: Reference-FS's target filesystem must have same value of asynchronous if we have Zarr V3 installed. Seems like the async code in the V3 engine doesn't know how to open the reference:// mapper in async mode?

What would be the correct way of opening a kerchunk reference in xarray using Zarr v3 with fsspec?

cc @martindurant @maxrjones

[maxrjones]

This is one way (note asynchronous=True in remote_options).:

import json

import xarray as xr

with open("mur_kerchunk.json", "r") as f:
    refs = json.load(f)

backend_args = {"consolidated": False, "storage_options": {"fo": refs, "remote_protocol": "s3", "remote_options": {"anon": True, "asynchronous": True, }}}
ds = xr.open_dataset("reference://", engine="zarr", backend_kwargs=backend_args)
print(ds)

or another:

import json

import fsspec
import xarray as xr
import zarr
from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper

with open("mur_kerchunk.json", "r") as f:
    refs = json.load(f)

fs = fsspec.filesystem("reference", fo=refs, remote_protocol = "s3", remote_options={"anon": True})
store = zarr.storage.FsspecStore(AsyncFileSystemWrapper(fs), read_only=True)

ds = xr.open_zarr(store, zarr_format=2, consolidated=False)
print(ds)

[martindurant]

Thank you @maxrjones . We need to document this! We do expect engine="kerchunk" to do the right thing, thought, right?

AsyncFileSystemWrapper(fs)

Why is the referenceFS not already asynchronous?

[maxrjones]

You're right, the AsyncFileSystemWrapper in not necessary because FsspecStore automatically transforms the filesystem. But the initial filesystem is not already asynchronous:

import json

import fsspec
import xarray as xr
import zarr

with open("/Users/max/Documents/Code/zarr-developers/VirtualiZarr/data/mur_kerchunk.json", "r") as f:
    refs = json.load(f)

fs = fsspec.filesystem("reference", fo=refs, remote_protocol = "s3", remote_options={"anon": True})
print(fs.asynchronous)
store = zarr.storage.FsspecStore(fs, read_only=True)

ds = xr.open_zarr(store, zarr_format=2, consolidated=False)

False

[martindurant]

Like

fs = fsspec.filesystem("reference", fo=refs, remote_protocol = "s3", remote_options={"anon": True}, asynchronous=True)

?

[betolink]

Just FYI, this worked:

fs = fsspec.filesystem("reference",
                       fo=metadata,
                       remote_protocol="s3",
                       asynchronous=True,
                       remote_options={"asynchronous": True, **daac_fs.storage_options}
                      )
store = zarr.storage.FsspecStore(fs, read_only=True)
ds = xr.open_zarr(store, zarr_format=2, consolidated=False)

notice the asynchronous in both places, without it I got a warning. cc @maxrjones

[martindurant]

The only difference here: the "target" filesystem parameters are not needed, since you pass existing metadata. Reading from a target filesystem (which is sync) would be a problem here, and that complicates parquet-based references.