Unable to specify "target_options" entry of "storage_options" in xarray backend

[will-s-hart]

I have been using the xarray backend to open ARISE-SAI data on AWS - e.g.,

xr.open_dataset(
    "s3://ncar-cesm2-arise/CESM2-WACCM-SSP245/metadata/fsspec/b.e21.BWSSP245cmip6.f09_g17.CMIP6-SSP2-4.5-WACCM.001.cam.h1.PRECT.20150101-20241231.json",
    chunks={},
    engine="kerchunk",
    storage_options={
        "remote_options": {"anon": True},
        "target_options": {"anon": True},
    },
)

This previously worked, but now fails - I think the issue is that since #516, "target_options" is no longer passed to fsspec.filesystem(). I think this would be fixed by simply allowing kerchunk.utils.refs_as_store() to take additional arbitrary keyword arguments, and then passing those to refs_as_fs().

If this is not possible, any suggestions for workarounds would be great (note that "target_options" does seem to be needed somewhere to open the data, as dropping it leads to a NoCredentialsError).

[martindurant]

I think this would be fixed by simply allowing kerchunk.utils.refs_as_store() to take additional arbitrary keyword arguments, and then passing those to refs_as_fs()

Sounds very reasonable - shall we try it and see what else breaks or not?

[will-s-hart]

Happy to open a PR if helpful

[martindurant]

yes, please - ideally with a test.

[will-s-hart]

OK opened a PR, not sure I know enough about the codebase to write a useful test but let me know if I can help further