Skip to content

Error reading zarr store from an IPFS gateway #130

@lgloege

Description

@lgloege

Thank you for building intake-xarray, this is an awesome package! I am having trouble reading a zarr datastore via an IPFS gateway. I am trying to read a NOAA SST dataset with an intake catalog.

This code reads a NOAA SST dataset with just xarray

import xarray as xr
zarr_store = "https://dweb.link/ipfs/bafybeiepna7ilkhdwykd65i7aovmrapnmaifullesxg5pdsjy6cos5qfrm/"
ds = xr.open_dataset(zarr_store, engine="zarr")

This code works as expected. Now I want to build an intake catalog to read this file. I wrote the following simple catalog catalog_ipfs.yaml

---
plugins:
  source:
    - module: intake_xarray

sources:
  SST:      
    args:
      urlpath: https://dweb.link/ipfs/bafybeiepna7ilkhdwykd65i7aovmrapnmaifullesxg5pdsjy6cos5qfrm/
      xarray_kwargs: 
          engine: zarr
    driver: intake_xarray.netcdf.NetCDFSource

I then try reading data from it using the following code:

import intake
import xarray as xr
catolog = './catalog_ipfs.yaml'
cat = intake.open_catalog(catolog)
ds = cat.SST().read()

When I run this code I get this ValueError

ValueError: Starting with Zarr 2.11.0, stores must be subclasses of BaseStore, if your store exposes the MutableMapping interface wrap it in Zarr.storage.KVStore. Got <File-like object HTTPFileSystem, https://dweb.link/ipfs/bafybeiepna7ilkhdwykd65i7aovmrapnmaifullesxg5pdsjy6cos5qfrm/>

Any thoughts on how to resolve this issue? I am confused because I thought in the background intake_xarray was just doing this xr.open_dataset("https://dweb.link/ipfs/bafybeiepna7ilkhdwykd65i7aovmrapnmaifullesxg5pdsjy6cos5qfrm/", engine="zarr"), which I know works from the above example.

Here is the version of each package I am using:

intake.__version__ = '0.6.6'
intake_xarray.__version__ = '0.6.1'
xr.__version__ = '2022.6.0'
zarr.__version__ = '2.12.0'

I appreciate any help you can provide, thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions