Inconsistent behavior between Parquet and JSON when chunks are missing

[ashiklom]

Taking the first file from here (https://noaa-goes17.s3.amazonaws.com/index.html#ABI-L1b-RadF/2022/001/00/) as an example:

The following code:

import kerchunk.hdf
import json
                                                                                                                                                     
fname = "OR_ABI-L1b-RadF-M6C01_G17_s20220010000320_e20220010009386_c20220010009424.nc"
                                                                                                                                                     
h5chunks = kerchunk.hdf.SingleHdf5ToZarr(fname)
refs = h5chunks.translate()
                                                                                                                                                     
with open("test.json", "w") as f:
    f.write(json.dumps(refs, indent=2))

Produces the following JSON output (excerpt; slightly clipped):

"..."
"Rad/.zarray": "{\"chunks\":[226,226],\"compressor\":null,\"dtype\":\"<i2\",\"fill_value\":1023, ..."
"Rad/.zattrs": "{\"_ARRAY_DIMENSIONS\":[\"y\",\"x\"],\"_Unsigned\":\"true\",\"add_offset\":-25.9"
"Rad/0.16": "base64:eAHt0DENAAAAAqDD/pk1iIwGNMWAAQMGDBgwYMCAAQMGDBgwYMCAAQMGDBgwYMCAAQMGDBgwYMCA..."
"Rad/0.17": [
  "/css/geostationary/BackStage/GOES-17-ABI-L1B-FULLD/2022/001/00/OR_ABI-L1b-RadF-M6C01_G17_s202..."
  51538,
  1448
],
"Rad/0.18": [
  "/css/geostationary/BackStage/GOES-17-ABI-L1B-FULLD/2022/001/00/OR_ABI-L1b-RadF-M6C01_G17_s202..."
  52986,
  4155
],
"Rad/0.19": [
  "/css/geostationary/BackStage/GOES-17-ABI-L1B-FULLD/2022/001/00/OR_ABI-L1b-RadF-M6C01_G17_s202..."
  57141,
  5554
],
"Rad/0.20": [
  "/css/geostationary/BackStage/GOES-17-ABI-L1B-FULLD/2022/001/00/OR_ABI-L1b-RadF-M6C01_G17_s202..."
  22412,
  7527
],
"..."

Note that the radiance chunks begin at 0.16 --- there is no Rad/0.{0--15}. That's weird --- I'm assuming this is some HDF5 sparse data cleverness. But in any case, xarray.open_dataset("test.json", engine="kerchunk") and subsequent summarizing of the entire Rad array (dat.Rad.mean().values) works fine here.

However, if you spit this out as a Parquet dataset, then it produces a file with rows 0-15 containing nan paths and 0 values, and then the real data start at row 16. That's fine...except that reading that Parquet file fails with an error like this (full backtrace in details):

  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/mapping.py", line 105, in getitems
    out = self.fs.cat(keys2, on_error=oe)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py", line 836, in cat
    proto_dict = _protocol_groups(path, self.references)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py", line 52, in _protocol_groups
    protocol = _prot_in_references(path, references)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py", line 44, in _prot_in_references
    return split_protocol(ref[0])[0] if ref[0] else ref[0]
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/core.py", line 544, in split_protocol
    if "://" in urlpath:
       ^^^^^^^^^^^^^^^^
TypeError: argument of type 'float' is not iterable

I've traced this back to a references.get("Rad/0.0") call that returns a nan "url" that can't be parsed by subsequent code. Here's some relevant pdb traces:

> /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/core.py(544)split_protocol()
-> if "://" in urlpath:
(Pdb) u
> /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py(44)_prot_in_references()
-> return split_protocol(ref[0])[0] if ref[0] else ref[0]
(Pdb) ll
 41     def _prot_in_references(path, references):
 42         ref = references.get(path)
 43         if isinstance(ref, (list, tuple)):
 44  ->         return split_protocol(ref[0])[0] if ref[0] else ref[0]
(Pdb) p ref
[nan]
(Pdb) p path
'Rad/0.0.0'
(Pdb)

Traceback (most recent call last):
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/read.py", line 6, in <module>
    print(combined_ds.Rad.mean().values)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/_aggregations.py", line 1664, in mean
    return self.reduce(
           ^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/dataarray.py", line 3826, in reduce
    var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/variable.py", line 1663, in reduce
    result = super().reduce(
             ^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/namedarray/core.py", line 912, in reduce
    data = func(self.data, **kwargs)
                ^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/variable.py", line 449, in data
    return self._data.get_duck_array()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/indexing.py", line 837, in get_duck_a
rray
    self._ensure_cached()
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/indexing.py", line 831, in _ensure_ca
ched
    self.array = as_indexable(self.array.get_duck_array())
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/indexing.py", line 788, in get_duck_a
rray
    return self.array.get_duck_array()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/indexing.py", line 837, in get_duck_a
rray
    self._ensure_cached()
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/indexing.py", line 831, in _ensure_ca
ched
    self.array = as_indexable(self.array.get_duck_array())
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/indexing.py", line 788, in get_duck_a
rray
    return self.array.get_duck_array()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/indexing.py", line 658, in get_duck_a
rray
    array = array.get_duck_array()
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/coding/variables.py", line 81, in get_duck_array
    return self.func(self.array.get_duck_array())
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/indexing.py", line 651, in get_duck_array
    array = self.array[self.key]
            ~~~~~~~~~~^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/backends/zarr.py", line 104, in __getitem__
    return indexing.explicit_indexing_adapter(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/core/indexing.py", line 1015, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/backends/zarr.py", line 94, in _getitem
    return self._array[key]
           ~~~~~~~~~~~^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/zarr/core.py", line 798, in __getitem__
    result = self.get_orthogonal_selection(pure_selection, fields=fields)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/zarr/core.py", line 1080, in get_orthogonal_selection
    return self._get_selection(indexer=indexer, out=out, fields=fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/zarr/core.py", line 1343, in _get_selection
    self._chunk_getitems(
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/zarr/core.py", line 2179, in _chunk_getitems
    cdatas = self.chunk_store.getitems(ckeys, contexts=contexts)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/zarr/storage.py", line 1426, in getitems
    results_transformed = self.map.getitems(list(keys_transformed), on_error="return")
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/mapping.py", line 105, in getitems
    out = self.fs.cat(keys2, on_error=oe)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py", line 836, in cat
    proto_dict = _protocol_groups(path, self.references)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py", line 52, in _protocol_groups
    protocol = _prot_in_references(path, references)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py", line 44, in _prot_in_references
    return split_protocol(ref[0])[0] if ref[0] else ref[0]
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/core.py", line 544, in split_protocol
    if "://" in urlpath:
       ^^^^^^^^^^^^^^^^
TypeError: argument of type 'float' is not iterable

[martindurant]

Thanks for the report, I'll look into it. What version of fsspec and kerchunk do you have?

[ashiklom]

Thanks!

  name: kerchunk
  version: 0.2.6
  url: https://files.pythonhosted.org/packages/82/8c/117feba86c75d42b8434f8e65085165df4cd415c754bcde3df007f036293/kerchunk-0.2.6-py3-none-any.whl
  sha256: dc55fcea6560688ffc2390ff5882847fdf736982c1817553632a2bd7eb59de73

  name: fsspec
  version: 2024.6.1
  url: https://files.pythonhosted.org/packages/5e/44/73bea497ac69bafde2ee4269292fa3b41f1198f4bb7bbaaabde30ad29d4a/fsspec-2024.6.1-py3-none-any.whl
  sha256: 3cb443f8bcd2efb31295a5b9fdb02aee81d8452c80d28f97a6d0959e6cee101e

In case it's useful, full pixi lockfile is in details below:

version: 5
environments:
  default:
    channels:
    - url: https://conda.anaconda.org/conda-forge/
    indexes:
    - https://pypi.org/simple
    packages:
      linux-64:
      - conda: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
      - conda: https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2
      - conda: https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-h4bc722e_7.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/c-ares-1.32.3-h4bc722e_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2024.7.4-hbcca054_0.conda
      - conda: https://conda.anaconda.org/conda-forge/noarch/cached-property-1.5.2-hd8ed1ab_1.tar.bz2
      - conda: https://conda.anaconda.org/conda-forge/noarch/cached_property-1.5.2-pyha770c72_1.tar.bz2
      - conda: https://conda.anaconda.org/conda-forge/linux-64/cftime-1.6.4-py312h085067d_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/h5py-3.11.0-nompi_py312hb7ab980_102.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/hdf5-1.14.3-nompi_hdf9ad27_105.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/keyutils-1.6.1-h166bdaf_0.tar.bz2
      - conda: https://conda.anaconda.org/conda-forge/linux-64/krb5-1.21.3-h659f571_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.40-hf3520f5_7.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libaec-1.1.3-h59595ed_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libblas-3.9.0-23_linux64_openblas.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libcblas-3.9.0-23_linux64_openblas.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libcurl-8.9.1-hdb1bdb2_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20191231-he28a2e2_2.tar.bz2
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libev-4.33-hd590300_2.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libexpat-2.6.2-h59595ed_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-14.1.0-h77fa898_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libgfortran-ng-14.1.0-h69a702a_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libgfortran5-14.1.0-hc5f4f2c_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libgomp-14.1.0-h77fa898_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.9.0-23_linux64_openblas.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libnghttp2-1.58.0-h47da74e_1.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.1-hd590300_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libopenblas-0.3.27-pthreads_hac2b453_1.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.46.0-hde9e2c9_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libssh2-1.11.0-h0841786_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-14.1.0-hc0a3c3a_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.38.1-h0b41bf4_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libxcrypt-4.4.36-hd590300_1.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.3.1-h4ab18f5_1.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.5-h59595ed_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/numpy-2.0.1-py312h1103770_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/openssl-3.3.1-h4bc722e_2.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/python-3.12.5-h2ad013b_0_cpython.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.12-4_cp312.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/readline-8.2-h8228510_1.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_h4845f30_101.conda
      - conda: https://conda.anaconda.org/conda-forge/noarch/tzdata-2024a-h0c530f3_0.conda
      - conda: https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2
      - conda: https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.6-ha6fb4c9_0.conda
      - pypi: https://files.pythonhosted.org/packages/2d/6a/885bc91484e1aa8f618f6f0228d76d0e67000b0fdd6090673b777e311913/asciitree-0.3.3.tar.gz
      - pypi: https://files.pythonhosted.org/packages/45/86/4736ac618d82a20d87d2f92ae19441ebc7ac9e7a581d7e58bbe79233b24a/asttokens-2.4.1-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/84/6a/4825e220b2efb41e16293c0a8ca5d3682973abab44eee07a12362c0ae979/cramjam-2.8.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
      - pypi: https://files.pythonhosted.org/packages/d5/50/83c593b07763e1161326b3b8c6686f0f4b0f24d5526546bee538c89837d6/decorator-5.1.1-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/80/03/6ea8b1b2a5ab40a7a60dc464d3daa7aa546e0a74d74a9f8ff551ea7905db/executing-2.0.1-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/61/bf/fd60001b3abc5222d8eaa4a204cd8c0ae78e75adc688f33ce4bf25b7fafa/fasteners-0.19-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/5d/5a/5dc0de7208f33bcd3223836614eab9ba1cfcd69147010827dab1eabea573/fastparquet-2024.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
      - pypi: https://files.pythonhosted.org/packages/5e/44/73bea497ac69bafde2ee4269292fa3b41f1198f4bb7bbaaabde30ad29d4a/fsspec-2024.6.1-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/68/2d/63851081b19d1ccf245091255797cb358c53c886609b5056da5457f7dbbf/h5netcdf-1.3.0-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/73/48/4d2818054671bb272d1b12ca65748a4145dc602a463683b5c21b260becee/ipython-8.26.0-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/20/9f/bc63f0f0737ad7a60800bfd472a4836661adae21f9c2535f3957b1e54ceb/jedi-0.19.1-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/82/8c/117feba86c75d42b8434f8e65085165df4cd415c754bcde3df007f036293/kerchunk-0.2.6-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/8f/8e/9ad090d3553c280a8060fbf6e24dc1c0c29704ee7d1c372f0c174aa59285/matplotlib_inline-0.1.7-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/85/dd/c89fd8966b69ab4d6c6bd7496fb18bdf920f23dd9bcaec8c7ecce9871d0b/numcodecs-0.13.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
      - pypi: https://files.pythonhosted.org/packages/08/aa/cc0199a5f0ad350994d660967a8efb233fe0416e4639146c089643407ce6/packaging-24.1-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/40/10/79e52ef01dfeb1c1ca47a109a01a248754ebe990e159a844ece12914de83/pandas-2.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
      - pypi: https://files.pythonhosted.org/packages/c6/ac/dac4a63f978e4dcb3c6d3a78c4d8e0192a113d288502a1216950c41b1027/parso-0.8.4-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/9e/c3/059298687310d527a58bb01f3b1965787ee3b40dce76752eda8b44e9a2c5/pexpect-4.9.0-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/e8/23/22750c4b768f09386d1c3cc4337953e8936f48a888fa6dddfb669b2c9088/prompt_toolkit-3.0.47-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/f7/3f/01c8b82017c199075f8f788d0d906b9ffbbc5a47dc9918a945e13d5a2bda/pygments-2.18.0-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/9c/3d/a121f284241f08268b21359bd425f7d4825cffc5ac5cd0e1b3d82ffd2b10/pytz-2024.1-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/f1/7b/ce1eafaf1a76852e2ec9b22edecf1daa58175c090266e9f6c64afcd81d91/stack_data-0.6.3-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/00/c0/8f5d070730d7836adc9c9b6408dec68c6ced86b304a9b26a14df072a6e8c/traitlets-5.14.3-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/65/58/f9c9e6be752e9fcb8b6a0ee9fb87e6e7a1f6bcab2cdc73f02bb7ba91ada0/tzdata-2024.1-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/04/81/668707e5f2177791869b624be4c06fb2473bf97ee33296b18d1cf3092af7/ujson-5.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
      - pypi: https://files.pythonhosted.org/packages/63/a2/d8fe0f7e233c5e0201442d82279f3a0ffb30edcb8d7c4c5eba695c16647f/universal_pathlib-0.2.2-py3-none-any.whl
      - pypi: git+https://github.com/zarr-developers/VirtualiZarr@fdab54cca8d73af4e44903091e5e00bbeb1bb31c
      - pypi: https://files.pythonhosted.org/packages/fd/84/fd2ba7aafacbad3c4201d395674fc6348826569da3c0937e75505ead3528/wcwidth-0.2.13-py2.py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/45/95/233e1f9c939f5ba314297315df709e6a5e823bf3cade7211991b15aa65d2/xarray-2024.7.0-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/5d/bd/8d881d8ca6d80fcb8da2b2f94f8855384daf649499ddfba78ffd1ee2caa3/zarr-2.18.2-py3-none-any.whl
packages:
- kind: conda
  name: _libgcc_mutex
  version: '0.1'
  build: conda_forge
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
  sha256: fe51de6107f9edc7aa4f786a70f4a883943bc9d39b3bb7307c04c41410990726
  md5: d7c89558ba9fa0495403155b64376d81
  license: None
  purls: []
  size: 2562
  timestamp: 1578324546067
- kind: conda
  name: _openmp_mutex
  version: '4.5'
  build: 2_gnu
  build_number: 16
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2
  sha256: fbe2c5e56a653bebb982eda4876a9178aedfc2b545f25d0ce9c4c0b508253d22
  md5: 73aaf86a425cc6e73fcf236a5a46396d
  depends:
  - _libgcc_mutex 0.1 conda_forge
  - libgomp >=7.5.0
  constrains:
  - openmp_impl 9999
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 23621
  timestamp: 1650670423406
- kind: pypi
  name: asciitree
  version: 0.3.3
  url: https://files.pythonhosted.org/packages/2d/6a/885bc91484e1aa8f618f6f0228d76d0e67000b0fdd6090673b777e311913/asciitree-0.3.3.tar.gz
  sha256: 4aa4b9b649f85e3fcb343363d97564aa1fb62e249677f2e18a96765145cc0f6e
- kind: pypi
  name: asttokens
  version: 2.4.1
  url: https://files.pythonhosted.org/packages/45/86/4736ac618d82a20d87d2f92ae19441ebc7ac9e7a581d7e58bbe79233b24a/asttokens-2.4.1-py2.py3-none-any.whl
  sha256: 051ed49c3dcae8913ea7cd08e46a606dba30b79993209636c4875bc1d637bc24
  requires_dist:
  - six>=1.12.0
  - typing ; python_version < '3.5'
  - astroid<2,>=1 ; python_version < '3' and extra == 'astroid'
  - astroid<4,>=2 ; python_version >= '3' and extra == 'astroid'
  - pytest ; extra == 'test'
  - astroid<2,>=1 ; python_version < '3' and extra == 'test'
  - astroid<4,>=2 ; python_version >= '3' and extra == 'test'
- kind: conda
  name: bzip2
  version: 1.0.8
  build: h4bc722e_7
  build_number: 7
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-h4bc722e_7.conda
  sha256: 5ced96500d945fb286c9c838e54fa759aa04a7129c59800f0846b4335cee770d
  md5: 62ee74e96c5ebb0af99386de58cf9553
  depends:
  - __glibc >=2.17,<3.0.a0
  - libgcc-ng >=12
  license: bzip2-1.0.6
  license_family: BSD
  purls: []
  size: 252783
  timestamp: 1720974456583
- kind: conda
  name: c-ares
  version: 1.32.3
  build: h4bc722e_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/c-ares-1.32.3-h4bc722e_0.conda
  sha256: 3c5a844bb60b0d52d89c3f1bd828c9856417fe33a6102fd8bbd5c13c3351704a
  md5: 7624e34ee6baebfc80d67bac76cc9d9d
  depends:
  - __glibc >=2.17,<3.0.a0
  - libgcc-ng >=12
  license: MIT
  license_family: MIT
  purls: []
  size: 179736
  timestamp: 1721834714515
- kind: conda
  name: ca-certificates
  version: 2024.7.4
  build: hbcca054_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2024.7.4-hbcca054_0.conda
  sha256: c1548a3235376f464f9931850b64b02492f379b2f2bb98bc786055329b080446
  md5: 23ab7665c5f63cfb9f1f6195256daac6
  license: ISC
  purls: []
  size: 154853
  timestamp: 1720077432978
- kind: conda
  name: cached-property
  version: 1.5.2
  build: hd8ed1ab_1
  build_number: 1
  subdir: noarch
  noarch: python
  url: https://conda.anaconda.org/conda-forge/noarch/cached-property-1.5.2-hd8ed1ab_1.tar.bz2
  sha256: 561e6660f26c35d137ee150187d89767c988413c978e1b712d53f27ddf70ea17
  md5: 9b347a7ec10940d3f7941ff6c460b551
  depends:
  - cached_property >=1.5.2,<1.5.3.0a0
  license: BSD-3-Clause
  license_family: BSD
  purls:
  - pkg:pypi/cached-property?source=conda-forge-mapping
  size: 4134
  timestamp: 1615209571450
- kind: conda
  name: cached_property
  version: 1.5.2
  build: pyha770c72_1
  build_number: 1
  subdir: noarch
  noarch: python
  url: https://conda.anaconda.org/conda-forge/noarch/cached_property-1.5.2-pyha770c72_1.tar.bz2
  sha256: 6dbf7a5070cc43d90a1e4c2ec0c541c69d8e30a0e25f50ce9f6e4a432e42c5d7
  md5: 576d629e47797577ab0f1b351297ef4a
  depends:
  - python >=3.6
  license: BSD-3-Clause
  license_family: BSD
  purls:
  - pkg:pypi/cached-property?source=conda-forge-mapping
  size: 11065
  timestamp: 1615209567874
- kind: conda
  name: cftime
  version: 1.6.4
  build: py312h085067d_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/cftime-1.6.4-py312h085067d_0.conda
  sha256: 012a195194adaa5886bde21815f764b5652a0a9a818b9892b38c370111e9e54f
  md5: 864d9e92f012bcc49650428d5343c98a
  depends:
  - libgcc-ng >=12
  - numpy >=1.19,<3
  - python >=3.12,<3.13.0a0
  - python_abi 3.12.* *_cp312
  license: MIT
  license_family: MIT
  purls:
  - pkg:pypi/cftime?source=conda-forge-mapping
  size: 248107
  timestamp: 1718096615685
- kind: pypi
  name: cramjam
  version: 2.8.3
  url: https://files.pythonhosted.org/packages/84/6a/4825e220b2efb41e16293c0a8ca5d3682973abab44eee07a12362c0ae979/cramjam-2.8.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  sha256: d93b42d22bf3e17290c5e4cf58e715a419330bb5255c35933c14db82ecf3872c
  requires_dist:
  - black==22.3.0 ; extra == 'dev'
  - numpy ; extra == 'dev'
  - pytest>=5.30 ; extra == 'dev'
  - pytest-xdist ; extra == 'dev'
  - hypothesis ; extra == 'dev'
  requires_python: '>=3.7'
- kind: pypi
  name: decorator
  version: 5.1.1
  url: https://files.pythonhosted.org/packages/d5/50/83c593b07763e1161326b3b8c6686f0f4b0f24d5526546bee538c89837d6/decorator-5.1.1-py3-none-any.whl
  sha256: b8c3f85900b9dc423225913c5aace94729fe1fa9763b38939a95226f02d37186
  requires_python: '>=3.5'
- kind: pypi
  name: executing
  version: 2.0.1
  url: https://files.pythonhosted.org/packages/80/03/6ea8b1b2a5ab40a7a60dc464d3daa7aa546e0a74d74a9f8ff551ea7905db/executing-2.0.1-py2.py3-none-any.whl
  sha256: eac49ca94516ccc753f9fb5ce82603156e590b27525a8bc32cce8ae302eb61bc
  requires_dist:
  - asttokens>=2.1.0 ; extra == 'tests'
  - ipython ; extra == 'tests'
  - pytest ; extra == 'tests'
  - coverage ; extra == 'tests'
  - coverage-enable-subprocess ; extra == 'tests'
  - littleutils ; extra == 'tests'
  - rich ; python_version >= '3.11' and extra == 'tests'
  requires_python: '>=3.5'
- kind: pypi
  name: fasteners
  version: '0.19'
  url: https://files.pythonhosted.org/packages/61/bf/fd60001b3abc5222d8eaa4a204cd8c0ae78e75adc688f33ce4bf25b7fafa/fasteners-0.19-py3-none-any.whl
  sha256: 758819cb5d94cdedf4e836988b74de396ceacb8e2794d21f82d131fd9ee77237
  requires_python: '>=3.6'
- kind: pypi
  name: fastparquet
  version: 2024.5.0
  url: https://files.pythonhosted.org/packages/5d/5a/5dc0de7208f33bcd3223836614eab9ba1cfcd69147010827dab1eabea573/fastparquet-2024.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  sha256: 2b3cf7b4eb1b06e87b97a3a5c9124e4b1c08a8903ba017052c5fe2c482414a3d
  requires_dist:
  - pandas>=1.5.0
  - numpy
  - cramjam>=2.3
  - fsspec
  - packaging
  - python-lzo ; extra == 'lzo'
  requires_python: '>=3.9'
- kind: pypi
  name: fsspec
  version: 2024.6.1
  url: https://files.pythonhosted.org/packages/5e/44/73bea497ac69bafde2ee4269292fa3b41f1198f4bb7bbaaabde30ad29d4a/fsspec-2024.6.1-py3-none-any.whl
  sha256: 3cb443f8bcd2efb31295a5b9fdb02aee81d8452c80d28f97a6d0959e6cee101e
  requires_dist:
  - adlfs ; extra == 'abfs'
  - adlfs ; extra == 'adl'
  - pyarrow>=1 ; extra == 'arrow'
  - dask ; extra == 'dask'
  - distributed ; extra == 'dask'
  - pre-commit ; extra == 'dev'
  - ruff ; extra == 'dev'
  - numpydoc ; extra == 'doc'
  - sphinx ; extra == 'doc'
  - sphinx-design ; extra == 'doc'
  - sphinx-rtd-theme ; extra == 'doc'
  - yarl ; extra == 'doc'
  - dropbox ; extra == 'dropbox'
  - dropboxdrivefs ; extra == 'dropbox'
  - requests ; extra == 'dropbox'
  - adlfs ; extra == 'full'
  - aiohttp!=4.0.0a0,!=4.0.0a1 ; extra == 'full'
  - dask ; extra == 'full'
  - distributed ; extra == 'full'
  - dropbox ; extra == 'full'
  - dropboxdrivefs ; extra == 'full'
  - fusepy ; extra == 'full'
  - gcsfs ; extra == 'full'
  - libarchive-c ; extra == 'full'
  - ocifs ; extra == 'full'
  - panel ; extra == 'full'
  - paramiko ; extra == 'full'
  - pyarrow>=1 ; extra == 'full'
  - pygit2 ; extra == 'full'
  - requests ; extra == 'full'
  - s3fs ; extra == 'full'
  - smbprotocol ; extra == 'full'
  - tqdm ; extra == 'full'
  - fusepy ; extra == 'fuse'
  - gcsfs ; extra == 'gcs'
  - pygit2 ; extra == 'git'
  - requests ; extra == 'github'
  - gcsfs ; extra == 'gs'
  - panel ; extra == 'gui'
  - pyarrow>=1 ; extra == 'hdfs'
  - aiohttp!=4.0.0a0,!=4.0.0a1 ; extra == 'http'
  - libarchive-c ; extra == 'libarchive'
  - ocifs ; extra == 'oci'
  - s3fs ; extra == 's3'
  - paramiko ; extra == 'sftp'
  - smbprotocol ; extra == 'smb'
  - paramiko ; extra == 'ssh'
  - aiohttp!=4.0.0a0,!=4.0.0a1 ; extra == 'test'
  - numpy ; extra == 'test'
  - pytest ; extra == 'test'
  - pytest-asyncio!=0.22.0 ; extra == 'test'
  - pytest-benchmark ; extra == 'test'
  - pytest-cov ; extra == 'test'
  - pytest-mock ; extra == 'test'
  - pytest-recording ; extra == 'test'
  - pytest-rerunfailures ; extra == 'test'
  - requests ; extra == 'test'
  - aiobotocore<3.0.0,>=2.5.4 ; extra == 'test-downstream'
  - dask-expr ; extra == 'test-downstream'
  - dask[dataframe,test] ; extra == 'test-downstream'
  - moto[server]<5,>4 ; extra == 'test-downstream'
  - pytest-timeout ; extra == 'test-downstream'
  - xarray ; extra == 'test-downstream'
  - adlfs ; extra == 'test-full'
  - aiohttp!=4.0.0a0,!=4.0.0a1 ; extra == 'test-full'
  - cloudpickle ; extra == 'test-full'
  - dask ; extra == 'test-full'
  - distributed ; extra == 'test-full'
  - dropbox ; extra == 'test-full'
  - dropboxdrivefs ; extra == 'test-full'
  - fastparquet ; extra == 'test-full'
  - fusepy ; extra == 'test-full'
  - gcsfs ; extra == 'test-full'
  - jinja2 ; extra == 'test-full'
  - kerchunk ; extra == 'test-full'
  - libarchive-c ; extra == 'test-full'
  - lz4 ; extra == 'test-full'
  - notebook ; extra == 'test-full'
  - numpy ; extra == 'test-full'
  - ocifs ; extra == 'test-full'
  - pandas ; extra == 'test-full'
  - panel ; extra == 'test-full'
  - paramiko ; extra == 'test-full'
  - pyarrow ; extra == 'test-full'
  - pyarrow>=1 ; extra == 'test-full'
  - pyftpdlib ; extra == 'test-full'
  - pygit2 ; extra == 'test-full'
  - pytest ; extra == 'test-full'
  - pytest-asyncio!=0.22.0 ; extra == 'test-full'
  - pytest-benchmark ; extra == 'test-full'
  - pytest-cov ; extra == 'test-full'
  - pytest-mock ; extra == 'test-full'
  - pytest-recording ; extra == 'test-full'
  - pytest-rerunfailures ; extra == 'test-full'
  - python-snappy ; extra == 'test-full'
  - requests ; extra == 'test-full'
  - smbprotocol ; extra == 'test-full'
  - tqdm ; extra == 'test-full'
  - urllib3 ; extra == 'test-full'
  - zarr ; extra == 'test-full'
  - zstandard ; extra == 'test-full'
  - tqdm ; extra == 'tqdm'
  requires_python: '>=3.8'
- kind: pypi
  name: h5netcdf
  version: 1.3.0
  url: https://files.pythonhosted.org/packages/68/2d/63851081b19d1ccf245091255797cb358c53c886609b5056da5457f7dbbf/h5netcdf-1.3.0-py3-none-any.whl
  sha256: f2df69dcd3665dc9c4d43eb6529dedd113b2508090d12ac973573305a8406465
  requires_dist:
  - h5py
  - packaging
  - netcdf4 ; extra == 'test'
  - pytest ; extra == 'test'
  requires_python: '>=3.9'
- kind: conda
  name: h5py
  version: 3.11.0
  build: nompi_py312hb7ab980_102
  build_number: 102
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/h5py-3.11.0-nompi_py312hb7ab980_102.conda
  sha256: 08f9cea9414fce460e7dd6aa489e6c81af1eebe3766e8ae22fc55b7238e5b803
  md5: 966750c8f347ece01e80aa2114b4a76d
  depends:
  - cached-property
  - hdf5 >=1.14.3,<1.14.4.0a0
  - libgcc-ng >=12
  - numpy >=1.19,<3
  - python >=3.12,<3.13.0a0
  - python_abi 3.12.* *_cp312
  license: BSD-3-Clause
  license_family: BSD
  purls:
  - pkg:pypi/h5py?source=conda-forge-mapping
  size: 1245015
  timestamp: 1717665055969
- kind: conda
  name: hdf5
  version: 1.14.3
  build: nompi_hdf9ad27_105
  build_number: 105
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/hdf5-1.14.3-nompi_hdf9ad27_105.conda
  sha256: 2278fa07da6f96e807d402cd55480624d67d2dee202191aaaf278ce5ab23605a
  md5: 7e1729554e209627636a0f6fabcdd115
  depends:
  - libaec >=1.1.3,<2.0a0
  - libcurl >=8.8.0,<9.0a0
  - libgcc-ng >=12
  - libgfortran-ng
  - libgfortran5 >=12.3.0
  - libstdcxx-ng >=12
  - libzlib >=1.2.13,<2.0a0
  - openssl >=3.3.1,<4.0a0
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 3911675
  timestamp: 1717587866574
- kind: pypi
  name: ipython
  version: 8.26.0
  url: https://files.pythonhosted.org/packages/73/48/4d2818054671bb272d1b12ca65748a4145dc602a463683b5c21b260becee/ipython-8.26.0-py3-none-any.whl
  sha256: e6b347c27bdf9c32ee9d31ae85defc525755a1869f14057e900675b9e8d6e6ff
  requires_dist:
  - decorator
  - jedi>=0.16
  - matplotlib-inline
  - prompt-toolkit<3.1.0,>=3.0.41
  - pygments>=2.4.0
  - stack-data
  - traitlets>=5.13.0
  - exceptiongroup ; python_version < '3.11'
  - typing-extensions>=4.6 ; python_version < '3.12'
  - pexpect>4.3 ; sys_platform != 'win32' and sys_platform != 'emscripten'
  - colorama ; sys_platform == 'win32'
  - ipython[black,doc,kernel,matplotlib,nbconvert,nbformat,notebook,parallel,qtconsole] ; extra == 'all'
  - ipython[test,test-extra] ; extra == 'all'
  - black ; extra == 'black'
  - docrepr ; extra == 'doc'
  - exceptiongroup ; extra == 'doc'
  - intersphinx-registry ; extra == 'doc'
  - ipykernel ; extra == 'doc'
  - ipython[test] ; extra == 'doc'
  - matplotlib ; extra == 'doc'
  - setuptools>=18.5 ; extra == 'doc'
  - sphinx-rtd-theme ; extra == 'doc'
  - sphinx>=1.3 ; extra == 'doc'
  - sphinxcontrib-jquery ; extra == 'doc'
  - typing-extensions ; extra == 'doc'
  - tomli ; python_version < '3.11' and extra == 'doc'
  - ipykernel ; extra == 'kernel'
  - matplotlib ; extra == 'matplotlib'
  - nbconvert ; extra == 'nbconvert'
  - nbformat ; extra == 'nbformat'
  - ipywidgets ; extra == 'notebook'
  - notebook ; extra == 'notebook'
  - ipyparallel ; extra == 'parallel'
  - qtconsole ; extra == 'qtconsole'
  - pytest ; extra == 'test'
  - pytest-asyncio<0.22 ; extra == 'test'
  - testpath ; extra == 'test'
  - pickleshare ; extra == 'test'
  - packaging ; extra == 'test'
  - ipython[test] ; extra == 'test-extra'
  - curio ; extra == 'test-extra'
  - matplotlib!=3.2.0 ; extra == 'test-extra'
  - nbformat ; extra == 'test-extra'
  - numpy>=1.23 ; extra == 'test-extra'
  - pandas ; extra == 'test-extra'
  - trio ; extra == 'test-extra'
  requires_python: '>=3.10'
- kind: pypi
  name: jedi
  version: 0.19.1
  url: https://files.pythonhosted.org/packages/20/9f/bc63f0f0737ad7a60800bfd472a4836661adae21f9c2535f3957b1e54ceb/jedi-0.19.1-py2.py3-none-any.whl
  sha256: e983c654fe5c02867aef4cdfce5a2fbb4a50adc0af145f70504238f18ef5e7e0
  requires_dist:
  - parso<0.9.0,>=0.8.3
  - jinja2==2.11.3 ; extra == 'docs'
  - markupsafe==1.1.1 ; extra == 'docs'
  - pygments==2.8.1 ; extra == 'docs'
  - alabaster==0.7.12 ; extra == 'docs'
  - babel==2.9.1 ; extra == 'docs'
  - chardet==4.0.0 ; extra == 'docs'
  - commonmark==0.8.1 ; extra == 'docs'
  - docutils==0.17.1 ; extra == 'docs'
  - future==0.18.2 ; extra == 'docs'
  - idna==2.10 ; extra == 'docs'
  - imagesize==1.2.0 ; extra == 'docs'
  - mock==1.0.1 ; extra == 'docs'
  - packaging==20.9 ; extra == 'docs'
  - pyparsing==2.4.7 ; extra == 'docs'
  - pytz==2021.1 ; extra == 'docs'
  - readthedocs-sphinx-ext==2.1.4 ; extra == 'docs'
  - recommonmark==0.5.0 ; extra == 'docs'
  - requests==2.25.1 ; extra == 'docs'
  - six==1.15.0 ; extra == 'docs'
  - snowballstemmer==2.1.0 ; extra == 'docs'
  - sphinx-rtd-theme==0.4.3 ; extra == 'docs'
  - sphinx==1.8.5 ; extra == 'docs'
  - sphinxcontrib-serializinghtml==1.1.4 ; extra == 'docs'
  - sphinxcontrib-websupport==1.2.4 ; extra == 'docs'
  - urllib3==1.26.4 ; extra == 'docs'
  - flake8==5.0.4 ; extra == 'qa'
  - mypy==0.971 ; extra == 'qa'
  - types-setuptools==67.2.0.1 ; extra == 'qa'
  - django ; extra == 'testing'
  - attrs ; extra == 'testing'
  - colorama ; extra == 'testing'
  - docopt ; extra == 'testing'
  - pytest<7.0.0 ; extra == 'testing'
  requires_python: '>=3.6'
- kind: pypi
  name: kerchunk
  version: 0.2.6
  url: https://files.pythonhosted.org/packages/82/8c/117feba86c75d42b8434f8e65085165df4cd415c754bcde3df007f036293/kerchunk-0.2.6-py3-none-any.whl
  sha256: dc55fcea6560688ffc2390ff5882847fdf736982c1817553632a2bd7eb59de73
  requires_dist:
  - fsspec
  - numcodecs
  - numpy
  - ujson
  - zarr
  - cftime ; extra == 'cftime'
  - cftime ; extra == 'dev'
  - dask ; extra == 'dev'
  - h5netcdf ; extra == 'dev'
  - h5py ; extra == 'dev'
  - jinja2 ; extra == 'dev'
  - mypy ; extra == 'dev'
  - pytest ; extra == 'dev'
  - s3fs ; extra == 'dev'
  - types-ujson ; extra == 'dev'
  - xarray ; extra == 'dev'
  - xarray-datatree ; extra == 'dev'
  - cfgrib ; extra == 'dev'
  - scipy ; extra == 'dev'
  - netcdf4 ; extra == 'dev'
  - xarray ; extra == 'fits'
  - cfgrib ; extra == 'grib2'
  - h5py ; extra == 'hdf'
  - xarray ; extra == 'hdf'
  - scipy ; extra == 'netcdf3'
  requires_python: '>=3.7'
- kind: conda
  name: keyutils
  version: 1.6.1
  build: h166bdaf_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/keyutils-1.6.1-h166bdaf_0.tar.bz2
  sha256: 150c05a6e538610ca7c43beb3a40d65c90537497a4f6a5f4d15ec0451b6f5ebb
  md5: 30186d27e2c9fa62b45fb1476b7200e3
  depends:
  - libgcc-ng >=10.3.0
  license: LGPL-2.1-or-later
  purls: []
  size: 117831
  timestamp: 1646151697040
- kind: conda
  name: krb5
  version: 1.21.3
  build: h659f571_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/krb5-1.21.3-h659f571_0.conda
  sha256: 99df692f7a8a5c27cd14b5fb1374ee55e756631b9c3d659ed3ee60830249b238
  md5: 3f43953b7d3fb3aaa1d0d0723d91e368
  depends:
  - keyutils >=1.6.1,<2.0a0
  - libedit >=3.1.20191231,<3.2.0a0
  - libedit >=3.1.20191231,<4.0a0
  - libgcc-ng >=12
  - libstdcxx-ng >=12
  - openssl >=3.3.1,<4.0a0
  license: MIT
  license_family: MIT
  purls: []
  size: 1370023
  timestamp: 1719463201255
- kind: conda
  name: ld_impl_linux-64
  version: '2.40'
  build: hf3520f5_7
  build_number: 7
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.40-hf3520f5_7.conda
  sha256: 764b6950aceaaad0c67ef925417594dd14cd2e22fff864aeef455ac259263d15
  md5: b80f2f396ca2c28b8c14c437a4ed1e74
  constrains:
  - binutils_impl_linux-64 2.40
  license: GPL-3.0-only
  license_family: GPL
  purls: []
  size: 707602
  timestamp: 1718625640445
- kind: conda
  name: libaec
  version: 1.1.3
  build: h59595ed_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libaec-1.1.3-h59595ed_0.conda
  sha256: 2ef420a655528bca9d269086cf33b7e90d2f54ad941b437fb1ed5eca87cee017
  md5: 5e97e271911b8b2001a8b71860c32faa
  depends:
  - libgcc-ng >=12
  - libstdcxx-ng >=12
  license: BSD-2-Clause
  license_family: BSD
  purls: []
  size: 35446
  timestamp: 1711021212685
- kind: conda
  name: libblas
  version: 3.9.0
  build: 23_linux64_openblas
  build_number: 23
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libblas-3.9.0-23_linux64_openblas.conda
  sha256: edb1cee5da3ac4936940052dcab6969673ba3874564f90f5110f8c11eed789c2
  md5: 96c8450a40aa2b9733073a9460de972c
  depends:
  - libopenblas >=0.3.27,<0.3.28.0a0
  - libopenblas >=0.3.27,<1.0a0
  constrains:
  - liblapacke 3.9.0 23_linux64_openblas
  - libcblas 3.9.0 23_linux64_openblas
  - liblapack 3.9.0 23_linux64_openblas
  - blas * openblas
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 14880
  timestamp: 1721688759937
- kind: conda
  name: libcblas
  version: 3.9.0
  build: 23_linux64_openblas
  build_number: 23
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libcblas-3.9.0-23_linux64_openblas.conda
  sha256: 3e7a3236e7e03e308e1667d91d0aa70edd0cba96b4b5563ef4adde088e0881a5
  md5: eede29b40efa878cbe5bdcb767e97310
  depends:
  - libblas 3.9.0 23_linux64_openblas
  constrains:
  - liblapacke 3.9.0 23_linux64_openblas
  - liblapack 3.9.0 23_linux64_openblas
  - blas * openblas
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 14798
  timestamp: 1721688767584
- kind: conda
  name: libcurl
  version: 8.9.1
  build: hdb1bdb2_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libcurl-8.9.1-hdb1bdb2_0.conda
  sha256: 0ba60f83709068e9ec1ab543af998cb5a201c8379c871205447684a34b5abfd8
  md5: 7da1d242ca3591e174a3c7d82230d3c0
  depends:
  - krb5 >=1.21.3,<1.22.0a0
  - libgcc-ng >=12
  - libnghttp2 >=1.58.0,<2.0a0
  - libssh2 >=1.11.0,<2.0a0
  - libzlib >=1.3.1,<2.0a0
  - openssl >=3.3.1,<4.0a0
  - zstd >=1.5.6,<1.6.0a0
  license: curl
  license_family: MIT
  purls: []
  size: 416057
  timestamp: 1722439924963
- kind: conda
  name: libedit
  version: 3.1.20191231
  build: he28a2e2_2
  build_number: 2
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20191231-he28a2e2_2.tar.bz2
  sha256: a57d37c236d8f7c886e01656f4949d9dcca131d2a0728609c6f7fa338b65f1cf
  md5: 4d331e44109e3f0e19b4cb8f9b82f3e1
  depends:
  - libgcc-ng >=7.5.0
  - ncurses >=6.2,<7.0.0a0
  license: BSD-2-Clause
  license_family: BSD
  purls: []
  size: 123878
  timestamp: 1597616541093
- kind: conda
  name: libev
  version: '4.33'
  build: hd590300_2
  build_number: 2
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libev-4.33-hd590300_2.conda
  sha256: 1cd6048169fa0395af74ed5d8f1716e22c19a81a8a36f934c110ca3ad4dd27b4
  md5: 172bf1cd1ff8629f2b1179945ed45055
  depends:
  - libgcc-ng >=12
  license: BSD-2-Clause
  license_family: BSD
  purls: []
  size: 112766
  timestamp: 1702146165126
- kind: conda
  name: libexpat
  version: 2.6.2
  build: h59595ed_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libexpat-2.6.2-h59595ed_0.conda
  sha256: 331bb7c7c05025343ebd79f86ae612b9e1e74d2687b8f3179faec234f986ce19
  md5: e7ba12deb7020dd080c6c70e7b6f6a3d
  depends:
  - libgcc-ng >=12
  constrains:
  - expat 2.6.2.*
  license: MIT
  license_family: MIT
  purls: []
  size: 73730
  timestamp: 1710362120304
- kind: conda
  name: libffi
  version: 3.4.2
  build: h7f98852_5
  build_number: 5
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2
  sha256: ab6e9856c21709b7b517e940ae7028ae0737546122f83c2aa5d692860c3b149e
  md5: d645c6d2ac96843a2bfaccd2d62b3ac3
  depends:
  - libgcc-ng >=9.4.0
  license: MIT
  license_family: MIT
  purls: []
  size: 58292
  timestamp: 1636488182923
- kind: conda
  name: libgcc-ng
  version: 14.1.0
  build: h77fa898_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-14.1.0-h77fa898_0.conda
  sha256: b8e869ac96591cda2704bf7e77a301025e405227791a0bddf14a3dac65125538
  md5: ca0fad6a41ddaef54a153b78eccb5037
  depends:
  - _libgcc_mutex 0.1 conda_forge
  - _openmp_mutex >=4.5
  constrains:
  - libgomp 14.1.0 h77fa898_0
  license: GPL-3.0-only WITH GCC-exception-3.1
  license_family: GPL
  purls: []
  size: 842109
  timestamp: 1719538896937
- kind: conda
  name: libgfortran-ng
  version: 14.1.0
  build: h69a702a_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgfortran-ng-14.1.0-h69a702a_0.conda
  sha256: ef624dacacf97b2b0af39110b36e2fd3e39e358a1a6b7b21b85c9ac22d8ffed9
  md5: f4ca84fbd6d06b0a052fb2d5b96dde41
  depends:
  - libgfortran5 14.1.0 hc5f4f2c_0
  license: GPL-3.0-only WITH GCC-exception-3.1
  license_family: GPL
  purls: []
  size: 49893
  timestamp: 1719538933879
- kind: conda
  name: libgfortran5
  version: 14.1.0
  build: hc5f4f2c_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgfortran5-14.1.0-hc5f4f2c_0.conda
  sha256: a67d66b1e60a8a9a9e4440cee627c959acb4810cb182e089a4b0729bfdfbdf90
  md5: 6456c2620c990cd8dde2428a27ba0bc5
  depends:
  - libgcc-ng >=14.1.0
  constrains:
  - libgfortran-ng 14.1.0
  license: GPL-3.0-only WITH GCC-exception-3.1
  license_family: GPL
  purls: []
  size: 1457561
  timestamp: 1719538909168
- kind: conda
  name: libgomp
  version: 14.1.0
  build: h77fa898_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgomp-14.1.0-h77fa898_0.conda
  sha256: 7699df61a1f6c644b3576a40f54791561f2845983120477a16116b951c9cdb05
  md5: ae061a5ed5f05818acdf9adab72c146d
  depends:
  - _libgcc_mutex 0.1 conda_forge
  license: GPL-3.0-only WITH GCC-exception-3.1
  license_family: GPL
  purls: []
  size: 456925
  timestamp: 1719538796073
- kind: conda
  name: liblapack
  version: 3.9.0
  build: 23_linux64_openblas
  build_number: 23
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.9.0-23_linux64_openblas.conda
  sha256: 25c7aef86c8a1d9db0e8ee61aa7462ba3b46b482027a65d66eb83e3e6f949043
  md5: 2af0879961951987e464722fd00ec1e0
  depends:
  - libblas 3.9.0 23_linux64_openblas
  constrains:
  - liblapacke 3.9.0 23_linux64_openblas
  - libcblas 3.9.0 23_linux64_openblas
  - blas * openblas
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 14823
  timestamp: 1721688775172
- kind: conda
  name: libnghttp2
  version: 1.58.0
  build: h47da74e_1
  build_number: 1
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libnghttp2-1.58.0-h47da74e_1.conda
  sha256: 1910c5306c6aa5bcbd623c3c930c440e9c77a5a019008e1487810e3c1d3716cb
  md5: 700ac6ea6d53d5510591c4344d5c989a
  depends:
  - c-ares >=1.23.0,<2.0a0
  - libev >=4.33,<4.34.0a0
  - libev >=4.33,<5.0a0
  - libgcc-ng >=12
  - libstdcxx-ng >=12
  - libzlib >=1.2.13,<2.0.0a0
  - openssl >=3.2.0,<4.0a0
  license: MIT
  license_family: MIT
  purls: []
  size: 631936
  timestamp: 1702130036271
- kind: conda
  name: libnsl
  version: 2.0.1
  build: hd590300_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.1-hd590300_0.conda
  sha256: 26d77a3bb4dceeedc2a41bd688564fe71bf2d149fdcf117049970bc02ff1add6
  md5: 30fd6e37fe21f86f4bd26d6ee73eeec7
  depends:
  - libgcc-ng >=12
  license: LGPL-2.1-only
  license_family: GPL
  purls: []
  size: 33408
  timestamp: 1697359010159
- kind: conda
  name: libopenblas
  version: 0.3.27
  build: pthreads_hac2b453_1
  build_number: 1
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libopenblas-0.3.27-pthreads_hac2b453_1.conda
  sha256: 714cb82d7c4620ea2635a92d3df263ab841676c9b183d0c01992767bb2451c39
  md5: ae05ece66d3924ac3d48b4aa3fa96cec
  depends:
  - libgcc-ng >=12
  - libgfortran-ng
  - libgfortran5 >=12.3.0
  constrains:
  - openblas >=0.3.27,<0.3.28.0a0
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 5563053
  timestamp: 1720426334043
- kind: conda
  name: libsqlite
  version: 3.46.0
  build: hde9e2c9_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.46.0-hde9e2c9_0.conda
  sha256: daee3f68786231dad457d0dfde3f7f1f9a7f2018adabdbb864226775101341a8
  md5: 18aa975d2094c34aef978060ae7da7d8
  depends:
  - libgcc-ng >=12
  - libzlib >=1.2.13,<2.0a0
  license: Unlicense
  purls: []
  size: 865346
  timestamp: 1718050628718
- kind: conda
  name: libssh2
  version: 1.11.0
  build: h0841786_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libssh2-1.11.0-h0841786_0.conda
  sha256: 50e47fd9c4f7bf841a11647ae7486f65220cfc988ec422a4475fe8d5a823824d
  md5: 1f5a58e686b13bcfde88b93f547d23fe
  depends:
  - libgcc-ng >=12
  - libzlib >=1.2.13,<2.0.0a0
  - openssl >=3.1.1,<4.0a0
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 271133
  timestamp: 1685837707056
- kind: conda
  name: libstdcxx-ng
  version: 14.1.0
  build: hc0a3c3a_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-14.1.0-hc0a3c3a_0.conda
  sha256: 88c42b388202ffe16adaa337e36cf5022c63cf09b0405cf06fc6aeacccbe6146
  md5: 1cb187a157136398ddbaae90713e2498
  depends:
  - libgcc-ng 14.1.0 h77fa898_0
  license: GPL-3.0-only WITH GCC-exception-3.1
  license_family: GPL
  purls: []
  size: 3881307
  timestamp: 1719538923443
- kind: conda
  name: libuuid
  version: 2.38.1
  build: h0b41bf4_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.38.1-h0b41bf4_0.conda
  sha256: 787eb542f055a2b3de553614b25f09eefb0a0931b0c87dbcce6efdfd92f04f18
  md5: 40b61aab5c7ba9ff276c41cfffe6b80b
  depends:
  - libgcc-ng >=12
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 33601
  timestamp: 1680112270483
- kind: conda
  name: libxcrypt
  version: 4.4.36
  build: hd590300_1
  build_number: 1
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libxcrypt-4.4.36-hd590300_1.conda
  sha256: 6ae68e0b86423ef188196fff6207ed0c8195dd84273cb5623b85aa08033a410c
  md5: 5aa797f8787fe7a17d1b0821485b5adc
  depends:
  - libgcc-ng >=12
  license: LGPL-2.1-or-later
  purls: []
  size: 100393
  timestamp: 1702724383534
- kind: conda
  name: libzlib
  version: 1.3.1
  build: h4ab18f5_1
  build_number: 1
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.3.1-h4ab18f5_1.conda
  sha256: adf6096f98b537a11ae3729eaa642b0811478f0ea0402ca67b5108fe2cb0010d
  md5: 57d7dc60e9325e3de37ff8dffd18e814
  depends:
  - libgcc-ng >=12
  constrains:
  - zlib 1.3.1 *_1
  license: Zlib
  license_family: Other
  purls: []
  size: 61574
  timestamp: 1716874187109
- kind: pypi
  name: matplotlib-inline
  version: 0.1.7
  url: https://files.pythonhosted.org/packages/8f/8e/9ad090d3553c280a8060fbf6e24dc1c0c29704ee7d1c372f0c174aa59285/matplotlib_inline-0.1.7-py3-none-any.whl
  sha256: df192d39a4ff8f21b1895d72e6a13f5fcc5099f00fa84384e0ea28c2cc0653ca
  requires_dist:
  - traitlets
  requires_python: '>=3.8'
- kind: conda
  name: ncurses
  version: '6.5'
  build: h59595ed_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.5-h59595ed_0.conda
  sha256: 4fc3b384f4072b68853a0013ea83bdfd3d66b0126e2238e1d6e1560747aa7586
  md5: fcea371545eda051b6deafb24889fc69
  depends:
  - libgcc-ng >=12
  license: X11 AND BSD-3-Clause
  purls: []
  size: 887465
  timestamp: 1715194722503
- kind: pypi
  name: numcodecs
  version: 0.13.0
  url: https://files.pythonhosted.org/packages/85/dd/c89fd8966b69ab4d6c6bd7496fb18bdf920f23dd9bcaec8c7ecce9871d0b/numcodecs-0.13.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  sha256: 820be89729583c91601a6b35c052008cdd2665b25bfedb91b367cc155fb34ba0
  requires_dist:
  - numpy>=1.7
  - sphinx<7.0.0 ; extra == 'docs'
  - sphinx-issues ; extra == 'docs'
  - pydata-sphinx-theme ; extra == 'docs'
  - numpydoc ; extra == 'docs'
  - mock ; extra == 'docs'
  - msgpack ; extra == 'msgpack'
  - pcodec>=0.2.0 ; extra == 'pcodec'
  - coverage ; extra == 'test'
  - pytest ; extra == 'test'
  - pytest-cov ; extra == 'test'
  - importlib-metadata ; extra == 'test-extras'
  - zfpy>=1.0.0 ; extra == 'zfpy'
  - numpy<2.0.0 ; extra == 'zfpy'
  requires_python: '>=3.10'
- kind: conda
  name: numpy
  version: 2.0.1
  build: py312h1103770_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/numpy-2.0.1-py312h1103770_0.conda
  sha256: 0746a37d09036b4164ac14dd1328dd4e449a038383aac1e25e2d5f3a691518da
  md5: 9f444595d8d9682891f2f078fc19da43
  depends:
  - __glibc >=2.17,<3.0.a0
  - libblas >=3.9.0,<4.0a0
  - libcblas >=3.9.0,<4.0a0
  - libgcc-ng >=12
  - liblapack >=3.9.0,<4.0a0
  - libstdcxx-ng >=12
  - python >=3.12,<3.13.0a0
  - python_abi 3.12.* *_cp312
  constrains:
  - numpy-base <0a0
  license: BSD-3-Clause
  license_family: BSD
  purls:
  - pkg:pypi/numpy?source=conda-forge-mapping
  size: 8345605
  timestamp: 1721966364929
- kind: conda
  name: openssl
  version: 3.3.1
  build: h4bc722e_2
  build_number: 2
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/openssl-3.3.1-h4bc722e_2.conda
  sha256: b294b3cc706ad1048cdb514f0db3da9f37ae3fcc0c53a7104083dd0918adb200
  md5: e1b454497f9f7c1147fdde4b53f1b512
  depends:
  - __glibc >=2.17,<3.0.a0
  - ca-certificates
  - libgcc-ng >=12
  constrains:
  - pyopenssl >=22.1
  license: Apache-2.0
  license_family: Apache
  purls: []
  size: 2895213
  timestamp: 1721194688955
- kind: pypi
  name: packaging
  version: '24.1'
  url: https://files.pythonhosted.org/packages/08/aa/cc0199a5f0ad350994d660967a8efb233fe0416e4639146c089643407ce6/packaging-24.1-py3-none-any.whl
  sha256: 5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124
  requires_python: '>=3.8'
- kind: pypi
  name: pandas
  version: 2.2.2
  url: https://files.pythonhosted.org/packages/40/10/79e52ef01dfeb1c1ca47a109a01a248754ebe990e159a844ece12914de83/pandas-2.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  sha256: eee3a87076c0756de40b05c5e9a6069c035ba43e8dd71c379e68cab2c20f16ad
  requires_dist:
  - numpy>=1.22.4 ; python_version < '3.11'
  - numpy>=1.23.2 ; python_version == '3.11'
  - numpy>=1.26.0 ; python_version >= '3.12'
  - python-dateutil>=2.8.2
  - pytz>=2020.1
  - tzdata>=2022.7
  - hypothesis>=6.46.1 ; extra == 'test'
  - pytest>=7.3.2 ; extra == 'test'
  - pytest-xdist>=2.2.0 ; extra == 'test'
  - pyarrow>=10.0.1 ; extra == 'pyarrow'
  - bottleneck>=1.3.6 ; extra == 'performance'
  - numba>=0.56.4 ; extra == 'performance'
  - numexpr>=2.8.4 ; extra == 'performance'
  - scipy>=1.10.0 ; extra == 'computation'
  - xarray>=2022.12.0 ; extra == 'computation'
  - fsspec>=2022.11.0 ; extra == 'fss'
  - s3fs>=2022.11.0 ; extra == 'aws'
  - gcsfs>=2022.11.0 ; extra == 'gcp'
  - pandas-gbq>=0.19.0 ; extra == 'gcp'
  - odfpy>=1.4.1 ; extra == 'excel'
  - openpyxl>=3.1.0 ; extra == 'excel'
  - python-calamine>=0.1.7 ; extra == 'excel'
  - pyxlsb>=1.0.10 ; extra == 'excel'
  - xlrd>=2.0.1 ; extra == 'excel'
  - xlsxwriter>=3.0.5 ; extra == 'excel'
  - pyarrow>=10.0.1 ; extra == 'parquet'
  - pyarrow>=10.0.1 ; extra == 'feather'
  - tables>=3.8.0 ; extra == 'hdf5'
  - pyreadstat>=1.2.0 ; extra == 'spss'
  - sqlalchemy>=2.0.0 ; extra == 'postgresql'
  - psycopg2>=2.9.6 ; extra == 'postgresql'
  - adbc-driver-postgresql>=0.8.0 ; extra == 'postgresql'
  - sqlalchemy>=2.0.0 ; extra == 'mysql'
  - pymysql>=1.0.2 ; extra == 'mysql'
  - sqlalchemy>=2.0.0 ; extra == 'sql-other'
  - adbc-driver-postgresql>=0.8.0 ; extra == 'sql-other'
  - adbc-driver-sqlite>=0.8.0 ; extra == 'sql-other'
  - beautifulsoup4>=4.11.2 ; extra == 'html'
  - html5lib>=1.1 ; extra == 'html'
  - lxml>=4.9.2 ; extra == 'html'
  - lxml>=4.9.2 ; extra == 'xml'
  - matplotlib>=3.6.3 ; extra == 'plot'
  - jinja2>=3.1.2 ; extra == 'output-formatting'
  - tabulate>=0.9.0 ; extra == 'output-formatting'
  - pyqt5>=5.15.9 ; extra == 'clipboard'
  - qtpy>=2.3.0 ; extra == 'clipboard'
  - zstandard>=0.19.0 ; extra == 'compression'
  - dataframe-api-compat>=0.1.7 ; extra == 'consortium-standard'
  - adbc-driver-postgresql>=0.8.0 ; extra == 'all'
  - adbc-driver-sqlite>=0.8.0 ; extra == 'all'
  - beautifulsoup4>=4.11.2 ; extra == 'all'
  - bottleneck>=1.3.6 ; extra == 'all'
  - dataframe-api-compat>=0.1.7 ; extra == 'all'
  - fastparquet>=2022.12.0 ; extra == 'all'
  - fsspec>=2022.11.0 ; extra == 'all'
  - gcsfs>=2022.11.0 ; extra == 'all'
  - html5lib>=1.1 ; extra == 'all'
  - hypothesis>=6.46.1 ; extra == 'all'
  - jinja2>=3.1.2 ; extra == 'all'
  - lxml>=4.9.2 ; extra == 'all'
  - matplotlib>=3.6.3 ; extra == 'all'
  - numba>=0.56.4 ; extra == 'all'
  - numexpr>=2.8.4 ; extra == 'all'
  - odfpy>=1.4.1 ; extra == 'all'
  - openpyxl>=3.1.0 ; extra == 'all'
  - pandas-gbq>=0.19.0 ; extra == 'all'
  - psycopg2>=2.9.6 ; extra == 'all'
  - pyarrow>=10.0.1 ; extra == 'all'
  - pymysql>=1.0.2 ; extra == 'all'
  - pyqt5>=5.15.9 ; extra == 'all'
  - pyreadstat>=1.2.0 ; extra == 'all'
  - pytest>=7.3.2 ; extra == 'all'
  - pytest-xdist>=2.2.0 ; extra == 'all'
  - python-calamine>=0.1.7 ; extra == 'all'
  - pyxlsb>=1.0.10 ; extra == 'all'
  - qtpy>=2.3.0 ; extra == 'all'
  - scipy>=1.10.0 ; extra == 'all'
  - s3fs>=2022.11.0 ; extra == 'all'
  - sqlalchemy>=2.0.0 ; extra == 'all'
  - tables>=3.8.0 ; extra == 'all'
  - tabulate>=0.9.0 ; extra == 'all'
  - xarray>=2022.12.0 ; extra == 'all'
  - xlrd>=2.0.1 ; extra == 'all'
  - xlsxwriter>=3.0.5 ; extra == 'all'
  - zstandard>=0.19.0 ; extra == 'all'
  requires_python: '>=3.9'
- kind: pypi
  name: parso
  version: 0.8.4
  url: https://files.pythonhosted.org/packages/c6/ac/dac4a63f978e4dcb3c6d3a78c4d8e0192a113d288502a1216950c41b1027/parso-0.8.4-py2.py3-none-any.whl
  sha256: a418670a20291dacd2dddc80c377c5c3791378ee1e8d12bffc35420643d43f18
  requires_dist:
  - flake8==5.0.4 ; extra == 'qa'
  - mypy==0.971 ; extra == 'qa'
  - types-setuptools==67.2.0.1 ; extra == 'qa'
  - docopt ; extra == 'testing'
  - pytest ; extra == 'testing'
  requires_python: '>=3.6'
- kind: pypi
  name: pexpect
  version: 4.9.0
  url: https://files.pythonhosted.org/packages/9e/c3/059298687310d527a58bb01f3b1965787ee3b40dce76752eda8b44e9a2c5/pexpect-4.9.0-py2.py3-none-any.whl
  sha256: 7236d1e080e4936be2dc3e326cec0af72acf9212a7e1d060210e70a47e253523
  requires_dist:
  - ptyprocess>=0.5
- kind: pypi
  name: prompt-toolkit
  version: 3.0.47
  url: https://files.pythonhosted.org/packages/e8/23/22750c4b768f09386d1c3cc4337953e8936f48a888fa6dddfb669b2c9088/prompt_toolkit-3.0.47-py3-none-any.whl
  sha256: 0d7bfa67001d5e39d02c224b663abc33687405033a8c422d0d675a5a13361d10
  requires_dist:
  - wcwidth
  requires_python: '>=3.7.0'
- kind: pypi
  name: ptyprocess
  version: 0.7.0
  url: https://files.pythonhosted.org/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl
  sha256: 4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35
- kind: pypi
  name: pure-eval
  version: 0.2.3
  url: https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl
  sha256: 1db8e35b67b3d218d818ae653e27f06c3aa420901fa7b081ca98cbedc874e0d0
  requires_dist:
  - pytest ; extra == 'tests'
- kind: pypi
  name: pygments
  version: 2.18.0
  url: https://files.pythonhosted.org/packages/f7/3f/01c8b82017c199075f8f788d0d906b9ffbbc5a47dc9918a945e13d5a2bda/pygments-2.18.0-py3-none-any.whl
  sha256: b8e6aca0523f3ab76fee51799c488e38782ac06eafcf95e7ba832985c8e7b13a
  requires_dist:
  - colorama>=0.4.6 ; extra == 'windows-terminal'
  requires_python: '>=3.8'
- kind: conda
  name: python
  version: 3.12.5
  build: h2ad013b_0_cpython
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/python-3.12.5-h2ad013b_0_cpython.conda
  sha256: e2aad83838988725d4ffba4e9717b9328054fd18a668cff3377e0c50f109e8bd
  md5: 9c56c4df45f6571b13111d8df2448692
  depends:
  - __glibc >=2.17,<3.0.a0
  - bzip2 >=1.0.8,<2.0a0
  - ld_impl_linux-64 >=2.36.1
  - libexpat >=2.6.2,<3.0a0
  - libffi >=3.4,<4.0a0
  - libgcc-ng >=12
  - libnsl >=2.0.1,<2.1.0a0
  - libsqlite >=3.46.0,<4.0a0
  - libuuid >=2.38.1,<3.0a0
  - libxcrypt >=4.4.36
  - libzlib >=1.3.1,<2.0a0
  - ncurses >=6.5,<7.0a0
  - openssl >=3.3.1,<4.0a0
  - readline >=8.2,<9.0a0
  - tk >=8.6.13,<8.7.0a0
  - tzdata
  - xz >=5.2.6,<6.0a0
  constrains:
  - python_abi 3.12.* *_cp312
  license: Python-2.0
  purls: []
  size: 31663253
  timestamp: 1723143721353
- kind: pypi
  name: python-dateutil
  version: 2.9.0.post0
  url: https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl
  sha256: a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427
  requires_dist:
  - six>=1.5
  requires_python: '!=3.0.*,!=3.1.*,!=3.2.*,>=2.7'
- kind: conda
  name: python_abi
  version: '3.12'
  build: 4_cp312
  build_number: 4
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.12-4_cp312.conda
  sha256: 182a329de10a4165f6e8a3804caf751f918f6ea6176dd4e5abcdae1ed3095bf6
  md5: dccc2d142812964fcc6abdc97b672dff
  constrains:
  - python 3.12.* *_cpython
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 6385
  timestamp: 1695147396604
- kind: pypi
  name: pytz
  version: '2024.1'
  url: https://files.pythonhosted.org/packages/9c/3d/a121f284241f08268b21359bd425f7d4825cffc5ac5cd0e1b3d82ffd2b10/pytz-2024.1-py2.py3-none-any.whl
  sha256: 328171f4e3623139da4983451950b28e95ac706e13f3f2630a879749e7a8b319
- kind: conda
  name: readline
  version: '8.2'
  build: h8228510_1
  build_number: 1
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/readline-8.2-h8228510_1.conda
  sha256: 5435cf39d039387fbdc977b0a762357ea909a7694d9528ab40f005e9208744d7
  md5: 47d31b792659ce70f470b5c82fdfb7a4
  depends:
  - libgcc-ng >=12
  - ncurses >=6.3,<7.0a0
  license: GPL-3.0-only
  license_family: GPL
  purls: []
  size: 281456
  timestamp: 1679532220005
- kind: pypi
  name: six
  version: 1.16.0
  url: https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl
  sha256: 8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254
  requires_python: '>=2.7,!=3.0.*,!=3.1.*,!=3.2.*'
- kind: pypi
  name: stack-data
  version: 0.6.3
  url: https://files.pythonhosted.org/packages/f1/7b/ce1eafaf1a76852e2ec9b22edecf1daa58175c090266e9f6c64afcd81d91/stack_data-0.6.3-py3-none-any.whl
  sha256: d5558e0c25a4cb0853cddad3d77da9891a08cb85dd9f9f91b9f8cd66e511e695
  requires_dist:
  - executing>=1.2.0
  - asttokens>=2.1.0
  - pure-eval
  - pytest ; extra == 'tests'
  - typeguard ; extra == 'tests'
  - pygments ; extra == 'tests'
  - littleutils ; extra == 'tests'
  - cython ; extra == 'tests'
- kind: conda
  name: tk
  version: 8.6.13
  build: noxft_h4845f30_101
  build_number: 101
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_h4845f30_101.conda
  sha256: e0569c9caa68bf476bead1bed3d79650bb080b532c64a4af7d8ca286c08dea4e
  md5: d453b98d9c83e71da0741bb0ff4d76bc
  depends:
  - libgcc-ng >=12
  - libzlib >=1.2.13,<2.0.0a0
  license: TCL
  license_family: BSD
  purls: []
  size: 3318875
  timestamp: 1699202167581
- kind: pypi
  name: traitlets
  version: 5.14.3
  url: https://files.pythonhosted.org/packages/00/c0/8f5d070730d7836adc9c9b6408dec68c6ced86b304a9b26a14df072a6e8c/traitlets-5.14.3-py3-none-any.whl
  sha256: b74e89e397b1ed28cc831db7aea759ba6640cb3de13090ca145426688ff1ac4f
  requires_dist:
  - myst-parser ; extra == 'docs'
  - pydata-sphinx-theme ; extra == 'docs'
  - sphinx ; extra == 'docs'
  - argcomplete>=3.0.3 ; extra == 'test'
  - mypy>=1.7.0 ; extra == 'test'
  - pre-commit ; extra == 'test'
  - pytest-mock ; extra == 'test'
  - pytest-mypy-testing ; extra == 'test'
  - pytest<8.2,>=7.0 ; extra == 'test'
  requires_python: '>=3.8'
- kind: pypi
  name: tzdata
  version: '2024.1'
  url: https://files.pythonhosted.org/packages/65/58/f9c9e6be752e9fcb8b6a0ee9fb87e6e7a1f6bcab2cdc73f02bb7ba91ada0/tzdata-2024.1-py2.py3-none-any.whl
  sha256: 9068bc196136463f5245e51efda838afa15aaeca9903f49050dfa2679db4d252
  requires_python: '>=2'
- kind: conda
  name: tzdata
  version: 2024a
  build: h0c530f3_0
  subdir: noarch
  noarch: generic
  url: https://conda.anaconda.org/conda-forge/noarch/tzdata-2024a-h0c530f3_0.conda
  sha256: 7b2b69c54ec62a243eb6fba2391b5e443421608c3ae5dbff938ad33ca8db5122
  md5: 161081fc7cec0bfda0d86d7cb595f8d8
  license: LicenseRef-Public-Domain
  purls: []
  size: 119815
  timestamp: 1706886945727
- kind: pypi
  name: ujson
  version: 5.10.0
  url: https://files.pythonhosted.org/packages/04/81/668707e5f2177791869b624be4c06fb2473bf97ee33296b18d1cf3092af7/ujson-5.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  sha256: f8ccb77b3e40b151e20519c6ae6d89bfe3f4c14e8e210d910287f778368bb3d1
  requires_python: '>=3.8'
- kind: pypi
  name: universal-pathlib
  version: 0.2.2
  url: https://files.pythonhosted.org/packages/63/a2/d8fe0f7e233c5e0201442d82279f3a0ffb30edcb8d7c4c5eba695c16647f/universal_pathlib-0.2.2-py3-none-any.whl
  sha256: 9bc176112d593348bb29806a47e409eda78dff8d95391d66dd6f85e443aaa75d
  requires_dist:
  - fsspec>=2022.1.0
  - pytest==8.0.0 ; extra == 'dev'
  - pytest-sugar==0.9.7 ; extra == 'dev'
  - pytest-cov==4.1.0 ; extra == 'dev'
  - pytest-mock==3.12.0 ; extra == 'dev'
  - pylint==2.17.4 ; extra == 'dev'
  - mypy==1.8.0 ; extra == 'dev'
  - packaging ; extra == 'dev'
  - adlfs ; extra == 'dev'
  - aiohttp ; extra == 'dev'
  - requests ; extra == 'dev'
  - gcsfs ; extra == 'dev'
  - s3fs ; extra == 'dev'
  - moto[s3,server]<5 ; extra == 'dev'
  - webdav4[fsspec] ; extra == 'dev'
  - wsgidav ; extra == 'dev'
  - cheroot ; extra == 'dev'
  - pydantic ; extra == 'dev'
  - pydantic-settings ; extra == 'dev'
  - pytest==8.0.0 ; extra == 'tests'
  - pytest-sugar==0.9.7 ; extra == 'tests'
  - pytest-cov==4.1.0 ; extra == 'tests'
  - pytest-mock==3.12.0 ; extra == 'tests'
  - pylint==2.17.4 ; extra == 'tests'
  - mypy==1.8.0 ; extra == 'tests'
  - packaging ; extra == 'tests'
  requires_python: '>=3.8'
- kind: pypi
  name: virtualizarr
  version: 1.0.1.dev13+gfdab54c
  url: git+https://github.com/zarr-developers/VirtualiZarr@fdab54cca8d73af4e44903091e5e00bbeb1bb31c
  requires_dist:
  - xarray>=2024.6.0
  - kerchunk>=0.2.5
  - h5netcdf
  - numpy>=2.0.0
  - ujson
  - packaging
  - universal-pathlib
  - codecov ; extra == 'test'
  - fastparquet ; extra == 'test'
  - fsspec ; extra == 'test'
  - h5py ; extra == 'test'
  - mypy ; extra == 'test'
  - netcdf4 ; extra == 'test'
  - pandas-stubs ; extra == 'test'
  - pooch ; extra == 'test'
  - pre-commit ; extra == 'test'
  - pytest-cov ; extra == 'test'
  - pytest-mypy ; extra == 'test'
  - pytest ; extra == 'test'
  - ruff ; extra == 'test'
  - s3fs ; extra == 'test'
  - scipy ; extra == 'test'
  requires_python: '>=3.10'
- kind: pypi
  name: wcwidth
  version: 0.2.13
  url: https://files.pythonhosted.org/packages/fd/84/fd2ba7aafacbad3c4201d395674fc6348826569da3c0937e75505ead3528/wcwidth-0.2.13-py2.py3-none-any.whl
  sha256: 3da69048e4540d84af32131829ff948f1e022c1c6bdb8d6102117aac784f6859
  requires_dist:
  - backports-functools-lru-cache>=1.2.1 ; python_version < '3.2'
- kind: pypi
  name: xarray
  version: 2024.7.0
  url: https://files.pythonhosted.org/packages/45/95/233e1f9c939f5ba314297315df709e6a5e823bf3cade7211991b15aa65d2/xarray-2024.7.0-py3-none-any.whl
  sha256: 1b0fd51ec408474aa1f4a355d75c00cc1c02bd425d97b2c2e551fd21810e7f64
  requires_dist:
  - numpy>=1.23
  - packaging>=23.1
  - pandas>=2.0
  - scipy ; extra == 'accel'
  - bottleneck ; extra == 'accel'
  - numbagg ; extra == 'accel'
  - flox ; extra == 'accel'
  - opt-einsum ; extra == 'accel'
  - xarray[accel,dev,io,parallel,viz] ; extra == 'complete'
  - hypothesis ; extra == 'dev'
  - mypy ; extra == 'dev'
  - pre-commit ; extra == 'dev'
  - pytest ; extra == 'dev'
  - pytest-cov ; extra == 'dev'
  - pytest-env ; extra == 'dev'
  - pytest-xdist ; extra == 'dev'
  - pytest-timeout ; extra == 'dev'
  - ruff ; extra == 'dev'
  - xarray[complete] ; extra == 'dev'
  - netcdf4 ; extra == 'io'
  - h5netcdf ; extra == 'io'
  - scipy ; extra == 'io'
  - zarr ; extra == 'io'
  - fsspec ; extra == 'io'
  - cftime ; extra == 'io'
  - pooch ; extra == 'io'
  - pydap ; python_version < '3.10' and extra == 'io'
  - dask[complete] ; extra == 'parallel'
  - matplotlib ; extra == 'viz'
  - seaborn ; extra == 'viz'
  - nc-time-axis ; extra == 'viz'
  requires_python: '>=3.9'
- kind: conda
  name: xz
  version: 5.2.6
  build: h166bdaf_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2
  sha256: 03a6d28ded42af8a347345f82f3eebdd6807a08526d47899a42d62d319609162
  md5: 2161070d867d1b1204ea749c8eec4ef0
  depends:
  - libgcc-ng >=12
  license: LGPL-2.1 and GPL-2.0
  purls: []
  size: 418368
  timestamp: 1660346797927
- kind: pypi
  name: zarr
  version: 2.18.2
  url: https://files.pythonhosted.org/packages/5d/bd/8d881d8ca6d80fcb8da2b2f94f8855384daf649499ddfba78ffd1ee2caa3/zarr-2.18.2-py3-none-any.whl
  sha256: a638754902f97efa99b406083fdc807a0e2ccf12a949117389d2a4ba9b05df38
  requires_dist:
  - asciitree
  - numpy>=1.23
  - numcodecs>=0.10.0
  - fasteners ; sys_platform != 'emscripten'
  - sphinx ; extra == 'docs'
  - sphinx-automodapi ; extra == 'docs'
  - sphinx-design ; extra == 'docs'
  - sphinx-issues ; extra == 'docs'
  - sphinx-copybutton ; extra == 'docs'
  - pydata-sphinx-theme ; extra == 'docs'
  - numpydoc ; extra == 'docs'
  - numcodecs[msgpack] ; extra == 'docs'
  - notebook ; extra == 'jupyter'
  - ipytree>=0.2.2 ; extra == 'jupyter'
  - ipywidgets>=8.0.0 ; extra == 'jupyter'
  requires_python: '>=3.9'
- kind: conda
  name: zstd
  version: 1.5.6
  build: ha6fb4c9_0
  subdir: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.6-ha6fb4c9_0.conda
  sha256: c558b9cc01d9c1444031bd1ce4b9cff86f9085765f17627a6cd85fc623c8a02b
  md5: 4d056880988120e29d75bfff282e0f45
  depends:
  - libgcc-ng >=12
  - libstdcxx-ng >=12
  - libzlib >=1.2.13,<2.0.0a0
  license: BSD-3-Clause
  license_family: BSD
  purls: []
  size: 554846
  timestamp: 1714722996770

[martindurant]

It may be worth trying with the most recent fsspec, if you get to it before I have a chance to look.

[ashiklom]

Confirming that I hit the same error with the latest (GitHub) versions of kerchunk and fsspec:

  name: kerchunk
  version: 0.2.6.post22
  url: git+https://github.com/fsspec/kerchunk@5085fef3640cc0f5d4b0c067e61eff9fd9e90ecb

  name: fsspec
  version: 2024.6.1.post14+gee98ae3
  url: git+https://github.com/fsspec/filesystem_spec@ee98ae3cc43c68455d4632cb23dd57bb702e5684

For reference, here's the full script I'm using to test this out:

from kerchunk import hdf, df
import fsspec.implementations.reference
from fsspec.implementations.reference import LazyReferenceMapper
import json
                                                                                                                                                                                            
fname = "/css/geostationary/BackStage/GOES-17-ABI-L1B-FULLD/2022/001/00/OR_ABI-L1b-RadF-M6C01_G17_s20220010000320_e20220010009386_c20220010009424.nc"
                                                                                                                                                                                            
h5chunks = hdf.SingleHdf5ToZarr(fname)
refs = h5chunks.translate()
                                                                                                                                                                                            
# Write to JSON
with open("test.json", "w") as f:
    f.write(json.dumps(refs, indent=2))
                                                                                                                                                                                            
# Write to parquet
fs = fsspec.filesystem("file")
out = LazyReferenceMapper.create(record_size=10_000, root="test.parq", fs=fs)
out.flush()
df.refs_to_dataframe(refs, "test.parq")
                                                                                                                                                                                            
# Test the read
                                                                                                                                                                                            
import xarray as xr
                                                                                                                                                                                            
tjson = xr.open_dataset("test.json", engine="kerchunk")
tparq = xr.open_dataset("test.parq", engine="kerchunk")
                                                                                                                                                                                            
tjson.Rad.mean().values  # This works fine
tparq.Rad.mean().values  # This fails

[martindurant]

Please try with fsspec/filesystem_spec#1663

[ashiklom]

Thanks for the quick fix! That partially worked.

It did resolve the specific issue of having an array with missing keys (i.e., Rad/0.0 doesn't exist, but Rad/0.16 does).

However, in my tests, I discovered a related issue: In some situations, we will identify variables with attributes but no chunk data. (Since this is also an inconsistency between parquet and JSON behavior, I think it makes sense to keep it as part of the same issue).

For the same file, here's an excerpt of the output kerchunk JSON reference file:

        "..."
        "Rad/47.33": "base64:eAHt0DENAAAAA...",
        "Rad/47.34": "base64:eAHt0DENAAAAA...",
        "Rad/47.35": "base64:eAHt0DENAAAAA...",
        "algorithm_dynamic_input_data_container/.zarray": "{\"chunks\":[],\"compressor\":null,\"dtype\":\"<i4\",\"fill_value\":-2147483647,\"filters\":null,\"order\":\"C\",\"shape\":[],\"zarr_format\":2}",
        "algorithm_dynamic_input_data_container/.zattrs": "{\"_ARRAY_DIMENSIONS\":[],\"input_ABI_L0_data\":\"OR_ABI-L0-F-M6_G17_s20220010000320_e20220010009386_c*.nc\",\"long_name\":\"container for filenames of dynamic algorithm input data\"}",
        "algorithm_product_version_container/.zarray": "{\"chunks\":[],\"compressor\":null,\"dtype\":\"<i4\",\"fill_value\":-2147483647,\"filters\":null,\"order\":\"C\",\"shape\":[],\"zarr_format\":2}",
        "algorithm_product_version_container/.zattrs": "{\"_ARRAY_DIMENSIONS\":[],\"algorithm_version\":\"OR_ABI-L1b-ALG-RAD_v01r00.zip\",\"long_name\":\"container for algorithm package filename and product version\",\"product_version\":\"v01r00\"}",
        "band_id/.zarray": "{\"chunks\":[1],\"compressor\":null,\"dtype\":\"|i1\",\"fill_value\":null,\"filters\":null,\"order\":\"C\",\"shape\":[1],\"zarr_format\":2}",
        "band_id/.zattrs": "{\"_ARRAY_DIMENSIONS\":[\"band\"],\"long_name\":\"ABI band number\",\"standard_name\":\"sensor_band_identifier\",\"units\":\"1\"}",
        "band_id/0": "\u0001",
        "..."

Note that algorithm_dynamic_input_data_container and algorithm_product_version_container both have .zarray and .zattrs but no actual chunks.

Again, xr.open_dataset("test.json", engine="kerchunk") works fine with this (weird) JSON reference file, silently ignoring the missing chunks.

The corresponding "test.parq" includes the .zarray and .zattrs in the test.parq/.zmetadata file but no corresponding chunks; the folders for these variables don't exist (the directory structure is in details below).

band_id
band_wavelength
band_wavelength_star_look
channel_gain_field
channel_integration_time
DQF
earth_sun_distance_anomaly_in_AU
esun
focal_plane_temperature_threshold_decreasing
focal_plane_temperature_threshold_exceeded_count
focal_plane_temperature_threshold_increasing
kappa0
maximum_focal_plane_temperature
max_radiance_value_of_valid_pixels
mean_radiance_value_of_valid_pixels
min_radiance_value_of_valid_pixels
missing_pixel_count
nominal_satellite_height
nominal_satellite_subpoint_lat
nominal_satellite_subpoint_lon
percent_uncorrectable_L0_errors
planck_bc1
planck_bc2
planck_fk1
planck_fk2
Rad
saturated_pixel_count
star_id
std_dev_radiance_value_of_valid_pixels
t
time_bounds
t_star_look
undersaturated_pixel_count
valid_pixel_count
x
x_image
x_image_bounds
y
yaw_flip_flag
y_image
y_image_bounds
.zmetadata

Note: algorithm_dynamic_input_data_container, algorithm_product_version_container, and other variables with no data have no folders.

When I try to open this with xr.open_dataset("test.parq", engine="kerchunk"), I get the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/test.parq/goes_imager_projection/refs.0.parq'

The corresponding error backtrace is in the details below:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[3], line 1
----> 1 tparq = xr.open_dataset("test.parq", engine="kerchunk")
      2 tparq.Rad.mean().values
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/xarray/backends/api.py:588, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use
_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
    576 decoders = _resolve_decoders_kwargs(
    577     decode_cf,
    578     open_backend_dataset_parameters=backend.open_dataset_parameters,
   (...)
    584     decode_coords=decode_coords,
    585 )
    587 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 588 backend_ds = backend.open_dataset(
    589     filename_or_obj,
    590     drop_variables=drop_variables,
    591     **decoders,
    592     **kwargs,
    593 )
    594 ds = _dataset_from_backend_dataset(
    595     backend_ds,
    596     filename_or_obj,
   (...)
    606     **kwargs,
    607 )
    608 return ds
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/kerchunk/xarray_backend.py:12, in KerchunkBackend.open_dataset(self, filename_or_obj, storage_options, open_dataset_options, **kw)
      8 def open_dataset(
      9     self, filename_or_obj, *, storage_options=None, open_dataset_options=None, **kw
     10 ):
     11     open_dataset_options = (open_dataset_options or {}) | kw
---> 12     ref_ds = open_reference_dataset(
     13         filename_or_obj,
     14         storage_options=storage_options,
     15         open_dataset_options=open_dataset_options,
     16     )
     17     return ref_ds
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/kerchunk/xarray_backend.py:44, in open_reference_dataset(filename_or_obj, storage_options, open_dataset_options)
     41 if open_dataset_options is None:
     42     open_dataset_options = {}
---> 44 m = fsspec.get_mapper("reference://", fo=filename_or_obj, **storage_options)
     46 return xr.open_dataset(m, engine="zarr", consolidated=False, **open_dataset_options)
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/mapping.py:249, in get_mapper(url, check, create, missing_exceptions, alternate_root, **kwargs)
    218 """Create key-value interface for given URL and options
    219
    220 The URL will be of the form "protocol://location" and point to the root
   (...)
    246 ``FSMap`` instance, the dict-like key-value store.
    247 """
    248 # Removing protocol here - could defer to each open() on the backend
--> 249 fs, urlpath = url_to_fs(url, **kwargs)
    250 root = alternate_root if alternate_root is not None else urlpath
    251 return FSMap(root, fs, check, create, missing_exceptions=missing_exceptions)
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/core.py:408, in url_to_fs(url, **kwargs)
    406     inkwargs["fo"] = urls
    407 urlpath, protocol, _ = chain[0]
--> 408 fs = filesystem(protocol, **inkwargs)
    409 return fs, urlpath
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/registry.py:303, in filesystem(protocol, **storage_options)
    296     warnings.warn(
    297         "The 'arrow_hdfs' protocol has been deprecated and will be "
    298         "removed in the future. Specify it as 'hdfs'.",
    299         DeprecationWarning,
    300     )
    302 cls = get_filesystem_class(protocol)
--> 303 return cls(**storage_options)
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/spec.py:81, in _Cached.__call__(cls, *args, **kwargs)
     79     return cls._cache[token]
     80 else:
---> 81     obj = super().__call__(*args, **kwargs)
     82     # Setting _fs_token here causes some static linters to complain.
     83     obj._fs_token_ = token
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py:715, in ReferenceFileSystem.__init__(self, fo, target, ref_storage_args, target_protocol, target_options, remo
te_protocol, remote_options, fs, template_overrides, simple_templates, max_gap, max_block, cache_size, **kwargs)
    711             self.fss[protocol] = fs
    712 if remote_protocol is None:
    713     # get single protocol from references
    714     # TODO: warning here, since this can be very expensive?
--> 715     for ref in self.references.values():
    716         if callable(ref):
    717             ref = ref()
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py:68, in RefsValuesView.__iter__(self)
     66     yield self._mapping[field + "/0"]
     67     continue
---> 68 yield from self._mapping._generate_all_records(field)
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py:354, in LazyReferenceMapper._generate_all_records(self, field)
    352 nrec = math.ceil(nrec / self.record_size)
    353 for record in range(nrec):
--> 354     yield from self._generate_record(field, record)
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py:335, in LazyReferenceMapper._generate_record(self, field, record)
    333 def _generate_record(self, field, record):
    334     """The references for a given parquet file of a given field"""
--> 335     refs = self.open_refs(field, record)
    336     it = iter(zip(*refs.values()))
    337     if len(refs) == 3:
    338         # All urls
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/reference.py:160, in LazyReferenceMapper.setup.<locals>.open_refs(field, record)
    158 """cached parquet file loader"""
    159 path = self.url.format(field=field, record=record)
--> 160 data = io.BytesIO(self.fs.cat_file(path))
    161 df = self.pd.read_parquet(data, engine="fastparquet")
    162 refs = {c: df[c].values for c in df.columns}
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/spec.py:773, in AbstractFileSystem.cat_file(self, path, start, end, **kwargs)
    761 """Get the content of a file
    762
    763 Parameters
   (...)
    770 kwargs: passed to ``open()``.
    771 """
    772 # explicitly set buffering off?
--> 773 with self.open(path, "rb", **kwargs) as f:
    774     if start is not None:
    775         if start >= 0:
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/spec.py:1303, in AbstractFileSystem.open(self, path, mode, block_size, cache_options, compression, **kwargs)
   1301 else:
   1302     ac = kwargs.pop("autocommit", not self._intrans)
-> 1303     f = self._open(
   1304         path,
   1305         mode=mode,
   1306         block_size=block_size,
   1307         autocommit=ac,
   1308         cache_options=cache_options,
   1309         **kwargs,
   1310     )
   1311     if compression is not None:
   1312         from fsspec.compression import compr
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/local.py:191, in LocalFileSystem._open(self, path, mode, block_size, **kwargs)
    189 if self.auto_mkdir and "w" in mode:
    190     self.makedirs(self._parent(path), exist_ok=True)
--> 191 return LocalFileOpener(path, mode, fs=self, **kwargs)
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/local.py:355, in LocalFileOpener.__init__(self, path, mode, autocommit, fs, compression, **kwargs)
    353 self.compression = get_compression(path, compression)
    354 self.blocksize = io.DEFAULT_BUFFER_SIZE
--> 355 self._open()
                                                                                                                                                                                                                                                         
File /gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/.pixi/envs/default/lib/python3.12/site-packages/fsspec/implementations/local.py:360, in LocalFileOpener._open(self)
    358 if self.f is None or self.f.closed:
    359     if self.autocommit or "w" not in self.mode:
--> 360         self.f = open(self.path, mode=self.mode)
    361         if self.compression:
    362             compress = compr[self.compression]
                                                                                                                                                                                                                                                         
FileNotFoundError: [Errno 2] No such file or directory: '/gpfsm/dnb33/ashiklom/aist-eso/goes-virtualizarr/test.parq/goes_imager_projection/refs.0.parq'

I can fix this by manually removing the variables with no data from the .zmetadata file with (janky!) code like this:

# Clean up parquet zmetadata
with open("test.parq/.zmetadata", "r") as f:
    zm = json.load(f)

# Create a copy of the original metadata
with open("test.parq/.zmetadata.orig", "w") as f:
    f.write(json.dumps(zm, indent=2))

# Remove keys that don't have any array information
varnames = {k.split("/")[0] for k in zm["metadata"].keys() if "/" in k}
haskeys = {x.name for x in Path("test.parq").glob("*/")}
keepvars = varnames.difference(haskeys)
zm2 = zm.copy()
zm2_meta = zm2["metadata"]
for key in list(zm2_meta.keys()):
    for v in keepvars:
        if v in key:
            zm2_meta.pop(key)

with open("test.parq/.zmetadata", "w") as f:
    f.write(json.dumps(zm2))

Once I remove the offending variables from .zmetadata, I can open and analyze the parquet references without issue.

[martindurant]

OK, arrays with no references at all is another edge case that wasn't covered...

[martindurant]

I wrote the following test as a minimal reproducer - but it doesn't show the problem at all. an you see a difference?

def test_parquet_no_data(m):
    lz = fsspec.implementations.reference.LazyReferenceMapper.create(
        "memory://out.parq", fs=m
    )

    g = zarr.open_group(lz, mode="w")
    arr = g.create_dataset(name="one", dtype="int32", shape=(10,), chunks=(5,), compression=None,
                           fill_value=1)
    lz.flush()

    assert (arr[:] == 1).all()

[ashiklom]

Sorry for not getting back to this sooner.

I'm not really sure how to create a minimal reproducible example of this, but I have identified the problem: The variables in question are scalar variables with attributes but no values (and therefore, no offsets). See details here for an example h5dump; note that the STORAGE_LAYOUT -> OFFSET = HADDR_UNDEF.

DATASET "processing_parm_version_container" {
   DATATYPE  H5T_STD_I32LE
   DATASPACE  SCALAR
   OBJECTID { 360287970189639730 }
   STORAGE_LAYOUT {
      CONTIGUOUS
      SIZE 0
      OFFSET HADDR_UNDEF
   }
   FILTERS {
      NONE
   }
   FILLVALUE {
      FILL_TIME H5D_FILL_TIME_IFSET
      VALUE  -2147483647
   }
   ALLOCATION_TIME {
      H5D_ALLOC_TIME_LATE
   }
   DATA {
   (0): -2147483647
   }
   ATTRIBUTE "L1b_processing_parm_version" {
      DATATYPE  H5T_STRING {
         STRSIZE 26;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      OBJECTID { 504403158265495552 }
      DATA {
      (0): "OR-PARM-RAD_G17_v01r00.zip"
      }
   }
   ATTRIBUTE "long_name" {
      DATATYPE  H5T_STRING {
         STRSIZE 44;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      OBJECTID { 504403158265495553 }
      DATA {
      (0): "container for processing parameter filenames"
      }
   }
}

I.e., The problem isn't so much that hf[bad_variable].chunks is None but rather that hf[bad_variable].id.get_offset() is None.

The relevant kerchunk code is here:

https://github.com/fsspec/kerchunk/blob/main/kerchunk/hdf.py#L613-L617

This returns an empty dict, because the get_offset() for this variable is None.

But, there are still .zarray and .zattrs values to be referenced because this is an empty variable with attributes. So, the result is a refs list that contains .zarray and .zattrs keys but no offsets (and therefore no chunks, no refs.0.parq file, and therefore the "file not found" error reported above when the parquet engine tries to look for this).

In [17]: refs["refs"]["processing_parm_version_container/.zarray"]
Out[17]: '{"chunks":[],"compressor":null,"dtype":"<i4","fill_value":-2147483647,"filters":null,"order":"C","shape":[],"zarr_format":2}'

In [18]: refs["refs"]["processing_parm_version_container/.zattrs"]
Out[18]: '{"L1b_processing_parm_version":"OR-PARM-RAD_G17_v01r00.zip","_ARRAY_DIMENSIONS":[],"long_name":"container for processing parameter filenames"}'

Digging deeper, it looks like the fsspec JSON reader "succeeds" here because it just loops over all the references in the JSON, without worrying about the need for every .zattrs and .zarray to have a corresponding varname/0.0 chunk. (But presumably, this may fail downstream somewhere, when you try to request values for this array...).

https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/reference.py#L959-L980

However, the parquet loader fails here because it explicitly expects a references file to exist:

https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/reference.py#L159-L160

Specifically, it seems that, if the length of the object is zero, the implementation automatically assumes the scalar is stored in a chunk called 0. This assumption is incorrect for the (unusual) case where the array is completely empty and has no offsets.

https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/reference.py#L311-L316

[martindurant]

OK, I think I understand. I'm short of time this week and next, but please ping me again, and I will sort it out.

[ashiklom]

Hi! Just checking in to see if you've been able to make any progress on this. Let me know if there are any prototype implementations, PRs, etc. that you'd like me to test out.

Also, note that fsspec/filesystem_spec#1663 did actually solve the first part of this problem, which is arguably a bigger issue because I can't hack around that, whereas the empty variables issue can be hacked around by just removing the offending variables from the .zmetadata file.

[martindurant]

Sorry, I've not been working on this. I'll try to find some time.

So there were two problems:

• attempt to access references marked as NULL (missing), fixed by the linked commit
• variables with no references at all, because they have only NULLs or they have dimensions of ()

[ashiklom]

No worries! To clarify, the two issues are:

1. Variable arrays where only certain chunks (but not all chunks) are missing. Example: GOES data linked above, where the Rad array starts at chunk Rad/0.16 (but Rad/0.0 through Rad/0.15 are missing). I think this was resolved by the linked commit.
2. Variables that are only used as containers for attributes and have no data stored at all (i.e., dimensionality is ()).

[martindurant]

agreed

[martindurant]

I am looking back at this, and it appears to me that the 2. case was also already solved (test function below). In this case, the data is a scalar value, and if there is no refs parquet file, then an access to the data looks like the fill value.

def test_parquet_no_references(m):
    lz = fsspec.implementations.reference.LazyReferenceMapper.create(
        "memory://out.parq", fs=m
    )

    g = zarr.open_group(lz, mode="w")
    arr = g.create_dataset(
        name="one",
        dtype="int32",
        shape=(),
        chunks=(),
        compression=None,
        fill_value=1,
    )
    lz.flush()

    assert arr[...].tolist() == 1  #  scalar, equal to fill value

Notice that ... is the only way, I think, to get data from a scalar.