Making sense of closing multiple files opened by readers like xarray

[itcarroll]

In the wonderful XArray tutorial on Access Patterns to Remote Data with fsspec, they describe all the backgound to help make sense of the following practice:

with fs.open(uri, **kwargs) as f:
    ds = xr.open_dataset(f, engine="h5netcdf")
    mean = ds.sst.mean()
ds

But, they point out that ds isn't going to remain very useful since f was closed. I find the use of context managers impractical for this reason (and in Jupyter notebooks generally), but also because there is nothing comparable for xr.open_mfdataset.

Is the following a good practice? Is it the only practice? Is there some way I'm missing to create a PathLike (not a file-like) that is bound to a remote filesystem, so that xarray can handle opening and closing each one using the bound file system?

paths = [fs.open(i, **kwargs) for i in uris]
ds = xr.open_mfdataset(paths, engine="h5netcdf")
...
for item in paths:
    item.close()

That's really awkward...

[itcarroll]

With a little reading, I'm now wondering if a viable feature request is for creation of an fsspec.Path that inherits from pathlib.PurePath but has an abstract filesystem. This would be a generalization of pathlib.Path, which I believe only interacts with the local filesystem.

[itcarroll]

oh, now that I know what I'm looking for:

• os.path equivalent for fsspec #747
• pathlib.Path type #434

[itcarroll]

Ooh .... https://github.com/fsspec/universal_pathlib

[itcarroll]

Using universal_pathlib with XArray (or at least with the h5netcdf engine) does not yet seem supported. I would love to know what the universal_pathlib maintainers would suggest for an issue report, if any.

import xarray as xr
from upath import UPath

path = UPath("s3://its-live-data/test-space/sample-data/sst.mnmean.nc", anon=True)
xr.open_dataset(path, engine="h5netcdf")

Output

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 xr.open_dataset(path, engine="h5netcdf")

File [~/projects/dotfiles/.venv/lib/python3.12/site-packages/xarray/backends/api.py:606](http://localhost:8888/lab/tree/sandbox/~/projects/dotfiles/.venv/lib/python3.12/site-packages/xarray/backends/api.py#line=605), in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, create_default_indexes, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
    594 decoders = _resolve_decoders_kwargs(
    595     decode_cf,
    596     open_backend_dataset_parameters=backend.open_dataset_parameters,
   (...)    602     decode_coords=decode_coords,
    603 )
    605 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 606 backend_ds = backend.open_dataset(
    607     filename_or_obj,
    608     drop_variables=drop_variables,
    609     **decoders,
    610     **kwargs,
    611 )
    612 ds = _dataset_from_backend_dataset(
    613     backend_ds,
    614     filename_or_obj,
   (...)    625     **kwargs,
    626 )
    627 return ds

File [~/projects/dotfiles/.venv/lib/python3.12/site-packages/xarray/backends/h5netcdf_.py:540](http://localhost:8888/lab/tree/sandbox/~/projects/dotfiles/.venv/lib/python3.12/site-packages/xarray/backends/h5netcdf_.py#line=539), in H5netcdfBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, format, group, lock, invalid_netcdf, phony_dims, decode_vlen_strings, driver, driver_kwds, storage_options)
    537 emit_phony_dims_warning, phony_dims = _check_phony_dims(phony_dims)
    539 filename_or_obj = _normalize_filename_or_obj(filename_or_obj)
--> 540 store = H5NetCDFStore.open(
    541     filename_or_obj,
    542     format=format,
    543     group=group,
    544     lock=lock,
    545     invalid_netcdf=invalid_netcdf,
    546     phony_dims=phony_dims,
    547     decode_vlen_strings=decode_vlen_strings,
    548     driver=driver,
    549     driver_kwds=driver_kwds,
    550     storage_options=storage_options,
    551 )
    553 store_entrypoint = StoreBackendEntrypoint()
    555 ds = store_entrypoint.open_dataset(
    556     store,
    557     mask_and_scale=mask_and_scale,
   (...)    563     decode_timedelta=decode_timedelta,
    564 )

File [~/projects/dotfiles/.venv/lib/python3.12/site-packages/xarray/backends/h5netcdf_.py:240](http://localhost:8888/lab/tree/sandbox/~/projects/dotfiles/.venv/lib/python3.12/site-packages/xarray/backends/h5netcdf_.py#line=239), in H5NetCDFStore.open(cls, filename, mode, format, group, lock, autoclose, invalid_netcdf, phony_dims, decode_vlen_strings, driver, driver_kwds, storage_options)
    233         lock = combine_locks([HDF5_LOCK, get_write_lock(filename)])
    235 manager_cls = (
    236     CachingFileManager
    237     if isinstance(filename, str) and not is_remote_uri(filename)
    238     else PickleableFileManager
    239 )
--> 240 manager = manager_cls(h5netcdf.File, filename, mode=mode, kwargs=kwargs)
    242 return cls(
    243     manager,
    244     group=group,
   (...)    248     autoclose=autoclose,
    249 )

File [~/projects/dotfiles/.venv/lib/python3.12/site-packages/xarray/backends/file_manager.py:370](http://localhost:8888/lab/tree/sandbox/~/projects/dotfiles/.venv/lib/python3.12/site-packages/xarray/backends/file_manager.py#line=369), in PickleableFileManager.__init__(self, opener, mode, kwargs, *args)
    368 if mode != _OMIT_MODE:
    369     kwargs = kwargs | {"mode": mode}
--> 370 self._file: T_File | None = opener(*args, **kwargs)

File [~/projects/dotfiles/.venv/lib/python3.12/site-packages/h5netcdf/core.py:1672](http://localhost:8888/lab/tree/sandbox/~/projects/dotfiles/.venv/lib/python3.12/site-packages/h5netcdf/core.py#line=1671), in File.__init__(self, path, mode, format, invalid_netcdf, phony_dims, **kwargs)
   1670         self._preexisting_file = mode in {"r", "r+", "a"}
   1671         self._h5py = h5py
-> 1672         self.__h5file = self._h5py.File(
   1673             path, mode, track_order=track_order, **kwargs
   1674         )
   1675 except Exception:
   1676     self._closed = True

File [~/projects/dotfiles/.venv/lib/python3.12/site-packages/h5py/_hl/files.py:541](http://localhost:8888/lab/tree/sandbox/~/projects/dotfiles/.venv/lib/python3.12/site-packages/h5py/_hl/files.py#line=540), in File.__init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, fs_page_size, page_buf_size, min_meta_keep, min_raw_keep, locking, alignment_threshold, alignment_interval, meta_block_size, track_times, **kwds)
    539     name = repr(name).encode('ASCII', 'replace')
    540 else:
--> 541     name = filename_encode(name)
    543 if track_order is None:
    544     track_order = h5.get_config().track_order

File [~/projects/dotfiles/.venv/lib/python3.12/site-packages/h5py/_hl/compat.py:25](http://localhost:8888/lab/tree/sandbox/~/projects/dotfiles/.venv/lib/python3.12/site-packages/h5py/_hl/compat.py#line=24), in filename_encode(filename)
     17 def filename_encode(filename):
     18     """
     19     Encode filename for use in the HDF5 library.
     20 
   (...)     23     filenames in h5py for more information.
     24     """
---> 25     filename = os.fspath(filename)
     26     if sys.platform == "win32" and isinstance(filename, str):
     27         return filename.encode(WINDOWS_ENCODING, "strict")

TypeError: expected str, bytes or os.PathLike object, not S3Path

Things seem to go off the rails at guess_engine and guess_can_open, which both expect an argument of type str | os.PathLike[Any] | ReadBuffer | bytes | memoryview | AbstractDataStore. Since (as of 0.3.0) a UPath is not os.PathLike ... what should be done?

[martindurant]

Quick notes:

• fsspec.open_files will give you an fsspec.core.OpenFiles object. which will open all files in a context, give you a list of open files, and close them after; or you can iterate it. Unlike the one-file variant, it doesn't have .open, but you could call enter and exit manually.
• https://github.com/fsspec/universal_pathlib is indeed supposed to be the Path-like for fsspec. Whether or not it should implement Path methods like ospath() (i.e., support pathlike) is debatable, since those are normally understood as local. It would not be a big change to xarray to have it recognise these, but maybe would affect multiple backends.
• also potentially of interest is fsspec's Generic filesystem, and the unimplemented ides to have a .path accessor on all filesystems that does things that you would expect os.path to do locally (issue link forgotten).