Multi-level datasets are Zarr groups

CHANGES.md

@@ -2,6 +2,10 @@
 
 ### Other changes
 
+* Multi-level datasets (`.levels`) are now regular Zarr groups.
+  We now also write a `.zgroup` file into the root of the dataset folder,
+  so multi-level datasets can now be opened using `xr.open_datatree()`. (#1123)
+
 * Improved the filesystem data stores (`"file"`, `"s3"`, ...): 
   - Added parameter `engine` and its schema. It names an xarray backend to be used
     instead of the automatically detected one.

docs/source/mldatasets.md

@@ -75,6 +75,18 @@ has been made part of the levels format:
     - 2.zarr/
 ```
 
+Starting with xcube 1.9.1, multi-level datasets are regular Zarr groups (so they
+be opened using `xarray.open_datatree()`).
+
+```text
+- test_pyramid.levels/
+    - .zgroup
+    - .zlevels
+    - 0.zarr/
+    - 1.zarr/
+    - 2.zarr/
+```
+
 If present, it is a text file comprising a JSON object with the following 
 properties:
 
@@ -148,7 +160,6 @@ To be discussed
 * Do not write `0.link` file. Instead, provide in `.zlevels` where to find 
   each level.
 * No longer use `.zarr` extension for levels. Just use the index as name.
-* Make top-level directory a Zarr group (`.zgroup`), so the multi-level 
-  dataset can be opened as a group using the `zarr` package.
-
+* Now that datasets are Zarr groups it would be more intuitive to use
+  `.zattrs` instead of `.zlevels`.
 

examples/serve/demo/cube-1-250-250.levels/.zgroup

@@ -0,0 +1,3 @@
+{
+    "zarr_format": 2
+}

xcube/core/mldataset/fs.py

@@ -291,7 +291,10 @@ def write_dataset(
         else:
             num_levels_max = min(num_levels, ml_dataset.num_levels)
 
-        with fs.open(str(data_path / ".zlevels"), mode="w") as fp:
+        with fs.open(str(data_path / ".zgroup"), mode="wt") as fp:
+            json.dump({"zarr_format": 2}, fp, indent=2)
+
+        with fs.open(str(data_path / ".zlevels"), mode="wt") as fp:
             levels_data: dict[str, Any] = dict(
                 version=LEVELS_FORMAT_VERSION, num_levels=num_levels_max
             )