Merge branch 'main' into netcdf4-memory

Author: shoyer

.gitignore

@@ -90,3 +90,4 @@ doc/videos-gallery.txt
 # think we shouldn't...)
 uv.lock
 mypy_report/
+xarray-docs/

doc/internals/zarr-encoding-spec.rst

@@ -19,26 +19,57 @@ Xarray ``Dataset`` objects.
 
 Second, from Xarray's point of view, the key difference between
 NetCDF and Zarr is that all NetCDF arrays have *dimension names* while Zarr
-arrays do not. Therefore, in order to store NetCDF data in Zarr, Xarray must
-somehow encode and decode the name of each array's dimensions.
-
-To accomplish this, Xarray developers decided to define a special Zarr array
-attribute: ``_ARRAY_DIMENSIONS``. The value of this attribute is a list of
-dimension names (strings), for example ``["time", "lon", "lat"]``. When writing
-data to Zarr, Xarray sets this attribute on all variables based on the variable
-dimensions. When reading a Zarr group, Xarray looks for this attribute on all
-arrays, raising an error if it can't be found. The attribute is used to define
-the variable dimension names and then removed from the attributes dictionary
-returned to the user.
-
-Because of these choices, Xarray cannot read arbitrary array data, but only
-Zarr data with valid ``_ARRAY_DIMENSIONS`` or
-`NCZarr <https://docs.unidata.ucar.edu/nug/current/nczarr_head.html>`_ attributes
-on each array (NCZarr dimension names are defined in the ``.zarray`` file).
-
-After decoding the ``_ARRAY_DIMENSIONS`` or NCZarr attribute and assigning the variable
-dimensions, Xarray proceeds to [optionally] decode each variable using its
-standard CF decoding machinery used for NetCDF data (see :py:func:`decode_cf`).
+arrays do not. In Zarr v2, Xarray uses an ad-hoc convention to encode and decode
+the name of each array's dimensions. However, starting with Zarr v3, the
+``dimension_names`` attribute provides a formal convention for storing the
+NetCDF data model in Zarr.
+
+Dimension Encoding in Zarr Formats
+-----------------------------------
+
+Xarray encodes array dimensions differently depending on the Zarr format version:
+
+**Zarr V2 Format:**
+Xarray uses a special Zarr array attribute: ``_ARRAY_DIMENSIONS``. The value of this
+attribute is a list of dimension names (strings), for example ``["time", "lon", "lat"]``.
+When writing data to Zarr V2, Xarray sets this attribute on all variables based on the
+variable dimensions. This attribute is visible when accessing arrays directly with
+zarr-python.
+
+**Zarr V3 Format:**
+Xarray uses the native ``dimension_names`` field in the array metadata. This is part
+of the official Zarr V3 specification and is not stored as a regular attribute.
+When accessing arrays with zarr-python, this information is available in the array's
+metadata but not in the attributes dictionary.
+
+When reading a Zarr group, Xarray looks for dimension information in the appropriate
+location based on the format version, raising an error if it can't be found. The
+dimension information is used to define the variable dimension names and then
+(for Zarr V2) removed from the attributes dictionary returned to the user.
+
+CF Conventions
+--------------
+
+Xarray uses its standard CF encoding/decoding functionality for handling metadata
+(see :py:func:`decode_cf`). This includes encoding concepts such as dimensions and
+coordinates. The ``coordinates`` attribute, which lists coordinate variables
+(e.g., ``"yc xc"`` for spatial coordinates), is one part of the broader CF conventions
+used to describe metadata in NetCDF and Zarr.
+
+Compatibility and Reading
+-------------------------
+
+Because of these encoding choices, Xarray cannot read arbitrary Zarr arrays, but only
+Zarr data with valid dimension metadata. Xarray supports:
+
+- Zarr V2 arrays with ``_ARRAY_DIMENSIONS`` attributes
+- Zarr V3 arrays with ``dimension_names`` metadata
+- `NCZarr <https://docs.unidata.ucar.edu/nug/current/nczarr_head.html>`_ format
+  (dimension names are defined in the ``.zarray`` file)
+
+After decoding the dimension information and assigning the variable dimensions,
+Xarray proceeds to [optionally] decode each variable using its standard CF decoding
+machinery used for NetCDF data.
 
 Finally, it's worth noting that Xarray writes (and attempts to read)
 "consolidated metadata" by default (the ``.zmetadata`` file), which is another
@@ -49,34 +80,63 @@ warning about poor performance when reading non-consolidated stores unless they
 explicitly set ``consolidated=False``. See :ref:`io.zarr.consolidated_metadata`
 for more details.
 
-As a concrete example, here we write a tutorial dataset to Zarr and then
-re-open it directly with Zarr:
+Examples: Zarr Format Differences
+----------------------------------
+
+The following examples demonstrate how dimension and coordinate encoding differs
+between Zarr format versions. We'll use the same tutorial dataset but write it
+in different formats to show what users will see when accessing the files directly
+with zarr-python.
+
+**Example 1: Zarr V2 Format**
 
 .. jupyter-execute::
 
     import os
     import xarray as xr
     import zarr
 
+    # Load tutorial dataset and write as Zarr V2
     ds = xr.tutorial.load_dataset("rasm")
-    ds.to_zarr("rasm.zarr", mode="w", consolidated=False)
-    os.listdir("rasm.zarr")
+    ds.to_zarr("rasm_v2.zarr", mode="w", consolidated=False, zarr_format=2)
+
+    # Open with zarr-python and examine attributes
+    zgroup = zarr.open("rasm_v2.zarr")
+    print("Zarr V2 - Tair attributes:")
+    tair_attrs = dict(zgroup["Tair"].attrs)
+    for key, value in tair_attrs.items():
+        print(f"  '{key}': {repr(value)}")
 
 .. jupyter-execute::
+    :hide-code:
 
-    zgroup = zarr.open("rasm.zarr")
-    zgroup.tree()
+    import shutil
+    shutil.rmtree("rasm_v2.zarr")
+
+**Example 2: Zarr V3 Format**
 
 .. jupyter-execute::
 
-    dict(zgroup["Tair"].attrs)
+    # Write the same dataset as Zarr V3
+    ds.to_zarr("rasm_v3.zarr", mode="w", consolidated=False, zarr_format=3)
+
+    # Open with zarr-python and examine attributes
+    zgroup = zarr.open("rasm_v3.zarr")
+    print("Zarr V3 - Tair attributes:")
+    tair_attrs = dict(zgroup["Tair"].attrs)
+    for key, value in tair_attrs.items():
+        print(f"  '{key}': {repr(value)}")
+
+    # For Zarr V3, dimension information is in metadata
+    tair_array = zgroup["Tair"]
+    print(f"\nZarr V3 - dimension_names in metadata: {tair_array.metadata.dimension_names}")
 
 .. jupyter-execute::
     :hide-code:
 
     import shutil
+    shutil.rmtree("rasm_v3.zarr")
 
-    shutil.rmtree("rasm.zarr")
 
 Chunk Key Encoding
 ------------------

doc/whats-new.rst

@@ -45,10 +45,20 @@ Bug fixes
 
 - Fix the ``align_chunks`` parameter on the :py:meth:`~xarray.Dataset.to_zarr` method, it was not being
   passed to the underlying :py:meth:`~xarray.backends.api` method (:issue:`10501`, :pull:`10516`).
+- Fix error when encoding an empty :py:class:`numpy.datetime64` array
+  (:issue:`10722`, :pull:`10723`). By `Spencer Clark
+  <https://github.com/spencerkclark>`_.
 
 Documentation
 ~~~~~~~~~~~~~
 
+- Fixed Zarr encoding documentation with consistent examples and added comprehensive
+  coverage of dimension and coordinate encoding differences between Zarr V2 and V3 formats.
+  The documentation shows what users will see when accessing Zarr files
+  with raw zarr-python, and explains the relationship between ``_ARRAY_DIMENSIONS``
+  (Zarr V2), ``dimension_names`` metadata (Zarr V3), and CF ``coordinates`` attributes.
+  (:pull:`10720`)
+  By `Emmanuel Mathot <https://github.com/emmanuelmathot>`_.
 
 Internal Changes
 ~~~~~~~~~~~~~~~~

xarray/coding/times.py

@@ -1065,9 +1065,12 @@ def _eagerly_encode_cf_datetime(
             # parse with cftime instead
             raise OutOfBoundsDatetime
         assert np.issubdtype(dates.dtype, "datetime64")
-        if calendar in ["standard", "gregorian"] and np.nanmin(dates).astype(
-            "=M8[us]"
-        ).astype(datetime) < datetime(1582, 10, 15):
+        if (
+            calendar in ["standard", "gregorian"]
+            and dates.size > 0
+            and np.nanmin(dates).astype("=M8[us]").astype(datetime)
+            < datetime(1582, 10, 15)
+        ):
             raise_gregorian_proleptic_gregorian_mismatch_error = True
 
         time_unit, ref_date = _unpack_time_unit_and_ref_date(units)

xarray/tests/test_coding_times.py

@@ -2198,3 +2198,24 @@ def test_roundtrip_0size_timedelta(time_unit: PDDatetimeUnitOptions) -> None:
         decoded.load()
     assert decoded.dtype == np.dtype("=m8[s]")
     assert decoded.encoding == encoding
+
+
+def test_roundtrip_empty_datetime64_array(time_unit: PDDatetimeUnitOptions) -> None:
+    # Regression test for GitHub issue #10722.
+    encoding = {
+        "units": "days since 1990-1-1",
+        "dtype": np.dtype("float64"),
+        "calendar": "standard",
+    }
+    times = date_range("2000", periods=0, unit=time_unit)
+    variable = Variable(["time"], times, encoding=encoding)
+
+    encoded = conventions.encode_cf_variable(variable, name="foo")
+    assert encoded.dtype == np.dtype("float64")
+
+    decode_times = CFDatetimeCoder(time_unit=time_unit)
+    roundtripped = conventions.decode_cf_variable(
+        "foo", encoded, decode_times=decode_times
+    )
+    assert_identical(variable, roundtripped)
+    assert roundtripped.dtype == variable.dtype