Encoding a fill value of a structured type and serializing to Zarr (v2)

[BrianMichell]

Hello, I'm trying to write some data with the Zarr v2 backend. For the most part things are working as intended, however I am struggling to set the _FillValue for a structured type. If I omit the _FillValue parameter for this Variable serializes as expected. I'm wondering if support for user-specified fill values was dropped for structured types since their future in Zarr v3 was dubious, if we have done something incorrectly in our sample code, or whether this is a bug. If it is a bug I am more than happy to take a crack at fixing it!

Error: TypeError: Failed to decode variable 'metadata': unhashable type: 'writeable void-scalar'

System: Windows 11, Ubunut 22.04LTS
Python version: 3.12.10
Package manager: UV 0.7.8
Xarray version: 2025.7.1
Zarr version: 3.1.1

def test_structured_array_fillvalue_zarr_write() -> None:
    """Minimal reproducer for failure when writing structured array with fill_value using Xarray to Zarr."""

    shape = (4, 4, 2)
    dim_names = ['x', 'y', 'z']
    chunks = (2, 2, 2)

    some_structured_variable = np.dtype([
        ("foo", np.int32),
        ("bar", np.int32),
        ("baz", np.float16),
        ("qux", np.float16),
    ])

    ds = xr.Dataset()

    # Create a float array (e.g., "data") with a fill value
    float_data_zarr = zarr.zeros(shape=shape, dtype=np.float32, zarr_format=2)
    float_data_xr = xr.DataArray(float_data_zarr, dims=dim_names)
    float_data_xr.encoding = {
        "_FillValue": np.nan,
        "chunks": chunks,
        "chunk_key_encoding": V2ChunkKeyEncoding(separator="/").to_dict(),
        "compressor": numcodecs.Blosc(cname='zstd', clevel=5, shuffle=1, blocksize=0),
    }
    ds["data"] = float_data_xr

    fill_value = np.zeros((), dtype=some_structured_variable)
    fill_value["foo"] = 42
    fill_value["bar"] = 1337
    fill_value["baz"] = 3.14
    fill_value["qux"] = 0.00

    metadata_zarr = zarr.zeros(shape=shape[:-1], dtype=some_structured_variable, zarr_format=2)
    metadata_xr = xr.DataArray(metadata_zarr, dims=dim_names[:-1])

    metadata_xr.encoding = {
        "_FillValue": fill_value,
        "chunks": chunks[:-1],
        "chunk_key_encoding": V2ChunkKeyEncoding(separator="/").to_dict(),
        "compressor": numcodecs.Blosc(cname='zstd', clevel=5, shuffle=1, blocksize=0),
    }
    ds["metadata"] = metadata_xr

    # Write dataset with empty values
    ds.to_zarr(
        store="repro.zarr",
        mode="w",
        write_empty_chunks=False,
        zarr_format=2,
        compute=True,
    )

    # -----------------------------------------------
    # Simulate one pass of selective data population
    not_null = np.array([
        [True, False, False, False],
        [False, True, False, False],
        [False, False, True, False],
        [False, False, False, True]
    ])

    # Sample data values for overwrite
    meta_sample = (11, 22, -33.0, 44.0)
    meta_values = np.array([meta_sample] * 4, dtype=some_structured_variable)
    float_values = np.array([
        [100.0, 200.0],
        [300.0, 400.0],
        [500.0, 600.0],
        [700.0, 800.0]
    ], dtype=np.float32)

    # Select a subset to write in-place
    partial_ds = ds[["data", "metadata"]]

    partial_ds["metadata"].data[not_null] = meta_values
    partial_ds["metadata"].data[~not_null] = 0
    partial_ds["data"].data[not_null] = float_values

    # Define the write region
    region = {
        'x': slice(0, 2),
        'y': slice(0, 2),
        'z': slice(0, 2),
    }

    # Write updated region to disk
    sub_ds = partial_ds.isel(region)
    sub_ds.to_zarr(
        store="repro.zarr",
        region=region,
        mode="r+",
        write_empty_chunks=False,
        zarr_format=2,
    )

[ianhi]

TypeError: Failed to decode variable 'metadata': unhashable type: 'writeable void-scalar

Is this error originating from an xarray file or a zarr file?

Either way, I think that this is unexpected behavior and is likely a bug! Could you please open this as a bug report and post the full traceback

[BrianMichell]

Thanks for taking a look! Raised #10591. This was originating from Xarray.