Merge remote-tracking branch 'upstream/main' into tom/fix/public-buffers

Author: TomAugspurger

.github/CONTRIBUTING.md

@@ -1,4 +1,4 @@
 Contributing
 ============
 
-Please see the [project documentation](https://zarr.readthedocs.io/en/stable/contributing.html) for information about contributing to Zarr.
+Please see the [project documentation](https://zarr.readthedocs.io/en/stable/developers/contributing.html) for information about contributing to Zarr.

.github/workflows/hypothesis.yaml

@@ -25,12 +25,19 @@ jobs:
 
     strategy:
       matrix:
-        python-version: ['3.11']
+        python-version: ['3.12']
         numpy-version: ['2.2']
         dependency-set: ["optional"]
 
     steps:
     - uses: actions/checkout@v4
+    - name: Set HYPOTHESIS_PROFILE based on trigger
+      run: |
+        if [[ "${{ github.event_name }}" == "schedule" || "${{ github.event_name }}" == "workflow_dispatch" ]]; then
+          echo "HYPOTHESIS_PROFILE=nightly" >> $GITHUB_ENV
+        else
+          echo "HYPOTHESIS_PROFILE=ci" >> $GITHUB_ENV
+        fi
     - name: Set up Python
       uses: actions/setup-python@v5
       with:
@@ -58,6 +65,7 @@ jobs:
       if: success()
       id: status
       run: |
+        echo "Using Hypothesis profile: $HYPOTHESIS_PROFILE"
         hatch env run --env test.py${{ matrix.python-version }}-${{ matrix.numpy-version }}-${{ matrix.dependency-set }} run-hypothesis
 
     # explicitly save the cache so it gets updated, also do this even if it fails.

.github/workflows/test.yml

@@ -61,6 +61,8 @@ jobs:
         hatch env create test.py${{ matrix.python-version }}-${{ matrix.numpy-version }}-${{ matrix.dependency-set }}
         hatch env run -e test.py${{ matrix.python-version }}-${{ matrix.numpy-version }}-${{ matrix.dependency-set }} list-env
     - name: Run Tests
+      env:
+        HYPOTHESIS_PROFILE: ci
       run: |
         hatch env run --env test.py${{ matrix.python-version }}-${{ matrix.numpy-version }}-${{ matrix.dependency-set }} run-coverage
     - name: Upload coverage
@@ -102,7 +104,7 @@ jobs:
         hatch env run -e ${{ matrix.dependency-set }} list-env
     - name: Run Tests
       run: |
-        hatch env run --env ${{ matrix.dependency-set }} run
+        hatch env run --env ${{ matrix.dependency-set }} run-coverage
     - name: Upload coverage
       uses: codecov/codecov-action@v5
       with:

changes/2774.feature.rst

@@ -0,0 +1 @@
+Add `zarr.storage.FsspecStore.from_mapper()` so that `zarr.open()` supports stores of type `fsspec.mapping.FSMap`.

changes/2874.feature.rst

@@ -0,0 +1,9 @@
+Adds zarr-specific data type classes. This replaces the internal use of numpy data types for zarr
+v2 and a fixed set of string enums for zarr v3. This change is largely internal, but it does
+change the type of the ``dtype`` and ``data_type`` fields on the ``ArrayV2Metadata`` and
+``ArrayV3Metadata`` classes. It also changes the JSON metadata representation of the
+variable-length string data type, but the old metadata representation can still be
+used when reading arrays. The logic for automatically choosing the chunk encoding for a given data
+type has also changed, and this necessitated changes to the ``config`` API.
+
+For more on this new feature, see the `documentation </user-guide/data_types.html>`_

changes/3127.bugfix.rst

@@ -0,0 +1,2 @@
+When `zarr.save` has an argument `path=some/path/` and multiple arrays in `args`, the path resulted in `some/path/some/path` due to using the `path`
+argument twice while building the array path. This is now fixed.

changes/3128.bugfix.rst

@@ -0,0 +1 @@
+Fix `zarr.open` default for argument `mode` when `store` is `read_only`

changes/3130.feature.rst

@@ -0,0 +1 @@
+Port more stateful testing actions from `Icechunk <https://icechunk.io>`_.

docs/conf.py

@@ -106,7 +106,6 @@ def skip_submodules(
      "installation": "user-guide/installation.html",
      "api": "api/zarr/index",
      "release": "release-notes.html",
-     "release-notes": "release-notes.html"
 }
 
 # The language for content autogenerated by Sphinx. Refer to documentation

docs/user-guide/arrays.rst

@@ -182,7 +182,7 @@ which can be used to print useful diagnostics, e.g.::
    >>> z.info
    Type               : Array
    Zarr format        : 3
-   Data type          : DataType.int32
+   Data type          : Int32(endianness='little')
    Fill value         : 0
    Shape              : (10000, 10000)
    Chunk shape        : (1000, 1000)
@@ -200,7 +200,7 @@ prints additional diagnostics, e.g.::
    >>> z.info_complete()
    Type               : Array
    Zarr format        : 3
-   Data type          : DataType.int32
+   Data type          : Int32(endianness='little')
    Fill value         : 0
    Shape              : (10000, 10000)
    Chunk shape        : (1000, 1000)
@@ -248,7 +248,7 @@ built-in delta filter::
 The default compressor can be changed by setting the value of the using Zarr's
 :ref:`user-guide-config`, e.g.::
 
-   >>> with zarr.config.set({'array.v2_default_compressor.numeric': {'id': 'blosc'}}):
+   >>> with zarr.config.set({'array.v2_default_compressor.default': {'id': 'blosc'}}):
    ...     z = zarr.create_array(store={}, shape=(100000000,), chunks=(1000000,), dtype='int32', zarr_format=2)
    >>> z.filters
    ()
@@ -288,7 +288,7 @@ Here is an example using a delta filter with the Blosc compressor::
    >>> z.info
    Type               : Array
    Zarr format        : 3
-   Data type          : DataType.int32
+   Data type          : Int32(endianness='little')
    Fill value         : 0
    Shape              : (10000, 10000)
    Chunk shape        : (1000, 1000)
@@ -603,7 +603,7 @@ Sharded arrays can be created by providing the ``shards`` parameter to :func:`za
   >>> a.info_complete()
   Type               : Array
   Zarr format        : 3
-  Data type          : DataType.uint8
+  Data type          : UInt8()
   Fill value         : 0
   Shape              : (10000, 10000)
   Shard shape        : (1000, 1000)
@@ -612,10 +612,10 @@ Sharded arrays can be created by providing the ``shards`` parameter to :func:`za
   Read-only          : False
   Store type         : LocalStore
   Filters            : ()
-  Serializer         : BytesCodec(endian=<Endian.little: 'little'>)
+  Serializer         : BytesCodec(endian=None)
   Compressors        : (ZstdCodec(level=0, checksum=False),)
   No. bytes          : 100000000 (95.4M)
-  No. bytes stored   : 3981552
+  No. bytes stored   : 3981473
   Storage ratio      : 25.1
   Shards Initialized : 100