Merge branch 'main' into return-scalar-for-zero-dim-indexing

# Conflicts:
#	tests/test_api.py

Author: brokkoli71

.github/ISSUE_TEMPLATE/config.yml

@@ -5,7 +5,7 @@ contact_links:
     about: A new major feature should be discussed in the Zarr specifications repository.
   - name: Discuss something on ZulipChat
     url: https://ossci.zulipchat.com/
-    about: For questions like "How do I do X with Zarr?", you can move to our ZulipChat.
+    about: For questions like "How do I do X with Zarr?", consider posting your question to our developer chat.
   - name: Discuss something on GitHub Discussions
     url: https://github.com/zarr-developers/zarr-python/discussions
     about: For questions like "How do I do X with Zarr?", you can move to GitHub Discussions.

.pre-commit-config.yaml

@@ -6,11 +6,11 @@ ci:
 default_stages: [pre-commit, pre-push]
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.4
+    rev: v0.9.9
     hooks:
-    - id: ruff
-      args: ["--fix", "--show-fixes"]
-    - id: ruff-format
+      - id: ruff
+        args: ["--fix", "--show-fixes"]
+      - id: ruff-format
   - repo: https://github.com/codespell-project/codespell
     rev: v2.4.1
     hooks:
@@ -19,10 +19,10 @@ repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v5.0.0
     hooks:
-    - id: check-yaml
-    - id: trailing-whitespace
+      - id: check-yaml
+      - id: trailing-whitespace
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.14.1
+    rev: v1.15.0
     hooks:
       - id: mypy
         files: src|tests
@@ -31,9 +31,10 @@ repos:
           - packaging
           - donfig
           - numcodecs[crc32c]
-          - numpy==2.1  # until https://github.com/numpy/numpy/issues/28034 is resolved
+          - numpy==2.1 # until https://github.com/numpy/numpy/issues/28034 is resolved
           - typing_extensions
           - universal-pathlib
+          - obstore>=0.5.1
           # Tests
           - pytest
   - repo: https://github.com/scientific-python/cookie

README.md

@@ -70,7 +70,7 @@
   </td>
 </tr>
 <tr>
-	<td>Zulip</td>
+	<td>Developer Chat</td>
 	<td>
 		<a href="https://ossci.zulipchat.com/">
 		<img src="https://img.shields.io/badge/zulip-join_chat-brightgreen.svg" />
@@ -101,13 +101,13 @@ Zarr is a Python package providing an implementation of compressed, chunked, N-d
 
 ## Main Features
 
-- [**Create**](https://zarr.readthedocs.io/en/stable/tutorial.html#creating-an-array) N-dimensional arrays with any NumPy `dtype`.
-- [**Chunk arrays**](https://zarr.readthedocs.io/en/stable/tutorial.html#chunk-optimizations) along any dimension.
-- [**Compress**](https://zarr.readthedocs.io/en/stable/tutorial.html#compressors) and/or filter chunks using any NumCodecs codec.
-- [**Store arrays**](https://zarr.readthedocs.io/en/stable/tutorial.html#tutorial-storage) in memory, on disk, inside a zip file, on S3, etc...
-- [**Read**](https://zarr.readthedocs.io/en/stable/tutorial.html#reading-and-writing-data) an array [**concurrently**](https://zarr.readthedocs.io/en/stable/tutorial.html#parallel-computing-and-synchronization) from multiple threads or processes.
-- Write to an array concurrently from multiple threads or processes.
-- Organize arrays into hierarchies via [**groups**](https://zarr.readthedocs.io/en/stable/tutorial.html#groups).
+- [**Create**](https://zarr.readthedocs.io/en/stable/user-guide/arrays.html#creating-an-array) N-dimensional arrays with any NumPy `dtype`.
+- [**Chunk arrays**](https://zarr.readthedocs.io/en/stable/user-guide/performance.html#chunk-optimizations) along any dimension.
+- [**Compress**](https://zarr.readthedocs.io/en/stable/user-guide/arrays.html#compressors) and/or filter chunks using any NumCodecs codec.
+- [**Store arrays**](https://zarr.readthedocs.io/en/stable/user-guide/storage.html) in memory, on disk, inside a zip file, on S3, etc...
+- [**Read**](https://zarr.readthedocs.io/en/stable/user-guide/arrays.html#reading-and-writing-data) an array [**concurrently**](https://zarr.readthedocs.io/en/stable/user-guide/performance.html#parallel-computing-and-synchronization) from multiple threads or processes.
+- [**Write**](https://zarr.readthedocs.io/en/stable/user-guide/arrays.html#reading-and-writing-data) to an array concurrently from multiple threads or processes.
+- Organize arrays into hierarchies via [**groups**](https://zarr.readthedocs.io/en/stable/quickstart.html#hierarchical-groups).
 
 ## Where to get it
 

changes/1661.feature.rst

@@ -0,0 +1 @@
+Add experimental ObjectStore storage class based on obstore.

changes/2796.chore.rst

@@ -0,0 +1,2 @@
+Define a new versioning policy based on Effective Effort Versioning. This replaces the old
+Semantic Versioning-based policy.

changes/2924.chore.rst

@@ -369,6 +369,7 @@ def setup(app: sphinx.application.Sphinx) -> None:
     "python": ("https://docs.python.org/3/", None),
     "numpy": ("https://numpy.org/doc/stable/", None),
     "numcodecs": ("https://numcodecs.readthedocs.io/en/stable/", None),
+    "obstore": ("https://developmentseed.org/obstore/latest/", None),
 }
 
 

docs/conf.py

@@ -261,90 +261,83 @@ Merging pull requests
 ~~~~~~~~~~~~~~~~~~~~~
 
 Pull requests submitted by an external contributor should be reviewed and approved by at least
-one core developers before being merged. Ideally, pull requests submitted by a core developer
-should be reviewed and approved by at least one other core developers before being merged.
+one core developer before being merged. Ideally, pull requests submitted by a core developer
+should be reviewed and approved by at least one other core developer before being merged.
 
 Pull requests should not be merged until all CI checks have passed (GitHub Actions
 Codecov) against code that has had the latest main merged in.
 
 Compatibility and versioning policies
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Because Zarr is a data storage library, there are two types of compatibility to
-consider: API compatibility and data format compatibility.
-
-API compatibility
-"""""""""""""""""
-
-All functions, classes and methods that are included in the API
-documentation (files under ``docs/api/*.rst``) are considered as part of the Zarr **public API**,
-except if they have been documented as an experimental feature, in which case they are part of
-the **experimental API**.
-
-Any change to the public API that does **not** break existing third party
-code importing Zarr, or cause third party code to behave in a different way, is a
-**backwards-compatible API change**. For example, adding a new function, class or method is usually
-a backwards-compatible change. However, removing a function, class or method; removing an argument
-to a function or method; adding a required argument to a function or method; or changing the
-behaviour of a function or method, are examples of **backwards-incompatible API changes**.
-
-If a release contains no changes to the public API (e.g., contains only bug fixes or
-other maintenance work), then the micro version number should be incremented (e.g.,
-2.2.0 -> 2.2.1). If a release contains public API changes, but all changes are
-backwards-compatible, then the minor version number should be incremented
-(e.g., 2.2.1 -> 2.3.0). If a release contains any backwards-incompatible public API changes,
-the major version number should be incremented (e.g., 2.3.0 -> 3.0.0).
-
-Backwards-incompatible changes to the experimental API can be included in a minor release,
-although this should be minimised if possible. I.e., it would be preferable to save up
-backwards-incompatible changes to the experimental API to be included in a major release, and to
-stabilise those features at the same time (i.e., move from experimental to public API), rather than
-frequently tinkering with the experimental API in minor releases.
+Versioning
+""""""""""
+Versions of this library are identified by a triplet of integers with the form
+``<major>.<minor>.<patch>``, for example ``3.0.4``. A release of ``zarr-python`` is associated with a new
+version identifier. That new identifier is generated by incrementing exactly one of the components of
+the previous version identifier by 1. When incrementing the ``major`` component of the version identifier,
+the ``minor`` and ``patch`` components is reset to 0. When incrementing the minor component,
+the patch component is reset to 0.
+
+Releases are classified by the library changes contained in that release. This classification
+determines which component of the version identifier is incremented on release.
+
+* ``major`` releases (for example, ``2.18.0`` -> ``3.0.0``) are for changes that will
+  require extensive adaptation efforts from many users and downstream projects.
+  For example, breaking changes to widely-used user-facing APIs should only be applied in a major release.
+
+
+  Users and downstream projects should carefully consider the impact of a major release before
+  adopting it.
+  In advance of a major release, developers should communicate the scope of the upcoming changes,
+  and help users prepare for them.
+
+* ``minor`` releases (or example, ``3.0.0`` -> ``3.1.0``) are for changes that do not require
+  significant effort from most users or downstream downstream projects to respond to. API changes
+  are possible in minor releases if the burden on users imposed by those changes is sufficiently small.
+
+  For example, a recently released API may need fixes or refinements that are breaking, but low impact
+  due to the recency of the feature. Such API changes are permitted in a minor release.
+
+
+  Minor releases are safe for most users and downstream projects to adopt.
+
+
+* ``patch`` releases (for example, ``3.1.0`` -> ``3.1.1``) are for changes that contain no breaking
+  or behaviour changes for downstream projects or users. Examples of changes suitable for a patch release are
+  bugfixes and documentation improvements.
+
+
+  Users should always feel safe upgrading to a the latest patch release.
+
+Note that this versioning scheme is not consistent with `Semantic Versioning <https://semver.org/>`_.
+Contrary to SemVer, the Zarr library may release breaking changes in ``minor`` releases, or even
+``patch`` releases under exceptional circumstances. But we should strive to avoid doing so.
+
+A better model for our versioning scheme is `Intended Effort Versioning <https://jacobtomlinson.dev/effver/>`_,
+or "EffVer". The guiding principle off EffVer is to categorize releases based on the *expected effort
+required to upgrade to that release*.
+
+Zarr developers should make changes as smooth as possible for users. This means making
+backwards-compatible changes wherever possible. When a backwards-incompatible change is necessary,
+users should be notified well in advance, e.g. via informative deprecation warnings.
 
 Data format compatibility
-"""""""""""""""""""""""""
-
-The data format used by Zarr is defined by a specification document, which should be
-platform-independent and contain sufficient detail to construct an interoperable
-software library to read and/or write Zarr data using any programming language. The
-latest version of the specification document is available on the
-`Zarr specifications website <https://zarr-specs.readthedocs.io>`_.
-
-Here, **data format compatibility** means that all software libraries that implement a
-particular version of the Zarr storage specification are interoperable, in the sense
-that data written by any one library can be read by all others. It is obviously
-desirable to maintain data format compatibility wherever possible. However, if a change
-is needed to the storage specification, and that change would break data format
-compatibility in any way, then the storage specification version number should be
-incremented (e.g., 2 -> 3).
-
-The versioning of the Zarr software library is related to the versioning of the storage
-specification as follows. A particular version of the Zarr library will
-implement a particular version of the storage specification. For example, Zarr version
-2.2.0 implements the Zarr storage specification version 2. If a release of the Zarr
-library implements a different version of the storage specification, then the major
-version number of the Zarr library should be incremented. E.g., if Zarr version 2.2.0
-implements the storage spec version 2, and the next release of the Zarr library
-implements storage spec version 3, then the next library release should have version
-number 3.0.0. Note however that the major version number of the Zarr library may not
-always correspond to the spec version number. For example, Zarr versions 2.x, 3.x, and
-4.x might all implement the same version of the storage spec and thus maintain data
-format compatibility, although they will not maintain API compatibility.
-
-When to make a release
-~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Zarr library is an implementation of a file format standard defined externally --
+see the `Zarr specifications website <https://zarr-specs.readthedocs.io>`_ for the list of
+Zarr file format specifications.
 
-Ideally, any bug fixes that don't change the public API should be released as soon as
-possible. It is fine for a micro release to contain only a single bug fix.
 
-When to make a minor release is at the discretion of the core developers. There are no
-hard-and-fast rules, e.g., it is fine to make a minor release to make a single new
-feature available; equally, it is fine to make a minor release that includes a number of
-changes.
+If an existing Zarr format version changes, or a new version of the Zarr format is released, then
+the Zarr library will generally require changes. It is very likely that a new Zarr format will
+require extensive breaking changes to the Zarr library, and so support for a new Zarr format in the
+Zarr library will almost certainly come in new ``major`` release.
+When the Zarr library adds support for a new Zarr format, there may be a period of accelerated
+changes as developers refine newly added APIs and deprecate old APIs. In such a transitional phase
+breaking changes may be more frequent than usual.
 
-Major releases obviously need to be given careful consideration, and should be done as
-infrequently as possible, as they will break existing code and/or affect data
-compatibility in some way.
 
 Release procedure
 ~~~~~~~~~~~~~~~~~
@@ -387,5 +380,7 @@ pre-releases will be available under
 Post-release
 """"""""""""
 
-- Review and merge the pull request on the `conda-forge feedstock <https://github.com/conda-forge/zarr-feedstock>`_ that will be automatically generated.
+- Review and merge the pull request on the
+  `conda-forge feedstock <https://github.com/conda-forge/zarr-feedstock>`_ that will be
+  automatically generated.
 - Create a new "Unreleased" section in the release notes

docs/developers/contributing.rst

@@ -20,7 +20,7 @@ Zarr-Python
 **Useful links**:
 `Source Repository <https://github.com/zarr-developers/zarr-python>`_ |
 `Issue Tracker <https://github.com/zarr-developers/zarr-python/issues>`_ |
-`Zulip Chat <https://ossci.zulipchat.com/>`_ |
+`Developer Chat <https://ossci.zulipchat.com/>`_ |
 `Zarr specifications <https://zarr-specs.readthedocs.io>`_
 
 Zarr-Python is a Python library for reading and writing Zarr groups and arrays. Highlights include:

docs/index.rst

@@ -119,6 +119,28 @@ Zarr allows you to create hierarchical groups, similar to directories::
 
 This creates a group with two datasets: ``foo`` and ``bar``.
 
+Batch Hierarchy Creation
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Zarr provides tools for creating a collection of arrays and groups with a single function call.
+Suppose we want to copy existing groups and arrays into a new storage backend:
+
+    >>> # Create nested groups and add arrays
+    >>> root = zarr.group("data/example-3.zarr", attributes={'name': 'root'})
+    >>> foo = root.create_group(name="foo")
+    >>> bar = root.create_array(
+    ...     name="bar", shape=(100, 10), chunks=(10, 10), dtype="f4"
+    ... )
+    >>> nodes = {'': root.metadata} | {k: v.metadata for k,v in root.members()}
+    >>> print(nodes)
+    >>> from zarr.storage import MemoryStore
+    >>> new_nodes = dict(zarr.create_hierarchy(store=MemoryStore(), nodes=nodes))
+    >>> new_root = new_nodes['']
+    >>> assert new_root.attrs == root.attrs
+
+Note that :func:`zarr.create_hierarchy` will only initialize arrays and groups -- copying array data must
+be done in a separate step.
+
 Persistent Storage
 ------------------