Merge branch 'zarr-developers:main' into main

Author: sanketverma1704

assets/images/Zarr_accumulation-App_Interface-1.png

@@ -69,7 +69,7 @@ v2 have surfaced. These include:
   stability and innovation, some framework for community exploration
   and development of Zarr extensions is needed, so that innovation can
   happen in a coordinated way with predictable consequences and
-  behaviour of implementations which may not support all extensions.
+  behaviour of implementations which may not support all extension points.
 
 * **High-latency storage**. Zarr v2 was originally developed to support
   local file system storage. Because of this, the design of Zarr v2
@@ -156,6 +156,9 @@ Implementations of the Zarr version 3 core specification are not
 required to be able to read or write data conforming to the Zarr
 version 2 specification, and vice versa.
 
+Zarr v2 datasets may be upgraded by only writing v3 metadata if their
+data-types and codecs are supported in v3.
+
 
 ## Detailed description
 
@@ -170,32 +173,27 @@ specifications section.
 This ZEP proposes a modular specification framework, with the following components:
 
 * **[Zarr core
-  specification](https://zarr-specs.readthedocs.io/en/latest/core/v3.0.html)**
-  -- This specification is the starting point for any Zarr
-  implementation. It defines in an abstract way a format for storing
-  N-dimensional array data.
-
-* **Zarr extension specifications** -- This is an open-ended set of
-  specifications which add new features to and/or modify the Zarr
-  format in some way. There is a further breakdown of different
-  extension types, described in more detail below.
+  specification](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html)**
+  -- This specification is the starting point for any Zarr implementation. It
+  defines in an abstract way a format for storing N-dimensional array data.
 
-* **Zarr codec specifications** -- This is an open-ended set of
-  specifications, each of which defines a protocol for encoding and
-  decoding Zarr chunk data.
+* **Zarr extension point specifications** -- This is an open-ended set of
+  specifications which add new features to and/or modify the Zarr format in some
+  way. There is a further breakdown of different extension types, described in
+  more detail below, covering data types, codecs, and storage transformers.
 
 * **Zarr store specifications** -- This is an open-ended set of
   specifications, each of which defines a mapping from the abstract
   store API defined in the Zarr core specification to a set of
   concrete operations for a specific storage technology, such as a
-  POSIX file system or cloud object storage.
+  file system or cloud object storage.
 
 The primary goal of this modularity is to allow specifications of new
 extensions, codecs and stores to be added over time, without needing
 any change to the core specification.
 
 Note that the core specification allows for decentralised publishing
-of extension, codec and store specifications. In other words, any
+of extension point and store specifications. In other words, any
 individual or organisation may publish such a specification. It is
 hoped that the majority of such specifications will be published on
 the zarr-specs website after a review process by the Zarr community,
@@ -211,36 +209,36 @@ mean specifications that define conventions for the use of Zarr for a
 particular type of data, such as restrictions and expectations
 regarding the groups, arrays and attributes that will be found within
 a Zarr dataset. How these usage convention specifications will be
-managed, published and used is out of scope for this ZEP.
+managed, published and used is out of scope for this ZEP (see
+[ZEP 4](https://github.com/zarr-developers/zeps/pull/28) instead).
 
 A draft of the Zarr version 3 core specification is available at the
 following URL:
 
-   * https://zarr-specs.readthedocs.io/en/latest/core/v3.0.html
+<https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html>
+
+Changes during the draft process of the spec and this ZEP are listed in the
+[change log](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#change-log).
 
 Note publication of the draft specification on the zarr-specs website
 preceded establishiment of the ZEP process, and does not imply
 acceptance of this ZEP. The specification should be considered a
 proposal at this time, pending acceptance of this ZEP.
 
-To illustrate the principle of the other specification types, the
-following specifications have also been published on the zarr-specs
-website:
+Together with the core specification, this ZEP includes
 
-* [Codecs](https://zarr-specs.readthedocs.io/en/latest/codecs.html)
-  (part of this ZEP as well)
+* [Codecs](https://zarr-specs.readthedocs.io/en/latest/v3/codecs.html)
+  (blosc, gzip, endian, transpose)
 
 * Stores - [File system
-  store](https://zarr-specs.readthedocs.io/en/latest/stores/filesystem/v1.0.html)
-  (not part of this ZEP)
+  store](https://zarr-specs.readthedocs.io/en/latest/v3/stores/filesystem/v1.0.html)
 
-* Data types - [Datetime data
-  types](https://zarr-specs.readthedocs.io/en/latest/extensions/data-types/datetime/v1.0.html)
-  (not part of this ZEP)
 
-* Storage transformers - [Sharding storage
-  transformer](https://zarr-specs.readthedocs.io/en/latest/extensions/storage-transformers/sharding/v1.0.html)
-  (part of [ZEP2](https://zarr.dev/zeps/draft/ZEP0002.html))
+To illustrate the principle of the other specification types, the
+[sharding storage transformer](https://zarr-specs.readthedocs.io/en/latest/v3/array-storage-transformers/sharding/v1.0.html)
+specification has also been published on the zarr-specs website, but is still
+work in progress and not part of this ZEP, but part of
+[ZEP2](https://zarr.dev/zeps/draft/ZEP0002.html).
 
 
 ### Extensibility
@@ -268,27 +266,8 @@ specification describes the situations under which the extension can
 be ignored and processing can proceed, and conversely where the
 extension cannot be ignored and processing must terminate.
 
-A further grouping of different kinds of extension is defined:
-
-* **Generic extensions** -- These are extensions which apply to the
-  processing of any data within a Zarr hierarchy.
-
-* **Array extensions** -- These are extensions which apply to
-  processing of an array within a Zarr hierarchy.
-
-* **Data type extensions** -- These are extensions which define a new
-  data type for array items, where the data type is not included in
-  the set of core data types.
-
-* **Chunk grid extensions** -- These are extensions which define a new
-  way of dividing array items into chunks.
-
-* **Chunk memory layout extensions** -- These are extensions which
-  define a new way of organising the data from an array chunk into a
-  contiguous sequence of bytes.
-
-* **Storage transformer extensions** -- These are extensions which
-  define a new storage transformation.
+The different kind of extension points are defined in the core spec:
+<https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#extension-points>
 
 For each of these kinds of extensions, the core specification defines
 a mechanism by which the extension is declared in the metadata
@@ -316,9 +295,6 @@ been reduced relative to Zarr v2. Note in particular the following:
   supporting variable length data types, and defined within a data
   type extension specification.
 
-* Support for filters has been removed from the core specification. It
-  is proposed this be covered in an extension specification.
-
 
 ### Accommodating high-latency storage
 
@@ -357,24 +333,13 @@ objects with a given prefix, which can be used to reduce the size of
 the result set and usually returns much faster than listing an entire
 bucket. It also provides an opportunity to better leverage the support
 for emulating a directory listing, via providing both a "prefix" and a
-"delimiter" parameter.
-
-The new design separates the metadata and chunk data storage objects
-via a different key prefix. All metadata objects have the key prefix
-"/meta" and all chunk data objects have the key prefix "/data". This
-allows a faster listing of all metadata keys within a given store. The
-list of metadata keys is then sufficient to provide some useful
-information to a user regarding the contents and structure of a Zarr
-hierarchy.
+"delimiter" parameter. The new design separates the metadata and chunk
+data storage objects by adding a prefix to chunk data.
 
 The new design also changes the key suffix for metadata objects. For
-example, for an array with hierarchy path "/foo/bar", the key for the
-corresponding metadata object in Zarr v2 would be
-"/foo/bar/.zarray". In Zarr v3 this becomes
-"/meta/root/foo/bar.array.json". This change means that all the names
-and types (array or group) of children of the "/foo" group can be
-discovered and listed via a single storage operation to list the
-objects with prefix "/meta/root/foo" and delimiter "/".
+example, for an array with hierarchy path `/foo/bar`, the key for the
+corresponding metadata object in Zarr v2 would be `/foo/bar/.zarray`.
+In Zarr v3 this becomes `/foo/bar/zarr.json`.
 
 
 ### Storage transformers
@@ -395,7 +360,7 @@ original state whenever data is requested. To use multiple storage transformers,
 those may be stacked to combine different functionalities.
 
 One example extension that would be implemented as a storage transformer is the
-[sharding specification](https://zarr-specs.readthedocs.io/en/latest/extensions/storage-transformers/sharding/v1.0.html).
+[sharding specification](https://zarr-specs.readthedocs.io/en/latest/v3/array-storage-transformers/sharding/v1.0.html).
 Besides this, the following (non-exhaustive) list of concepts that might possibly be
 implemented as storage transformers:
 
@@ -421,82 +386,70 @@ Related work for sharding (motivation for storage transformers):
 
 ## Implementation
 
-The following projects contains the implementation of Zarr Version 3 protocol:
-
-- zarrita: https://github.com/alimanfoo/zarrita
-
-> Zarrita is a minimal, exploratory implementation of the [Zarr version
-3.0 core protocol](https://zarr-specs.readthedocs.io/en/latest/core/v3.0.html#zarr-core-specification-v3-0).
-This is a technical spike only, not for production use.
-
-- xtensor: https://github.com/xtensor-stack/xtensor-zarr
-
-> `xtensor-zarr` offers an API to create and access a Zarr (v2 or v3)
-hierarchy in a store (locally or in the cloud), read and write
-arrays (in various formats) and groups in the hierarchy, and explore
-the hierarchy.
+Support for Zarr v3 in `zarr-python` is in progress, updating to the current
+spec is tracked in <https://github.com/zarr-developers/zarr-python/issues/1290>.
 
 
 ## Discussion
 
 1. Discussions around storage transformers:
-    * https://github.com/zarr-developers/zarr-specs/pull/134
+    * <https://github.com/zarr-developers/zarr-specs/pull/134>
 
 2. Discussions around sharding:
     * For the specification:
-      * https://github.com/zarr-developers/zarr-specs/issues/127
-      * https://github.com/zarr-developers/zarr-specs/pull/134
+      * <https://github.com/zarr-developers/zarr-specs/issues/127>
+      * <https://github.com/zarr-developers/zarr-specs/pull/134>
     * Initial issue in `zarr-python`:
-      * https://github.com/zarr-developers/zarr-python/issues/877
+      * <https://github.com/zarr-developers/zarr-python/issues/877>
     * Different prototype implementations:
-      * https://github.com/alimanfoo/zarrita/pull/40
-      * https://github.com/zarr-developers/zarr-python/pull/876
-      * https://github.com/zarr-developers/zarr-python/pull/947
+      * <https://github.com/alimanfoo/zarrita/pull/40>
+      * <https://github.com/zarr-developers/zarr-python/pull/876>
+      * <https://github.com/zarr-developers/zarr-python/pull/947>
     * Other related discussions:
-      * https://forum.image.sc/t/ome-zarr-chunking-questions/66794
-      * https://forum.image.sc/t/sharding-support-in-ome-zarr/55409
-      * https://forum.image.sc/t/deciding-on-optimal-chunk-size/63023
-      * https://github.com/thewtex/shardedstore/issues/17
+      * <https://forum.image.sc/t/ome-zarr-chunking-questions/66794>
+      * <https://forum.image.sc/t/sharding-support-in-ome-zarr/55409>
+      * <https://forum.image.sc/t/deciding-on-optimal-chunk-size/63023>
+      * <https://github.com/thewtex/shardedstore/issues/17>
 
 3. Discussion around Zarr V3 Protocol:
     * Issues in `zarr-specs`:
       * ZEP1 project board:
-        * https://github.com/orgs/zarr-developers/projects/2
+        * <https://github.com/orgs/zarr-developers/projects/2>
       * Zarr V3 Mission:
-        * https://github.com/zarr-developers/zarr-specs/issues/140
+        * <https://github.com/zarr-developers/zarr-specs/issues/140>
       * Zarr V3 Implementation(Zarrita):
-        * https://github.com/zarr-developers/zarr-specs/issues/84
+        * <https://github.com/zarr-developers/zarr-specs/issues/84>
       * Content-addressable Storage Transformer for Zarr V3:
-        * https://github.com/zarr-developers/zarr-specs/issues/82
+        * <https://github.com/zarr-developers/zarr-specs/issues/82>
       * Additional discussions related to Zarr V3:
-        * https://github.com/zarr-developers/zarr-specs/issues/53
-        * https://github.com/zarr-developers/zarr-specs/issues/13
+        * <https://github.com/zarr-developers/zarr-specs/issues/53>
+        * <https://github.com/zarr-developers/zarr-specs/issues/13>
     * PRs in `zarr-specs`:
       * Zarr V3 Protocol Development Branch:
-        * https://github.com/zarr-developers/zarr-specs/pull/16
+        * <https://github.com/zarr-developers/zarr-specs/pull/16>
       * Zarr V3 Conceptual Model, Data Types, Chunks layouts, Codecs and
       other important changes: 
-        * https://github.com/zarr-developers/zarr-specs/pull/17
-        * https://github.com/zarr-developers/zarr-specs/pull/18
-        * https://github.com/zarr-developers/zarr-specs/pull/22
-        * https://github.com/zarr-developers/zarr-specs/pull/24
-        * https://github.com/zarr-developers/zarr-specs/pull/25
-        * https://github.com/zarr-developers/zarr-specs/pull/27
-        * https://github.com/zarr-developers/zarr-specs/pull/28
-        * https://github.com/zarr-developers/zarr-specs/pull/29
-        * https://github.com/zarr-developers/zarr-specs/pull/30
-        * https://github.com/zarr-developers/zarr-specs/pull/32
-        * https://github.com/zarr-developers/zarr-specs/pull/35
+        * <https://github.com/zarr-developers/zarr-specs/pull/17>
+        * <https://github.com/zarr-developers/zarr-specs/pull/18>
+        * <https://github.com/zarr-developers/zarr-specs/pull/22>
+        * <https://github.com/zarr-developers/zarr-specs/pull/24>
+        * <https://github.com/zarr-developers/zarr-specs/pull/25>
+        * <https://github.com/zarr-developers/zarr-specs/pull/27>
+        * <https://github.com/zarr-developers/zarr-specs/pull/28>
+        * <https://github.com/zarr-developers/zarr-specs/pull/29>
+        * <https://github.com/zarr-developers/zarr-specs/pull/30>
+        * <https://github.com/zarr-developers/zarr-specs/pull/32>
+        * <https://github.com/zarr-developers/zarr-specs/pull/35>
       * Additional changes to Zarr V3 Protocol:
-        * https://github.com/zarr-developers/zarr-specs/pull/54
-        * https://github.com/zarr-developers/zarr-specs/pull/143
-        * https://github.com/zarr-developers/zarr-specs/pull/146
+        * <https://github.com/zarr-developers/zarr-specs/pull/54>
+        * <https://github.com/zarr-developers/zarr-specs/pull/143>
+        * <https://github.com/zarr-developers/zarr-specs/pull/146>
 
 
 ## References and Footnotes
 
 * <a name="ref-ZARR2SPEC"></a> [ZARR2SPEC] -
-  https://zarr.readthedocs.io/en/stable/spec/v2.html
+  <https://zarr.readthedocs.io/en/stable/spec/v2.html>
 
 
 ## License

assets/images/Zarr_accumulation-App_Interface-2.png

@@ -161,11 +161,7 @@ specification.
 
 Sharding can be configured per array in the array metadata by specifying the
 following array metadata keys:
-* `extension` must always be
-  `"https://purl.org/zarr/spec/storage_transformers/sharding/1.0"` for this
-  extension.
-* `type` specifies a binary shard format. In this version, the only binary
-  format is the `indexed` format.
+* `name` must always be `"sharding` for this storage transformer.
 * `configuration` contains only the following configuration key:
   * `chunks_per_shard` is an array of integers providing the number of chunks
     that are combined in a shard for each dimension of the Zarr array, where

draft/ZEP0001.md

@@ -18,6 +18,8 @@ Type: Specification
 
 Created: 2022-10-17
 
+Discussion: https://github.com/orgs/zarr-developers/discussions/52
+
 ## Abstract
 
 To allow the chunks of a zarr array to be rectangular grid rather than a regular grid,