Merge pull request #214 from jbms/chunk-key-encoding

Add chunk_key_encoding array metadata field

Author: jbms

docs/v3/core/v3.0.rst

@@ -9,7 +9,7 @@
 
 Specification URI:
     https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html
-    
+
 Editors:
     * Alistair Miles (`@alimanfoo <https://github.com/alimanfoo>`_), Wellcome Sanger Institute
     * Jonathan Striebel (`@jstriebel <https://github.com/jstriebel>`_), Scalable Minds
@@ -462,7 +462,7 @@ This follows the
     representation is the ``UTF-8`` encoded Unicode string.
 
 .. note::
-    The prefix ``__zarr`` is reserved for core zarr data, and extensions 
+    The prefix ``__zarr`` is reserved for core zarr data, and extensions
     can use other files and folders starting with ``__``.
 
 
@@ -529,10 +529,10 @@ Core data types
      - 8-byte little endian
    * - ``float16`` (optionally supported)
      - IEEE 754 half-precision floating point: sign bit, 5 bits exponent, 10 bits mantissa
-     - 2-byte little endian IEEE 754 binary16 
+     - 2-byte little endian IEEE 754 binary16
    * - ``float32``
      - IEEE 754 single-precision floating point: sign bit, 8 bits exponent, 23 bits mantissa
-     - 4-byte little endian IEEE 754 binary32 
+     - 4-byte little endian IEEE 754 binary32
    * - ``float64``
      - IEEE 754 double-precision floating point: sign bit, 11 bits exponent, 52 bits mantissa
      - 8-byte little endian IEEE 754 binary64
@@ -647,17 +647,8 @@ that chunk, where "%" is the modulo operator. For example, if a
 is contained within the chunk at grid index (1, 7, 2) and has coordinates
 (2, 10, 100) within that chunk.
 
-The identifier for chunk with grid index (``k``, ``j``, ``i``, ...) is
-formed by taking the initial prefix ``c``, and appending for each dimension:
-
-- the ``separator`` character specified within the ``chunk_grid`` metadata object (see
-  the section on `Array metadata`_ below), followed by,
-
-- the ASCII decimal string representation of the chunk index within that dimension.
-
-For example, in a 3 dimensional array, with a separator of ``/`` the identifier
-for the chunk at grid index (1, 23, 45) is the string "c/1/23/45".  With a
-separator of ``.``, the identifier is the string "c.1.23.45".
+The store key corresponding to a given grid cell is determined based on the
+`chunk_key_encoding`_ member of the `Array metadata`_.
 
 Note that this specification does not consider the case where the
 chunk grid and the array space are not aligned at the origin vertices
@@ -668,14 +659,6 @@ origin element of the array may occur at an arbitrary position within
 any chunk, which is required to allow arrays to be extended by an
 arbitrary length in a "negative" direction along any dimension.
 
-.. note:: A main difference with spec v2 is that the default chunk separator
-   changed from ``.`` to ``/``, as in N5.  This decreases the maximum number of
-   items in hierarchical stores like directory stores.
-
-.. note:: Arrays may have 0 dimensions (when for example representing scalars),
-   in which case the coordinate of a chunk is the empty tuple, and the chunk key
-   will consist of the string ``c``.
-
 .. note:: Chunks at the border of an array always have the full chunk size, even when
    the array only covers parts of it. For example, having an array with ``"shape": [30, 30]`` and
    ``"chunk_shape": [16, 16]``, the chunk ``0,1`` would also contain unused values for the indices
@@ -863,7 +846,7 @@ mandatory names:
     if provided, its value must be one or a list of the data type identifiers
     defined in this specification or an extension. Fallback extension datatypes
     are specified as an object with ``name`` and (optionally) ``configuration``.
-    
+
     If an implementation does not recognise the extension or specific data type,
     but a ``fallback`` is present, then the implementation may proceed using the
     first known ``fallback`` value as the data type. For fixed-sized data types,
@@ -883,10 +866,10 @@ mandatory names:
     as defined in this specification, then the value must be an object with the
     names ``name`` and ``configuration``. The value of ``name`` must be the
     string ``"regular"``, and the value of ``configuration`` an object with the
-    names ``chunk_shape`` and ``separator``. ``chunk_shape`` must be an array of
+    member ``chunk_shape``. ``chunk_shape`` must be an array of
     integers providing the lengths of the chunk along each dimension of the
-    array. ``separator`` must be either ``"/"`` or ``"."``. For example,
-    ``{"type": "regular", "configuration": {"chunk_shape": [2, 5], "separator":"/"}}``
+    array.  For example,
+    ``{"type": "regular", "configuration": {"chunk_shape": [2, 5]}}``
     means a regular grid where the chunks have length 2 along the first
     dimension and length 5 along the second dimension.
 
@@ -895,6 +878,71 @@ mandatory names:
     must be a string referring to a v3 chunk grid specification. The
     ``configuration`` is optional and defined by the extension.
 
+``chunk_key_encoding``
+^^^^^^^^^^^^^^^^^^^^^^
+
+    The mapping from chunk grid cell coordinates to keys in the underlying
+    store.
+
+    The value must be an object with required string member ``name``, specifying
+    the encoding type, and optional member ``configuration`` specifying encoding
+    type-dependent parameters; the ``configuration`` value must be an object if
+    it is specified.
+
+    The following encodings are defined:
+
+    - ``default``
+
+      The ``configuration`` object may contain one optional member,
+      ``separator``, which must be either ``"/"`` or ``"."``.  If not specified,
+      ``separator`` defaults to ``"/"``.
+
+      The key for a chunk with grid index (``k``, ``j``, ``i``, ...) is
+      formed by taking the initial prefix ``c``, and appending for each dimension:
+
+      - the ``separator`` character, followed by,
+
+      - the ASCII decimal string representation of the chunk index within that dimension.
+
+      For example, in a 3 dimensional array, with a separator of ``/`` the identifier
+      for the chunk at grid index (1, 23, 45) is the string ``"c/1/23/45"``.  With a
+      separator of ``.``, the identifier is the string ``"c.1.23.45"``.
+
+      .. note:: A main difference with spec v2 is that the default chunk separator
+         changed from ``.`` to ``/``, as in N5.  This decreases the maximum number of
+         items in hierarchical stores like directory stores.
+
+      .. note:: Arrays may have 0 dimensions (when for example representing scalars),
+         in which case the coordinate of a chunk is the empty tuple, and the chunk key
+         will consist of the string ``c``.
+
+    - ``v2``
+
+      The ``configuration`` object may contain one optional member,
+      ``separator``, which must be either ``"/"`` or ``"."``.  If not specified,
+      ``separator`` defaults to ``"."``.
+
+      The identifier for chunk with at least one dimension is formed by
+      concatenating for each dimension:
+
+      - the ASCII decimal string representation of the chunk index within that
+        dimension, followed by
+
+      - the ``separator`` character, except that it is omitted for the last
+        dimension.
+
+      For example, in a 3 dimensional array, with a separator of ``.`` the identifier
+      for the chunk at grid index (1, 23, 45) is the string ``"1.23.45"``.  With a
+      separator of ``/``, the identifier is the string ``"1/23/45"``.
+
+      For chunk grids with 0 dimensions, the single chunk has the key ``"0"``.
+
+      .. note::
+
+         This encoding is intended only to allow existing v2 arrays to be
+         converted to v3 without having to rename chunks.  It is not recommended
+         to be used when writing new arrays.
+
 ``fill_value``
 ^^^^^^^^^^^^^^
 
@@ -1006,8 +1054,13 @@ compressed using gzip compression prior to storage::
         "chunk_grid": {
             "name": "regular",
             "configuration": {
-              "chunk_shape": [1000, 100],
-              "separator" : "/"
+                "chunk_shape": [1000, 100]
+            }
+        },
+        "chunk_key_encoding": {
+            "name": "default",
+            "configuration": {
+                "separator": "/"
             }
         },
         "codecs": [{
@@ -1035,15 +1088,20 @@ above, but using a (currently made up) extension data type::
         "data_type": {
             "name": "datetime",
             "configuration": {
-              "unit": "ns"
+                "unit": "ns"
             },
             "fallback": "int64"
         },
         "chunk_grid": {
             "name": "regular",
             "configuration": {
-              "chunk_shape": [1000, 100],
-              "separator" : "/"
+                "chunk_shape": [1000, 100]
+            }
+        },
+        "chunk_key_encoding": {
+            "name": "default",
+            "configuration": {
+                "separator": "/"
             }
         },
         "codecs": [{
@@ -1056,14 +1114,14 @@ above, but using a (currently made up) extension data type::
     }
 
 .. note::
-   
+
    Comparison with zarr spec v2:
-   
+
    - ``dtype`` has been renamed to ``data_type``,
-   - ``chunks`` has been renamed to ``chunk_grid``,
+   - ``chunks`` has been replaced with ``chunk_grid``,
+   - ``dimension_separator`` has been replaced with ``chunk_key_encoding``,
    - ``order`` has been replaced by the :ref:`transpose <transpose-codec-v1>` codec,
-   - the separate ``filters`` and ``compressor`` fields been combined into the single ``codecs`` field,
-   - ``zarr_format`` is now a string URL rather than a number.
+   - the separate ``filters`` and ``compressor`` fields been combined into the single ``codecs`` field.
 
 
 Group metadata
@@ -1551,12 +1609,13 @@ Extension points
 Different types of extensions can exist and they can be grouped as follows:
 
 =========== ======================= ================================================
-level       extension               metadata                                        
+level       extension               metadata
 =========== ======================= ================================================
-array       data type               `data_type`_                                    
-array       chunk grid              `chunk_grid`_                                   
-array       codecs                  `codecs`_                                       
-array       storage transformer     `storage_transformers (array)`_                 
+array       data type               `data_type`_
+array       chunk grid              `chunk_grid`_
+array       chunk key encoding      `chunk_key_encoding`_
+array       codecs                  `codecs`_
+array       storage transformer     `storage_transformers (array)`_
 =========== ======================= ================================================
 
 If such extension points are used by groups or arrays, they are required, except