WIP

jstriebel · jstriebel · commit a0d021b03086 · 2022-02-22T10:50:02.000+01:00
diff --git a/docs/conf.py b/docs/conf.py
@@ -28,6 +28,7 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+  'sphinxcontrib.mermaid'
 ]
 
 # Add any paths that contain templates here, relative to this directory.
diff --git a/docs/index.rst b/docs/index.rst
@@ -10,8 +10,9 @@ Under construction.
    :caption: Contents:
 
    protocol
-   stores
    codecs
+   stores
+   storage_transformers
 
 
 Indices and tables
diff --git a/docs/protocol/core/v3.0.rst b/docs/protocol/core/v3.0.rst
@@ -383,6 +383,17 @@ conceptual model underpinning the Zarr protocol.
     interface`_ which is a common set of operations that stores may
     provide.
 
+.. _storage transformer:
+.. _storage transformers:
+
+*Storage transformer*
+
+    To enhance the storage capabilities, storage transformers may
+    change the storage structure and behaviour of data coming from
+    an array_ in the underlying store_. Upon retrival the original data is
+    restored within the transformer. Any number of `predefined storage
+    transformers`_ can be registered and stacked.
+
 
 Node names
 ==========
@@ -895,6 +906,8 @@ ignored if not understood::
     }
 
 
+.. _array-metadata:
+
 Array metadata
 --------------
 
@@ -1019,6 +1032,17 @@ The following names are optional:
     specification. When the ``compressor`` name is absent, this means that no
     compressor is used.
 
+``storage_transformers``
+
+    Specifies a codec to be used for encoding and decoding chunks. The
+    value must be an object containing the name ``codec`` whose value
+    is a URI that identifies a codec and dereferences to a human-readable
+    representation of the codec specification. The codec
+    object may also contain a ``configuration`` object which consists of the
+    parameter names and values as defined by the corresponding codec
+    specification. When the ``compressor`` name is absent, this means that no
+    compressor is used.
+
 
 All other names within the array metadata object are reserved for
 future versions of this specification.
@@ -1499,6 +1523,39 @@ Let "+" be the string concatenation operator.
        For listable store, ``list_dir(parent(P))`` can be an alternative.
 
 
+Storage transformers
+====================
+
+A Zarr storage transformer allows to change the zarr-compatible data before storing it.
+The stored transformed data is restored to its original state whenever data is requested
+by the Array.
+
+A storage transformer serves the same `Abstract store interface`_ as the store_.
+However, it should not persistently store any information necessary to restore the original data,
+but instead propagates this to the next storage transformer or the final store.
+From the perspective of an Array or a previous stage transformer both store and storage transformer follow the same
+protocol and can be interchanged regarding the protocol. The behaviour can still be different,
+e.g. requests may be cached or the form of the underlying data can change.
+
+Storage Transformers may be stacked to combine different functionalities:
+
+.. mermaid::
+
+    graph LR
+      Array --> t1
+      subgraph stack [Storage transformers]
+        t1[Transformer 1] --> t2[...] --> t3[Transformer N]
+      end
+      t3 --> Store
+
+A fixed set of storage providers is recommended for implementation with this protocol:
+
+
+Predefined storage transformers
+-------------------------------
+
+- :ref:`sharding-storage-transformer-v1`
+
 Protocol extensions
 ===================
 
diff --git a/docs/requirements.txt b/docs/requirements.txt
@@ -1,3 +1,3 @@
 sphinx==2.0.1
 pydata-sphinx-theme
-
+sphinxcontrib-mermaid
diff --git a/docs/storage_transformers.rst b/docs/storage_transformers.rst
@@ -0,0 +1,11 @@
+====================
+Storage Transformers
+====================
+
+Under construction.
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Contents:
+
+   storage_transformers/sharding/v1.0
diff --git a/docs/storage_transformers/sharding/sharding.png b/docs/storage_transformers/sharding/sharding.png
diff --git a/docs/storage_transformers/sharding/v1.0.rst b/docs/storage_transformers/sharding/v1.0.rst
@@ -0,0 +1,196 @@
+.. _sharding-storage-transformer-v1:
+
+==========================================
+Sharding storage transformer (version 1.0)
+==========================================
+-----------------------------
+ Editor's draft 18 02 2022
+-----------------------------
+
+Specification URI:
+    @@TODO
+    http://purl.org/zarr/spec/storage_transformers/sharding/1.0
+Issue tracking:
+    `GitHub issues <https://github.com/zarr-developers/zarr-specs/labels/storage_transformers-sharding-v1.0>`_
+Suggest an edit for this spec:
+    `GitHub editor <https://github.com/zarr-developers/zarr-specs/blob/core-protocol-v3.0-dev/docs/storage_transformers/sharding/v1.0.rst>`_
+
+Copyright 2022 `Zarr core development
+team <https://github.com/orgs/zarr-developers/teams/core-devs>`_ (@@TODO
+list institutions?). This work is licensed under a `Creative Commons
+Attribution 3.0 Unported
+License <https://creativecommons.org/licenses/by/3.0/>`_.
+
+----
+
+
+Abstract
+========
+
+This specification defines an implementation of the Zarr abstract
+storage transformer API introducing sharding.
+
+
+Motivation
+==========
+
+Sharding decouples the concept of chunks from storage keys, which become shards.
+This is helpful when the requirements for those don't align:
+
+- Compressible units of chunks often need to be read and written in smaller
+  chunks, whereas
+- storage often is optimized for larger data per entry and fewer entries, e.g.
+  as restricted by the file block size and maximum inode number for typical
+  file systems.
+
+This does not necessarily fit the access patterns of the data, so chunks might
+need to be smaller than one storage key. In those cases sharding decouples those
+entities. One shard corresponds to one storage key, but can contain multiple chunks:
+
+.. image:: sharding.png
+
+
+Document conventions
+====================
+
+Conformance requirements are expressed with a combination of
+descriptive assertions and [RFC2119]_ terminology. The key words
+"MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
+"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the normative
+parts of this document are to be interpreted as described in
+[RFC2119]_. However, for readability, these words do not appear in all
+uppercase letters in this specification.
+
+All of the text of this specification is normative except sections
+explicitly marked as non-normative, examples, and notes. Examples in
+this specification are introduced with the words "for example".
+
+
+Configuration
+=============
+
+:ref:`array-metadata`.
+
+.. code-block::
+
+    {
+      storage_transformers: [
+        {
+          "storage_transformer": "https://purl.org/zarr/spec/storage_transformers/sharding/1.0",
+          "configuration": {
+            "format": "indexed",
+            "chunks_per_shard": [
+                2,
+                2
+            ]
+          }
+      ]
+    }
+
+
+Sharding Mechanism
+=========================
+
+@@TODO
+
+
+Binary shard format
+===================
+
+The only binary format is the ``indexed`` format, as specified by the ``format``
+configuration key. Other binary formats might be added in future versions.
+
+In the indexed binary format chunks are written successively in a shard, where
+unused space between them is allowed, followed by an index referencing them.
+The index holds an `offset, length` pair of little-endian uint64 per chunk,
+the chunks-order in the index is row-major (C) order, e.g. for (2, 2) chunks
+per shard an index would look like:
+
+.. code-block::
+
+    | chunk (0, 0)    | chunk (0, 1)    | chunk (1, 0)    | chunk (1, 1)    |
+    | offset | length | offset | length | offset | length | offset | length |
+    | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 |
+
+
+Empty chunks are denoted by setting both offset and length to `2^64 - 1``.
+The index always has the full shape of all possible chunks per shard,
+even if they are outside of the array size.
+
+The actual order of the chunk-content is not fixed and may be chosen by the implementation
+as all possible write orders are valid according to this specification and therefore can
+be read by any other implementation. When writing partial chunks into an existing shard no
+specific order of the existing chunks may be expected. Some writing strategies might be
+
+* **Fixed order**: Specify a fixed order (e.g. row-, column-major or Morton order).
+  When replacing existing chunks larger or equal sized chunks may be replaced in-place,
+  leaving unused space up to an upper limit  which might possibly be specified.
+  Please note that for regular-sized uncompressed data all chunks have the same size and
+  can therefore be replaced in-place.
+* **Append-only**: Any chunk to write is appended to the existing shard, followed by an updated index.
+
+Any configuration parameters for the write strategy must not be part of the metadata document,
+in a shard I'd propose to use Morton order, but this can easily be changed and customized, since any order can be read.
+
+
+Key translation
+===============
+
+The Zarr store interface is defined in terms of `keys` and `values`,
+where a `key` is a sequence of characters and a `value` is a sequence
+of bytes.
+
+@@TODO
+
+
+Store API implementation
+========================
+
+@@TODO
+
+The section below defines an implementation of the Zarr abstract store
+interface (@@TODO link) in terms of the native operations of this
+storage system. Below ``fspath_to_key()`` is a function that
+translates file system paths to store keys, and ``key_to_fspath()`` is
+a function that translates store keys to file system paths, as defined
+in the section above.
+
+* ``get(key) -> value`` : Read and return the contents of the file at
+  file system path ``key_to_fspath(key)``.
+
+* ``set(key, value)`` : Write ``value`` as the contents of the file at
+  file system path ``key_to_fspath(key)``.
+
+* ``delete(key)`` : Delete the file or directory at file system path
+  ``key_to_fspath(key)``.
+
+* ``list()`` : Recursively walk the file system from the base
+  directory, returning an iterator over keys obtained by calling
+  ``fspath_to_key(fp)`` for each descendant file path ``fp``.
+
+* ``list_prefix(prefix)`` : Obtain a file system path by calling
+  ``key_to_fspath(prefix)``. If the result is a directory path,
+  recursively walk the file system from this directory, returning an
+  iterator over keys obtained by calling ``fspath_to_key(fp)`` for
+  each descendant file path ``fp``.
+
+* ``list_dir(prefix)`` : Obtain a file system path by calling
+  ``key_to_fspath(prefix)``. If the result is a director path, list
+  the directory children. Return a set of keys obtained by calling
+  ``fspath_to_key(fp)`` for each child file path ``fp``, and a set of
+  prefixes obtained by calling ``fspath_to_key(dp)`` for each child
+  directory path ``dp``.
+
+
+References
+==========
+
+.. [RFC2119] S. Bradner. Key words for use in RFCs to Indicate
+   Requirement Levels. March 1997. Best Current Practice. URL:
+   https://tools.ietf.org/html/rfc2119
+
+
+Change log
+==========
+
+@@TODO

Original file line number	Diff line number	Diff line change
`@@ -28,6 +28,7 @@`
`28`	`28`	`# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom`
`29`	`29`	`# ones.`
`30`	`30`	`extensions = [`
	`31`	`+ 'sphinxcontrib.mermaid'`
`31`	`32`	`]`
`32`	`33`
`33`	`34`	`# Add any paths that contain templates here, relative to this directory.`