Merge pull request #222 from jbms/codec-partial-decode

Describe partial decode support for codecs

Author: jstriebel

docs/v3/codecs/blosc/v1.0.rst

@@ -24,8 +24,7 @@ is licensed under a `Creative Commons Attribution 3.0 Unported License
 Abstract
 ========
 
-This specification defines an implementation of the Zarr abstract
-store API using a file system.
+Defines a ``bytes -> bytes`` codec that uses the blosc container format.
 
 
 Status of this document
@@ -106,6 +105,8 @@ default block size::
 Format and algorithm
 ====================
 
+This is a ``bytes -> bytes`` codec.
+
 Blosc is a meta-compressor, which divides an input buffer into blocks,
 then applies an internal compression algorithm to each block, then
 packs the encoded blocks together into a single output buffer with a

docs/v3/codecs/endian/v1.0.rst

@@ -26,8 +26,8 @@ is licensed under a `Creative Commons Attribution 3.0 Unported License
 Abstract
 ========
 
-This specification defines an implementation of the Zarr abstract
-store API using a file system.
+Defines an ``array -> bytes`` codec that encodes arrays of fixed-size numeric
+data types as little endian or big endian in lexicographical order.
 
 
 Status of this document
@@ -65,6 +65,8 @@ endian:
 Format and algorithm
 ====================
 
+This is an ``array -> bytes`` codec.
+
 Each element of the array is encoded using the specified endian variant of its
 default binary representation.  Array elements are encoded in lexicographical
 order.  For example, with ``endian`` specified as ``big``, the ``int32`` data

docs/v3/codecs/gzip/v1.0.rst

@@ -24,8 +24,7 @@ is licensed under a `Creative Commons Attribution 3.0 Unported License
 Abstract
 ========
 
-This specification defines an implementation of the Zarr abstract
-store API using a file system.
+Defines a ``bytes -> bytes`` codec that applies gzip compression.
 
 
 Status of this document
@@ -79,6 +78,8 @@ the Gzip codec configured with a compression level of 1::
 Format and algorithm
 ====================
 
+This is a ``bytes -> bytes`` codec.
+
 Encoding and decoding is performed using the algorithm defined in
 [RFC1951]_.
 

docs/v3/codecs/sharding-indexed/v1.0.rst

@@ -27,8 +27,7 @@ is licensed under a `Creative Commons Attribution 3.0 Unported License
 Abstract
 ========
 
-This specification defines an implementation of the Zarr codec specification 
-for sharding.
+This specification defines a Zarr ``array -> bytes`` codec for sharding.
 
 Sharding logically splits chunks ("shards") into sub-chunks ("inner chunks") 
 that can be individually compressed and accessed. This allows to colocate 
@@ -121,6 +120,8 @@ Sharding can be configured per array in the :ref:`array-metadata` as follows::
 Binary shard format
 ===================
 
+This is an ``array -> bytes`` codec.
+
 In the ``sharding_indexed`` binary format, chunks are written successively in a 
 shard, where unused space between them is allowed, followed by an index 
 referencing them. The index is placed at the end of the file and has a size of 

docs/v3/codecs/transpose/v1.0.rst

@@ -26,7 +26,8 @@ is licensed under a `Creative Commons Attribution 3.0 Unported License
 Abstract
 ========
 
-Defines a codec that permutes the dimensions of the chunk array.
+Defines an ``array -> array`` codec that permutes the dimensions of the chunk
+array.
 
 
 Status of this document
@@ -71,9 +72,7 @@ order:
 Format and algorithm
 ====================
 
-The decoded chunk representation to which this codec is applied must be an
-array.  Implementations must fail if this codec is specified immediately after
-another codec that produces a byte string as its encoded representation.
+This is an ``array -> array`` codec.
 
 Given a chunk array ``A`` with shape ``A_shape`` as the decoded representation,
 the encoded representation is an array ``B`` with the same data type as ``A``

docs/v3/core/v3.0.rst

@@ -1089,6 +1089,36 @@ each of these two representations are defined to be either:
 - a multi-dimensional array of some shape and data type, or
 - a byte string.
 
+Based on the input and output representations for the encode transform,
+codecs can be classified as one of three kinds:
+
+- ``array -> array``
+- ``array -> bytes``
+- ``bytes -> bytes``
+
+.. note::
+
+   ``bytes -> array`` codecs, where after encoding an array as a byte
+   string, it is subsequently transformed back into an array, to then later
+   be transformed back into a byte string, are not currently allowed, due to
+   the lack of a clear use case.
+
+If multiple codecs are specified for an array, each codec is applied
+sequentially; when encoding, the encoded output of codec ``i`` serves as the
+decoded input of codec ``i+1``, and similarly when decoding, the decoded output
+of codec ``i+1`` serves as the encoded input to codec ``i``.  Since ``bytes ->
+array`` codecs are not supported, it follows that the list of codecs must be of
+the following form:
+
+- zero or more ``array -> array`` codecs; followed by
+- at most one ``array -> bytes`` codec; followed by
+- zero or more ``bytes -> bytes`` codecs.
+
+If no ``array -> bytes`` codec is specified, then the default byte
+representation for the data type of the array is used.  For all data types
+currently defined by the core spec, that is equivalent to the ``endian`` codec
+with an endianness of ``little``.
+
 Logically, a codec ``c`` must define three properties:
 
 - ``c.compute_encoded_representation_type(decoded_representation_type)``, a
@@ -1106,11 +1136,31 @@ Logically, a codec ``c`` must define three properties:
 - ``c.decode(encoded_value, decoded_representation_type)``, a procedure that
   computes the decoded representation, and is used when reading an array.
 
-If more than one codec is specified for an array, each codec is applied
-sequentially; when encoding, the encoded output of codec ``i`` serves as the
-decoded input of codec ``i+1``, and similarly when decoding, the decoded
-output of codec ``i+1`` serves as the encoded input to codec ``i``.
+Implementations MAY support partial decoding for certain codecs
+(e.g. sharding, blosc).  Logically, partial decoding may be defined in terms
+of an additional operation:
+
+- ``c.partial_decode(input_handle, decoded_representation_type,
+  decoded_regions)``, where:
+
+  - ``input_handle`` provides an interface for requesting partial reads of
+    the encoded representation and itself supports the same
+    ``partial_decode`` interface;
+  - ``decoded_representation_type`` is the same as for ``c.decode``;
+  - ``decoded_regions`` specifies the regions of the decoded representation
+    that must be returned.
+
+  If the encoded representation is a multi-dimensional array, then
+  ``decoded_regions`` specifies a subset of the array's domain.  If the
+  encoded representation is a byte string, then ``decoded_regions``
+  specifies a list of byte ranges.
+
+.. note::
 
+   If ``partial_decode`` is not supported by a particular codec, it can
+   always be implemented in terms of ``decode`` by simply decoding in full
+   and then satisfying any ``decoded_regions`` requests directly from the
+   cached decoded representation.
 
 Determination of encoded representations
 ----------------------------------------