Merge pull request #42 from alimanfoo/filters

Implementation of filters.

Author: alimanfoo

.coveragerc

@@ -0,0 +1,3 @@
+[run]
+omit = zarr/meta_v1.py
+

c-blosc

@@ -8,5 +8,5 @@ API reference
     api/core
     api/hierarchy
     api/storage
-    api/compressors
+    api/codecs
     api/sync

docs/api.rst

@@ -0,0 +1,27 @@
+Compressors and filters (``zarr.codecs``)
+=========================================
+.. module:: zarr.codecs
+
+This module contains compressor and filter classes for use with Zarr.
+
+Other codecs can be registered dynamically with Zarr. All that is required
+is to implement a class that provides the same interface as the classes listed
+below, and then to add the class to the ``codec_registry``. See the source
+code of this module for details.
+
+.. autoclass:: Codec
+
+    .. automethod:: encode
+    .. automethod:: decode
+    .. automethod:: get_config
+    .. automethod:: from_config
+
+.. autoclass:: Blosc
+.. autoclass:: Zlib
+.. autoclass:: BZ2
+.. autoclass:: LZMA
+.. autoclass:: Delta
+.. autoclass:: FixedScaleOffset
+.. autoclass:: Quantize
+.. autoclass:: PackBits
+.. autoclass:: Categorize

docs/api/codecs.rst

@@ -8,3 +8,4 @@ The Array class (``zarr.core``)
     .. automethod:: __setitem__
     .. automethod:: resize
     .. automethod:: append
+    .. automethod:: view

docs/api/compressors.rst

@@ -12,3 +12,5 @@ can be used as a Zarr array store.
 .. autoclass:: DictStore
 .. autoclass:: DirectoryStore
 .. autoclass:: ZipStore
+
+.. autofunction:: migrate_1to2

docs/api/core.rst

@@ -17,6 +17,7 @@ Highlights
 * Read an array concurrently from multiple threads or processes.
 * Write to an array concurrently from multiple threads or processes.
 * Organize arrays into hierarchies via groups.
+* Use filters to preprocess data and improve compression.
 
 Status
 ------

docs/api/storage.rst

@@ -13,13 +13,28 @@ Support has been added for organizing arrays into hierarchies via groups. See
 the tutorial section on :ref:`tutorial_groups` and the :mod:`zarr.hierarchy`
 API docs for more information.
 
-To accommodate support for hierarchies the Zarr format has been modified. See
-the :ref:`spec_v2` for more information.
+Filters
+~~~~~~~
+
+Support has been added for configuring filters to preprocess chunk data prior 
+to compression. See the tutorial section on :ref:`tutorial_filters` and the 
+:mod:`zarr.filters` API docs for more information.
 
 Other changes
 ~~~~~~~~~~~~~
 
-* The bundled Blosc library has been upgraded to version 1.10.2.
+To accommodate support for hierarchies and filters, the Zarr metadata format 
+has been modified. See the :ref:`spec_v2` for more information. To migrate an 
+array stored using Zarr version 1.x, use the :func:`zarr.storage.migrate_1to2` 
+function.
+
+The bundled Blosc library has been upgraded to version 1.10.2.
+
+Acknowledgments
+~~~~~~~~~~~~~~~
+
+Thanks to Matthew Rocklin (mrocklin_), Stephan Hoyer (shoyer_) and
+Francesc Alted (FrancescAlted_) for contributions and comments.
 
 .. _release_1.1.0:
 

docs/index.rst

@@ -4,7 +4,7 @@ Zarr storage specification version 2
 ====================================
 
 This document provides a technical specification of the protocol and format 
-used for storing a Zarr array. The key words "MUST", "MUST NOT", "REQUIRED", 
+used for storing Zarr arrays. The key words "MUST", "MUST NOT", "REQUIRED", 
 "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and 
 "OPTIONAL" in this document are to be interpreted as described in `RFC 2119 
 <https://www.ietf.org/rfc/rfc2119.txt>`_.
@@ -56,42 +56,47 @@ chunks
 dtype
     A string or list defining a valid data type for the array. See also
     the subsection below on data type encoding.
-compression
-    A string identifying the primary compression library used to compress
-    each chunk of the array.
-compression_opts
-    An integer, string or dictionary providing options to the primary
-    compression library.
+compressor
+    A JSON object identifying the primary compression codec and providing 
+    configuration parameters, or ``null`` if no compressor is to be used. 
+    The object MUST contain an ``"id"`` key identifying the codec to be used.
 fill_value
     A scalar value providing the default value to use for uninitialized
-    portions of the array.
+    portions of the array, or ``null`` if no fill_value is to be used.
 order
     Either "C" or "F", defining the layout of bytes within each chunk of the
     array. "C" means row-major order, i.e., the last dimension varies fastest;
     "F" means column-major order, i.e., the first dimension varies fastest.
+filters
+    A list of JSON objects providing codec configurations, or ``null`` if no
+    filters are to be applied. Each codec configuration object MUST contain a
+    ``"id"`` key identifying the codec to be used.
 
 Other keys MUST NOT be present within the metadata object.
 
 For example, the JSON object below defines a 2-dimensional array of 64-bit 
 little-endian floating point numbers with 10000 rows and 10000 columns, divided 
 into chunks of 1000 rows and 1000 columns (so there will be 100 chunks in total 
 arranged in a 10 by 10 grid). Within each chunk the data are laid out in C 
-contiguous order, and each chunk is compressed using the Blosc compression 
-library::
+contiguous order. Each chunk is encoded using a delta filter and compressed
+using the Blosc compression library prior to storage::
 
     {
         "chunks": [
             1000,
             1000
         ],
-        "compression": "blosc",
-        "compression_opts": {
-            "clevel": 5,
+        "compressor": {
+            "id": "blosc",
             "cname": "lz4",
+            "clevel": 5,
             "shuffle": 1
         },
         "dtype": "<f8",
-        "fill_value": null,
+        "fill_value": "NaN",
+        "filters": [
+            {"id": "delta", "dtype": "<f8", "astype": "<f4"}
+        ],
         "order": "C",
         "shape": [
             10000,
@@ -142,7 +147,6 @@ Positive Infinity  ``"Infinity"``
 Negative Infinity  ``"-Infinity"``
 =================  ===============
 
-
 Chunks
 ~~~~~~
 
@@ -176,6 +180,16 @@ array dimension is not exactly divisible by the length of the corresponding
 chunk dimension then some chunks will overhang the edge of the array. The 
 contents of any chunk region falling outside the array are undefined.
 
+Filters
+~~~~~~~
+
+Optionally a sequence of one or more filters can be used to transform chunk
+data prior to compression. When storing data, filters are applied in the order
+specified in array metadata to encode data, then the encoded data are passed to
+the primary compressor. When retrieving data, stored chunk data are
+decompressed by the primary compressor then decoded using filters in the
+reverse order.
+
 Hierarchies
 -----------
 
@@ -279,7 +293,7 @@ Create an array::
     >>> import zarr
     >>> store = zarr.DirectoryStore('example')
     >>> a = zarr.create(shape=(20, 20), chunks=(10, 10), dtype='i4',
-    ...                 fill_value=42, compression='zlib', compression_opts=1,
+    ...                 fill_value=42, compressor=zarr.Zlib(level=1),
     ...                 store=store, overwrite=True)
 
 No chunks are initialized yet, so only the ".zarray" and ".zattrs" keys
@@ -297,10 +311,13 @@ Inspect the array metadata::
             10,
             10
         ],
-        "compression": "zlib",
-        "compression_opts": 1,
+        "compressor": {
+            "id": "zlib",
+            "level": 1
+        },
         "dtype": "<i4",
         "fill_value": 42,
+        "filters": null,
         "order": "C",
         "shape": [
             20,
@@ -452,6 +469,10 @@ Changes in version 2
 * Added support for storing multiple arrays in the same store and organising
   arrays into hierarchies using groups.
 * Array metadata is now stored under the ".zarray" key instead of the "meta"
-  key
+  key.
 * Custom attributes are now stored under the ".zattrs" key instead of the
-  "attrs" key
+  "attrs" key.
+* Added support for filters.
+* Changed encoding of "fill_value" field within array metadata.
+* Changed encoding of compressor information within array metadata to be
+  consistent with representation of filter information.