Merge pull request #99 from alimanfoo/fix-key-suffix-omissions-20201016

Author: Carreau

docs/protocol/core/v3.0.rst

@@ -1129,50 +1129,20 @@ that an implementation of this specification could be modular and
 allow for different store implementations to be used.
 
 The store interface defines a set of operations involving `keys` and
-`values`. In the context of this interface, a `key` is any
-string containing only characters in the ranges ``a-z``, ``A-Z``,
-``0-9``, or in the set ``/.-_``, and a `value` is any sequence of
-bytes. It is assumed that the store holds (`key`, `value`) pairs, with
-only one such pair for any given `key`. I.e., a store is a mapping
-from keys to values. It is also assumed that keys are case sensitive, 
-i.e., the keys "foo" and "FOO" are different.
+`values`. In the context of this interface, a `key` is any string
+containing only characters in the ranges ``a-z``, ``A-Z``, ``0-9``, or
+in the set ``/.-_``, where the final character is **not** a ``/``
+character. A `value` is any sequence of bytes.
 
-A store can make the following assumption on the structures of the keys it will receive:
-
-- A key always:
-  - start with ``meta/``
-  - start  with ``data/``
-  - is exactly ``zarr.json``.
-
-- Most of the keys:
-  - start with ``meta/root``
-  - start with ``data/root``
-
-
-- List operations ``list_dir`` will always be passed keys ending with a trailing
-  slash, that is to say it will only be asked to work with complete node names.
-
-Store implementation can assume they will only be given trailing slashes, and
-protocol implementation MUST pass trailing slashes to underlying stores.
-
-For example, a store containing the following keys:
-
- - ``meta/root/2018/.group``
- - ``meta/root/2018-01/.group``
- - ``meta/root/2018/bar/.array``
- - ``data/root/2018/bar/0.0``
-
-The following queries are invalid:
-  - ``list_dir('201')`` is invalid as ``"201"`` is not an existing node.
-  - ``list_dir('2018')`` is invalid queries as ``"2018"`` does not ends with a ``/``,
-
-This is valid:
-  - ``list_dir('2018/')``
-  - ``list_dir('2018-01/')``
-
-This allows store implementation to avoid having to check for trailing slashes,
-and avoid issues like "list_dir('2018')" returning values likes ``-01``
+It is assumed that the store holds (`key`, `value`) pairs, with only
+one such pair for any given `key`. I.e., a store is a mapping from
+keys to values. It is also assumed that keys are case sensitive, i.e.,
+the keys "foo" and "FOO" are different.
 
+The store interface also defines some operations involving
+`prefixes`. In the context of this interface, a prefix is a string
+containing only characters that are valid for use in `keys` and ending
+with a trailing ``/`` character.
 
 The store operations are grouped into three sets of capabilities:
 **readable**, **writeable** and **listable**. It is not necessary for
@@ -1197,15 +1167,10 @@ A **writeable store** supports the following operations:
     | Parameters: `key`
     | Output: none
 
-``delete_prefix`` - Delete all keys with the given prefix from the store, include the prefix itself if it exists as a key:
-
-    | Parameter: `key`
-    | Output: None
-
+``delete_prefix`` - Delete all keys with the given prefix from the store:
 
-    Clients of delete_prefix should pay attention to pass a trailing slash on
-    the key to delete a complete dataset or group, otherwise the store may
-    delete similar keys.
+    | Parameter: `prefix`
+    | Output: none
 
 A **listable store** supports any one or more of the following
 operations:
@@ -1215,7 +1180,6 @@ operations:
     | Parameters: none
     | Output: set of `keys`
 
-
 ``list_prefix`` - Retrieve all keys with a given prefix.
 
     | Parameters: `prefix`
@@ -1228,56 +1192,22 @@ operations:
     with a trailing slash ``/`` and store can assume there is as least one key
     that stars with prefix.
 
-
 ``list_dir`` - Retrieve all keys and prefixes with a given prefix and
 which do not contain the character "/" after the given prefix.
 
-    | Parameters: `prefix`, ends with a trailing slash ``/``
+    | Parameters: `prefix`
     | Output: set of `keys` and set of `prefixes`
 
     For example, if a store contains the keys "a/b", "a/c", "a/d/e",
     "a/f/g", then ``list_dir("a/")`` would return keys "a/b" and "a/c"
-    and prefixes "a/d/" and "a/f/".
-
-    On non-existing prefix, store may return the empty set.
-
+    and prefixes "a/d/" and "a/f/". ``list_dir("b/")`` would return
+    the empty set.
 
-    Note: The requirement on trailing slashes is to avoid
-    search returning keys in the same hierarchy level but longer name, and
-    potentially expensive logic testing for the present of trailing slash on
-    each query. e.g:
 
-     - /meta/foo
-     - /meta/foo/dataset
-     - /meta/foobar
-
-     list_dir('/meta/foo') == '/meta/foo'&'/meta/foobar'
-     list_dir('/meta/foo/') == '/meta/foo/dataset'
-
-
-    Stores Must return trailing slashes in key responses when those
-    are prefix of other keys.
-
-    Like would ``list_dir('/meta/mydir')`` returns:
-       - ``/meta/path1``
-       - ``/meta/path2``
-       - ``/meta/path3/``
-       - ``/meta/path4/``
-
-    Thus we know that ``path1``, and ``path2`` are terminal objects with data,
-    and that ``/meta/path3`` and ``/meta/path4``.
-
-
-    This is similar to ``ls -p`` on Unix systems.
-
-    Note: In practice this means that this means most returned keys always ends in
-    ``/``, ``.json``, ``.array``, ``.group``, they will otherwise be chunks
-    data.
-
-Note that because keys are case sensitive, it is assumed that the operations 
-``set("foo", a)`` and ``set("FOO", b)`` will result in two separate (key, value) 
-pairs being stored. Subsequently ``get("foo")`` will return *a* and ``get("FOO")`` 
-will return *b*. 
+Note that because keys are case sensitive, it is assumed that the
+operations ``set("foo", a)`` and ``set("FOO", b)`` will result in two
+separate (key, value) pairs being stored. Subsequently ``get("foo")``
+will return *a* and ``get("FOO")`` will return *b*.
 
 
 Store implementations
@@ -1351,30 +1281,31 @@ Storage keys
 The entry point metadata document is stored under the key ``zarr.json``.
 
 For a group at a non-root hierarchy path `P`, the metadata key for the
-group metadata document is formed by concatenating ``meta/root``, `P`,
-and ``.group``.
+group metadata document is formed by concatenating "meta/root", `P`,
+".group", and the metadata key suffix (which defaults to ".json").
 
 For example, for a group at hierarchy path ``/foo/bar``, the
-corresponding metadata key is ``meta/root/foo/bar.group``.
+corresponding metadata key is "meta/root/foo/bar.group.json".
 
 For an array at a non-root hierarchy path `P`, the metadata key for
-the array metadata document is formed by concatenating "meta/root", `P`,
-and ".array". The data key for array chunks is formed by concatenating
-"data", `P`, "/", and the chunk identifier as defined by the chunk
-grid layout.
+the array metadata document is formed by concatenating "meta/root",
+`P`, ".array", and the metadata key suffix.
 
-To get the path ``P`` from a key, either remove the trailing ``.array`` or
-``.group`` as well as the ``meta/root`` prefix.
+The data key for array chunks is formed by concatenating "data", `P`,
+"/", and the chunk identifier as defined by the chunk grid layout.
+
+To get the path ``P`` from a metadata key, remove the trailing
+".array.json" or ".group.json" and the "meta/root" prefix.
 
 For example, for an array at hierarchy path "/foo/baz", the
-corresponding metadata key is ``meta/root/foo/baz.array``. If the array
-has two dimensions and a regular chunk grid, the data key for the
-chunk with grid coordinates (0, 0) is "data/root/foo/baz/c0/0".
+corresponding metadata key is "meta/root/foo/baz.array.json". If the
+array has two dimensions and a regular chunk grid, the data key for
+the chunk with grid coordinates (0, 0) is "data/root/foo/baz/c0/0".
 
-If the root node is a group, the metadata key is ``meta/root.group``. If
-the root node is an array, the metadata key is "meta/root.array", and
-the data keys are formed by concatenating "data/root/" and the chunk
-identifier.
+If the root node is a group, the metadata key is
+"meta/root.group.json". If the root node is an array, the metadata key
+is "meta/root.array.json", and the data keys are formed by
+concatenating "data/root/" and the chunk identifier.
 
 
 .. list-table:: Metadata Storage Key example
@@ -1388,25 +1319,22 @@ identifier.
       - `zarr.json`
     * - Array (Root)
       - `/`
-      - `meta/root.array`
+      - `meta/root.array.json`
     * - Group (Root)
       - `/`
-      - `meta/root.group`
+      - `meta/root.group.json`
     * - Group
       - `/foo`
-      - `meta/root/foo.group`
+      - `meta/root/foo.group.json`
     * - Array
       - `/foo`
-      - `meta/root/foo.array`
+      - `meta/root/foo.array.json`
     * - Group
       - `/foo/bar`
-      - `meta/root/foo/bar.group`
+      - `meta/root/foo/bar.group.json`
     * - Array
       - `/foo/baz`
-      - `meta/root/foo/baz.array`
-
-
-
+      - `meta/root/foo/baz.array.json`
 
 
 .. list-table:: Data Storage Key example
@@ -1464,35 +1392,37 @@ Let "+" be the string concatenation operator.
 
 **Store element values in an array**
 
-    To store element in an array at path `P` and coordinate (`i`, `j`, ...)
-    perform ``set(data_key(P, i, j, ...), value)``, where `value` is the
-    serialisation of the corresponding chunk following the metadata that is
-    or will be stored in ``array_meta_key(P)``. 
+    To store element in an array at path `P` and coordinate (`i`, `j`,
+    ...)  perform ``set(data_key(P, i, j, ...), value)``, where
+    `value` is the serialisation of the corresponding chunk, encoded
+    according to the information in the array metadata stored under
+    the key ``array_meta_key(P)``.
 
 **Retrieve element values in an array**
 
-    To retrieve element in an array at path `P` and coordinate (`i`, `j`, ...)
-    perform ``get(data_key(P, i, j, ...), value)``, where `value` is the
-    serialisation of the corresponding chunk following the metadata stored at
-    ``array_meta_key(P)``. 
+    To retrieve element in an array at path `P` and coordinate (`i`,
+    `j`, ...)  perform ``get(data_key(P, i, j, ...), value)``, where
+    `value` is the serialisation of the corresponding chunk, encoded
+    according to the array metadata stored at ``array_meta_key(P)``.
 
 **Discover children of a group**
 
     To discover the children of a group at hierarchy path `P`, perform
     ``list_dir("meta/root" + P + "/")``. Any returned key ending in
-    ".array" indicates an array. Any returned key ending in
-    ".group" indicates a group. Any returned prefix indicates a
+    ".array.json" indicates an array. Any returned key ending in
+    ".group.json" indicates a group. Any returned prefix indicates a
     child group implied by some descendant.
 
     For example, if a group is created at path "/foo/bar" and an array
     is created at path "/foo/baz/qux", then the store will contain the
-    keys "meta/root/foo/bar.group" and "meta/root/foo/bar/baz/qux.array". Groups
-    at paths "/", "/foo" and "/foo/baz" have not been explicitly
-    created but are implied by their descendants. To list the children
-    of the group at path "/foo", perform ``list_dir("meta/root/foo/")``,
-    which will return the key "meta/root/foo/bar.group" and the prefix
-    "meta/root/foo/baz/". From this it can be inferred that child groups
-    "/foo/bar" and "/foo/baz" are present.
+    keys "meta/root/foo/bar.group.json" and
+    "meta/root/foo/bar/baz/qux.array.json". Groups at paths "/",
+    "/foo" and "/foo/baz" have not been explicitly created but are
+    implied by their descendants. To list the children of the group at
+    path "/foo", perform ``list_dir("meta/root/foo/")``, which will
+    return the key "meta/root/foo/bar.group.json" and the prefix
+    "meta/root/foo/baz/". From this it can be inferred that child
+    groups "/foo/bar" and "/foo/baz" are present.
 
     If a store does not support any of the list operations then
     discovery of group children is not possible, and the contents of
@@ -1501,54 +1431,49 @@ Let "+" be the string concatenation operator.
 
 **Discover all nodes in a hierarchy**
 
-    To discover all nodes in a hierarchy, one can call ``list("meta/")``.
-     - all keys represent either explicit group or arrays.
-     - all intermediate prefixes ending in a ``/`` are implicit groups.
+    To discover all nodes in a hierarchy, one can call
+    ``list("meta/")``. All keys represent either explicit group or
+    arrays. All intermediate prefixes ending in a ``/`` are implicit
+    groups.
 
 **Delete a group or array**
 
-    To delete an array it is necessary to
-      - delete the metadata document for the array, (meta/P.array)
-      - delete all keys which prefix have path pointing to this to this array.  (data/P/\*)
+    To delete an array at path `P`:
+      - delete the metadata document for the array, ``delete(array_meta_key(P))``
+      - delete all data keys which prefix have path pointing to this to this array,
+	``delete_prefix("data/root" + P + "/")``
 
-    To delete a implicit group.
-      - delete all arrays under this group
-      - it should be sufficient to delete all the keys starting with prefix meta/P/ and data/P/
+    To delete an implicit group at path `P`:
+      - delete all nodes under this group - it should be sufficient to
+	perform ``delete_prefix("meta/root" + P + "/")`` and
+	``delete_prefix("data/root" + P + "/")``.
 
-    To delete an explicit group.
-      - delete all arrays under this group,
-      - delete all keys with meta/P/ prefix, meta/P/groups all keys with /data/P prefix,
+    To delete an explicit group at path `P`:
+      - delete the metadata document for the group, ``delete(group_meta_key(P))``
+      - delete all nodes under this group - it should be sufficient to
+	perform ``delete_prefix("meta/root" + P + "/")`` and
+	``delete_prefix("data/root" + P + "/")``.
 
-    Note that store implementation may decide to reify implicit groups and thus
-    protocol implementation should attempt to delete the .meta/P/.group file if
-    they really wish to delete an empty implicit group.
+    Note that store implementation may decide to reify implicit groups
+    and thus protocol implementation should attempt to delete the
+    group metadata file if they really wish to delete an empty
+    implicit group. @@TODO clarify this
 
     Store implementation are also allowed to delete any implicit parent of a
     deleted implicit groups, so a protocol implementation should make sure to
-    reify a parent group if they need to keep it. For example assuming the
-    following:
-
-     >>>  z = new_dataset()
-     >>>  z.create_array('/path/to/array')
-
-     >>>  z.delete_array('/path/to/array')
-
-     This may not be sufficient to delete the group ``/path/to/``, as a store
-     implementation, and thus removing ``/path/to/`` may need an implmentation
-     to explicitly call
-
-     >>> z.delete_group('/path/to/')
+    reify a parent group if they need to keep it. @@TODO clarify this
 
-     Even if an explicit group was not explicitly created.
 
 **Determine if a node exists**
 
-    To determine if a node exists at path `P`, you need to check the existence
-    of one of ``get("meta/root"+P+".array")``, ``get("meta/root"+P+".group")``
-    or ``get("meta/root"+P+"/")``.
+    To determine if a node exists at path ``P``, try in the following
+    order ``get(array_meta_key(P))`` (success implies an array at
+    ``P``); ``get(group_meta_key(P))`` (success implies an explicit
+    group at ``P``); ``list_dir("meta/root" + P + "/")`` (non-empty
+    result set implies an implicit group at ``P``).
 
     .. note::
-       For listable store, ``listdir(parent(P))`` can be an alternative.
+       For listable store, ``list_dir(parent(P))`` can be an alternative.
 
 
 Protocol extensions