Initial commit

normanrz · normanrz · commit 346b3b3bf2ed · 2025-02-18T11:52:11.000+01:00
diff --git a/README.md b/README.md
@@ -0,0 +1,52 @@
+# zarr-extensions
+
+This repository contains the specification for Zarr extensions for the [Zarr version 3 specification](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html).
+
+It is the normative source for registering names of Zarr v3 extensions.
+
+## Registering an extension
+
+To register an extension, open a new PR with a new extension directory under the relevant extension point:
+
+ * [Codecs](./codecs/README.md)
+ * [Data Types](./dtype/README.md)
+ * [Chunk Key Encoding](./chunk-key-encodings/README.md)
+ * [Chunk Grid](./chunk-grids/README.md)
+ * [Storage Transformers](./storage-transformers/README.md)
+
+Each extension MUST have a `README.md` file that describes the extension and its metadata specification.
+Extensions SHOULD have a `schema.json` file that contains the JSON schema for the metadata, if the README.md does not provide a link to an external schema.
+Please note that all extensions documents will be licensed under the [Creative Commons Attribution 3.0 Unported License](https://creativecommons.org/licenses/by/3.0/).
+Only open a PR if you are willing to license your extension under this license.
+
+The PR will be reviewed by the [Zarr steering council](https://github.com/orgs/zarr-developers/teams/steering-council).
+We aim to be very open about registering extensions.
+The review will be done largely based on avoiding confusing extension names and preventing malicious activity as well as maintaining the formal requirements of the extensions.
+Extension maintainers are responsible for their extensions.
+Updates to the extensions will also be reviewed by the steering council.
+
+
+## Document conventions
+
+These conventions are used for all extension specification documents in this repository.
+
+Conformance requirements are expressed with a combination of descriptive
+assertions and [RFC2119] terminology. The key words "MUST", "MUST NOT",
+"REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY",
+and "OPTIONAL" in the normative parts of specification documents are to be interpreted as
+described in [RFC2119]. However, for readability, these words do not appear in
+all uppercase letters in specification documents.
+
+All of the text of specification documents are normative except sections explicitly
+marked as non-normative, examples, and notes. Examples in specification documents are
+introduced with the words "for example".
+
+[RFC2119] S. Bradner. Key words for use in RFCs to Indicate
+   Requirement Levels. March 1997. Best Current Practice. URL:
+   https://tools.ietf.org/html/rfc2119
+
+[RFC2119]: https://tools.ietf.org/html/rfc2119
+
+## License
+
+All extensions are licensed under the [Creative Commons Attribution 3.0 Unported License](https://creativecommons.org/licenses/by/3.0/).
diff --git a/chunk-grids/README.md b/chunk-grids/README.md
@@ -0,0 +1,3 @@
+# Chunk grids
+
+A chunk grid defines a set of chunks which contain the elements of an array. 
diff --git a/chunk-grids/regular/README.md b/chunk-grids/regular/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#regular-grids
diff --git a/chunk-key-encodings/README.md b/chunk-key-encodings/README.md
@@ -0,0 +1,3 @@
+# Chunk key encodings
+
+The mapping from chunk grid cell coordinates to keys in the underlying store.
diff --git a/chunk-key-encodings/default/README.md b/chunk-key-encodings/default/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#chunk-key-encoding
diff --git a/chunk-key-encodings/v2/README.md b/chunk-key-encodings/v2/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#chunk-key-encoding
diff --git a/codecs/README.md b/codecs/README.md
@@ -0,0 +1,8 @@
+# Codecs
+
+Chunks are encoded into a binary representation for storage in a store, using the chain of codecs.
+There are three types of codecs:
+
+* Array-to-array codecs
+* Array-to-bytes codecs
+* Bytes-to-bytes codecs
diff --git a/codecs/blosc/README.md b/codecs/blosc/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/blosc/v1.0.html
diff --git a/codecs/bytes/README.md b/codecs/bytes/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/bytes/v1.0.html
diff --git a/codecs/crc32c/README.md b/codecs/crc32c/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/crc32c/v1.0.html
diff --git a/codecs/gzip/README.md b/codecs/gzip/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/gzip/v1.0.html
diff --git a/codecs/sharding_indexed/README.md b/codecs/sharding_indexed/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/v1.0.html
diff --git a/codecs/sharding_indexed/sharding.png b/codecs/sharding_indexed/sharding.png
diff --git a/codecs/transpose/README.md b/codecs/transpose/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/transpose/v1.0.html
diff --git a/codecs/zstd/README.md b/codecs/zstd/README.md
@@ -0,0 +1,58 @@
+# Zstd codec
+
+Defines a `bytes -> bytes` codec that applies zstd compression.
+
+## Codec name
+
+The value of the `name` member in the codec object MUST be `zstd`.
+
+## Configuration parameters
+
+### `level`
+An integer from -131072 to 22 which controls the speed and level
+of compression (has no impact on decoding).  A value of 0 indicates to use
+the default compression level.  Otherwise, a higher level is expected to
+achieve a higher compression ratio at the cost of lower speed.
+
+### `checksum` (Optional)
+A boolean that indicates whether to store a checksum when writing that will
+be verified when reading. Should be omitted if false.
+
+## Example
+
+For example, the array metadata below specifies that the compressor is the Zstd
+codec configured with a compression level of 1 and with the checksum stored:
+
+```json
+{
+    "codecs": [{
+        "name": "zstd",
+        "configuration": {
+            "level": 1,
+            "checksum": true
+        }
+    }],
+}
+```
+
+## Format and algorithm
+
+This is a `bytes -> bytes` codec.
+
+Encoded data should conform to the Zstandard file format [RFC8878].
+
+## References
+
+[RFC8878] Y. Collet. Zstandard Compression and the
+   'application/zstd' Media Type. Februrary 2021. Informational. URL:
+   https://tools.ietf.org/html/rfc8878
+
+[RFC8878]: https://tools.ietf.org/html/rfc8878
+
+## Change log
+
+No changes yet.
+
+## Current maintainers
+
+* Jeremy Maitin-Shepard ([@jbms](https://github.com/jbms)), Google
diff --git a/codecs/zstd/schema.json b/codecs/zstd/schema.json
@@ -0,0 +1,27 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "properties": {
+    "name": {
+      "type": "string",
+      "enum": ["zstd"]
+    },
+    "configuration": {
+      "type": "object",
+      "properties": {
+        "level": {
+          "type": "number",
+          "minimum": -131072,
+          "maximum": 22
+        },
+        "checksum": {
+          "type": "boolean"
+        }
+      },
+      "required": ["level"],
+      "additionalProperties": false
+    }
+  },
+  "required": ["name", "configuration"],
+  "additionalProperties": false
+}
diff --git a/data-types/README.md b/data-types/README.md
@@ -0,0 +1,3 @@
+# Data types
+
+A data type describes the set of possible binary values that an array element may take.
diff --git a/data-types/bool/README.md b/data-types/bool/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/complex128/README.md b/data-types/complex128/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/complex64/README.md b/data-types/complex64/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/float16/README.md b/data-types/float16/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/float32/README.md b/data-types/float32/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/float64/README.md b/data-types/float64/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/int16/README.md b/data-types/int16/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/int32/README.md b/data-types/int32/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/int64/README.md b/data-types/int64/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/int8/README.md b/data-types/int8/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/r/README.md b/data-types/r/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/uint16/README.md b/data-types/uint16/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/uint32/README.md b/data-types/uint32/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/uint64/README.md b/data-types/uint64/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/data-types/uint8/README.md b/data-types/uint8/README.md
@@ -0,0 +1 @@
+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#data-types
diff --git a/storage-transformers/README.md b/storage-transformers/README.md
@@ -0,0 +1,3 @@
+# Storage transformers
+
+A storage transformer modifies a request to read or write data before passing that request to the following transformer or store.

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+# Chunk grids`
	`2`	`+`
	`3`	`+A chunk grid defines a set of chunks which contain the elements of an array.`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+See https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#regular-grids`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+# Chunk key encodings`
	`2`	`+`
	`3`	`+The mapping from chunk grid cell coordinates to keys in the underlying store.`