media-types: Define layer media types with and without '+gzip'

I'd prefer defining this as a structured syntax suffix following RFC
6839, and have filed a pull request to that effect [1].  However, the
current maintainer consensus seems to be to define the compressed and
uncompressed types directly without declaring a structured syntax
suffix pattern [2].  I'm not clear on the reason for avoiding the
structured syntax suffix, but that's the route I've taken in this
commit.

Now that you can choose both compressed or uncompressed media types,
it is easy to clarify DiffIDs by comparing types with and without the
+gzip compression.  media type.  It also allows you to create
image-layout instances where the layers are stored uncompressed, which
may be useful for cases such as:

* Binary diffing between layer blobs for cheaper updates of large
  layers [3].

* Compressing an image-layout tarball for a smaller smaller overall
  tarball (by avoiding the unnecessary fragmentation of compressing
  the individual blob entries).

[1]: opencontainers#332
[2]: opencontainers#332 (comment)
[3]: http://ircbot.wl.linuxfoundation.org/eavesdrop/%23opencontainers/%23opencontainers.2016-08-16.log.html#t2016-08-16T23:35:43

Signed-off-by: W. Trevor King <[email protected]>

Author: wking

config.md

@@ -28,7 +28,8 @@ Changing it means creating a new derived image, instead of changing the existing
 A layer DiffID is a SHA256 digest over the layer's uncompressed tar archive and serialized in the descriptor digest format, e.g., `sha256:a9561eb1b190625c9adb5a9513e72c4dedafc1cb2d4c5236c9a6957ec7dfd5a9`.
 Layers must be packed and unpacked reproducibly to avoid changing the layer DiffID, for example by using tar-split to save the tar headers.
 
-NOTE: the DiffID is different than the layer digest because it is taken over the uncompressed gzipped layer for `application/vnd.oci.image.layer.v1.tar+gzip` types.
+The difference between DiffIDs and the layer digests in the [manifest's `layers`](manifest.md#image-manifest-property-descriptions) is that the layer digest is taken over the blob regardless of compression, while the DiffID is taken after removing any compression.
+For an [`application/vnd.oci.image.layer.v1.tar+gzip`](layer.md) layer, the layer digest is taken over the `application/vnd.oci.image.layer.v1.tar+gzip` content, while the DiffID is take over the [`application/vnd.oci.image.layer.v1.tar`](media-types.md#oci-image-media-types) content.
 
 ### Layer ChainID
 

img/media-types.dot

@@ -3,7 +3,7 @@ digraph G {
     manifestList [shape=note, label="Manifest list\n<<optional>>\napplication/vnd.oci.image.manifest.list.v1+json"]
     manifest [shape=note, label="Image manifest\napplication/vnd.oci.image.manifest.v1+json"]
     config [shape=note, label="Image JSON\napplication/vnd.oci.image.config.v1+json"]
-    layer [shape=note, label="Layer tar+gzip\napplication/vnd.oci.image.layer.v1.tar+gzip\napplication/vnd.oci.image.layer.nondistributable.v1.tar+gzip"]
+    layer [shape=note, label="Layer tar archive\napplication/vnd.oci.image.layer.v1.tar\napplication/vnd.oci.image.layer.v1.tar+gzip\napplication/vnd.oci.image.layer.nondistributable.v1.tar\napplication/vnd.oci.image.layer.nondistributable.v1.tar+gzip"]
   }
 
   manifestList -> manifest [label="1..*"]

layer.md

@@ -4,12 +4,12 @@ This document describes how to serialize a filesystem and filesystem changes lik
 One or more layers are applied on top of each other to create a complete filesystem.
 This document will use a concrete example to illustrate how to create and consume these filesystem layers.
 
-This section defines the `application/vnd.oci.image.layer.v1.tar+gzip` and `application/vnd.oci.image.layer.nondistributable.v1.tar+gzip` [media types](media-types.md).
+This section defines the `application/vnd.oci.image.layer.v1.tar` and `application/vnd.oci.image.layer.nondistributable.v1.tar` [media types](media-types.md).
 
 ## Distributable Format
 
-Layer Changesets for the [mediatype](./media-types.md) `application/vnd.oci.image.layer.v1.tar+gzip` MUST be packaged in a [tar archive][tar-archive] compressed with [gzip][gzip].
-Layer Changesets for the [mediatype](./media-types.md) `application/vnd.oci.image.layer.v1.tar+gzip` MUST NOT include duplicate entries for file paths in the resulting [tar archive][tar-archive].
+Layer Changesets for the [media type](media-types.md) `application/vnd.oci.image.layer.v1.tar` MUST be packaged in [tar archive][tar-archive].
+Layer Changesets for the [media type](media-types.md) `application/vnd.oci.image.layer.v1.tar` MUST NOT include duplicate entries for file paths in the resulting [tar archive][tar-archive].
 
 ## Change Types
 
@@ -208,7 +208,7 @@ To signify that the resource `./etc/my-app-config` MUST be removed when the chan
 
 ## Applying Changesets
 
-Layer Changesets of [mediatype](./media-types.md) `application/vnd.oci.image.layer.v1.tar+gzip` are _applied_, rather than simply extracted as tar archives.
+Layer Changesets of [media type](media-types.md) `application/vnd.oci.image.layer.v1.tar` are _applied_, rather than simply extracted as tar archives.
 
 Applying a layer changeset requires special consideration for the [whiteout](#whiteouts) files.
 
@@ -311,12 +311,11 @@ Any given image is likely to be composed of several of these Image Filesystem Ch
 Due to legal requirements, certain layers may not be regularly distributable.
 Such "non-distributable" layers are typically downloaded directly from a distributor but never uploaded.
 
-Non-distributable layers SHOULD be tagged with an alternative mediatype of `application/vnd.oci.image.layer.nondistributable.v1.tar+gzip`.
+Non-distributable layers SHOULD be tagged with an alternative mediatype of `application/vnd.oci.image.layer.nondistributable.v1.tar`.
 Implementations SHOULD NOT upload layers tagged with this media type; however, such a media type SHOULD NOT affect whether an implementation downloads the layer.
 
 [Descriptors](descriptor.md) referencing non-distributable layers MAY include `urls` for downloading these layers directly; however, the presence of the `urls` field SHOULD NOT be used to determine whether or not a layer is non-distributable.
 
 [libarchive-tar]: https://github.com/libarchive/libarchive/wiki/ManPageTar5#POSIX_ustar_Archives
 [gnu-tar-standard]: http://www.gnu.org/software/tar/manual/html_node/Standard.html
 [tar-archive]: https://en.wikipedia.org/wiki/Tar_(computing)
-[gzip]: http://www.zlib.org/rfc-gzip.html

manifest.md

@@ -54,11 +54,15 @@ Unlike the [Manifest List](manifest-list.md), which contains information about a
         This [descriptor property](descriptor.md#properties) has additional restrictions for `layers[]`.
         Implementations MUST support at least the following media types:
 
-        - [`application/vnd.oci.image.layer.v1.tar+gzip`](layer.md)
-        - [`application/vnd.oci.image.layer.nondistributable.v1.tar+gzip`](layer.md#non-distributable-layers)
+        - [`application/vnd.oci.image.layer.v1.tar`](layer.md)
+        - [`application/vnd.oci.image.layer.v1.tar+gzip`](media-types.md#oci-image-media-types)
+        - [`application/vnd.oci.image.layer.nondistributable.v1.tar`](layer.md#non-distributable-layers)
+        - [`application/vnd.oci.image.layer.nondistributable.v1.tar+gzip`](media-types.md#oci-image-media-types)
 
         Manifests concerned with portability SHOULD use one of the above media types.
 
+        Entries in this field will frequently use the `+gzip` types.
+
 - **`annotations`** *string-string map*
 
     This OPTIONAL property contains arbitrary metadata for the image manifest.

media-types.md

@@ -6,8 +6,10 @@ The following media types identify the formats described here and their referenc
 - `application/vnd.oci.image.manifest.list.v1+json`: [Manifest list](manifest-list.md#manifest-list)
 - `application/vnd.oci.image.manifest.v1+json`: [Image manifest](manifest.md#image-manifest)
 - `application/vnd.oci.image.config.v1+json`: [Image config](config.md)
-- `application/vnd.oci.image.layer.v1.tar+gzip`: ["Layer", as a gzipped tar archive](layer.md)
-- `application/vnd.oci.image.layer.nondistributable.v1.tar+gzip`: ["Layer", as a gzipped tar archive with distribution restrictions](layer.md#non-distributable-layers)
+- `application/vnd.oci.image.layer.v1.tar`: ["Layer", as a tar archive](layer.md)
+- `application/vnd.oci.image.layer.v1.tar+gzip`: ["Layer", as a tar archive](layer.md) compressed with [gzip][rfc1952]
+- `application/vnd.oci.image.layer.nondistributable.v1.tar`: ["Layer", as a tar archive with distribution restrictions](layer.md#non-distributable-layers)
+- `application/vnd.oci.image.layer.nondistributable.v1.tar+gzip`: ["Layer", as a tar archive with distribution restrictions](layer.md#non-distributable-layers) compressed with [gzip][rfc1952]
 
 ## Media Type Conflicts
 
@@ -41,7 +43,7 @@ This section shows where the OCI Image Specification is compatible with formats
 
 - [application/vnd.docker.distribution.manifest.v2+json](https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md#image-manifest-field-descriptions)
 
-### application/vnd.oci.image.rootfs.tar.gzip
+### application/vnd.oci.image.layer.v1.tar+gzip
 
 **Interchangeable and fully compatible mime-types**
 
@@ -61,3 +63,7 @@ The following figure shows how the above media types reference each other:
 
 [Descriptors](descriptor.md) are used for all references.
 The manifest list being a "fat manifest" references one or more image manifests per target platform. An image manifest references exactly one target configuration and possibly many layers.
+
+[rfc1952]: https://tools.ietf.org/html/rfc1952
+[rfc6713]: https://tools.ietf.org/html/rfc6713
+[rfc6839]: https://tools.ietf.org/html/rfc6839

specs-go/v1/mediatype.go

@@ -25,11 +25,20 @@ const (
 	MediaTypeImageManifestList = "application/vnd.oci.image.manifest.list.v1+json"
 
 	// MediaTypeImageLayer is the media type used for layers referenced by the manifest.
-	MediaTypeImageLayer = "application/vnd.oci.image.layer.v1.tar+gzip"
+	MediaTypeImageLayer = "application/vnd.oci.image.layer.v1.tar"
+
+	// MediaTypeImageLayerGzip is the media type used for gzipped layers
+	// referenced by the manifest.
+	MediaTypeImageLayerGzip = "application/vnd.oci.image.layer.v1.tar+gzip"
 
 	// MediaTypeImageLayerNonDistributable is the media type for layers referenced by
 	// the manifest but with distribution restrictions.
-	MediaTypeImageLayerNonDistributable = "application/vnd.oci.image.layer.nondistributable.v1.tar+gzip"
+	MediaTypeImageLayerNonDistributable = "application/vnd.oci.image.layer.nondistributable.v1.tar"
+
+	// MediaTypeImageLayerNonDistributableGzip is the media type for
+	// gzipped layers referenced by the manifest but with distribution
+	// restrictions.
+	MediaTypeImageLayerNonDistributableGzip = "application/vnd.oci.image.layer.nondistributable.v1.tar+gzip"
 
 	// MediaTypeImageConfig specifies the media type for the image configuration.
 	MediaTypeImageConfig = "application/vnd.oci.image.config.v1+json"