layer: Require pax (IEEE Std 1003.1-2013)

The idea with a spec like this is to define behavior so that different
implementations can interoperate reliably.  When there is an interop
problem betweem implementations A and B, it should be clear from the
spec whether implementation A is broken, implementation B is broken,
or the spec is insufficiently clear.

For example, pax defines 'g' and 'x' typeflags [1] that aren't part of
the older ustar (IEEE Std 1003.1-1988) [2].  Before this commit, if
implementation A produced a layer with a 'g' or 'x' typeflag and
implementation B died unpacking it, it was unclear whose fault it was.
With this commit, it is clearly B's fault (because it does not
understand the pax typeflag).

If implementation A had produces a layer with an 'S' typeflag (which
GNU uses for sparse files [3]) and implementation B died unpacking it,
it is neither party's fault, because pax explicitly makes those values
implementation-defined.  Interop around them is up to out of band
communication between the layer author and layer consumer, and is not
covered by this spec.

The previous "File Types" section listed sockets, but the pax spec has:

  Attempts to archive a socket using ustar interchange format shall
  produce a diagnostic message.

And I see no socket entry in Go's set of tar-type constants [4], so
I'm not sure how they were supported before.

It looks like Go has supported pax since v1.1 [5,6].

[1]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html#tag_20_92_13_02
[2]: https://github.com/libarchive/libarchive/wiki/ManPageTar5#POSIX_ustar_Archives
[3]: https://github.com/libarchive/libarchive/wiki/ManPageTar5#gnu-tar-archives
[4]: https://golang.org/pkg/archive/tar/#pkg-constants
[5]: https://codereview.appspot.com/6700047
[6]: golang/go@1068279

Signed-off-by: W. Trevor King <[email protected]>

Author: wking

config.md

@@ -13,7 +13,7 @@ This specification uses the following terms:
     </dt>
     <dd>
         Image filesystems are composed of <i>layers</i>.
-        Each layer represents a set of filesystem changes in a tar-based <a href="layer.md">layer format</a>, recording files to be added, changed, or deleted relative to its parent layer.
+        Each layer represents a set of filesystem changes in a pax-based <a href="layer.md">layer format</a>, recording files to be added, changed, or deleted relative to its parent layer.
 	Layers do not have configuration metadata such as environment variables or default arguments - these are properties of the image as a whole rather than any particular layer.
         Using a layer-based or union filesystem such as AUFS, or by computing the diff from filesystem snapshots, the filesystem changeset can be used to present a series of image layers as if they were one cohesive filesystem.
     </dd>
@@ -30,8 +30,8 @@ This specification uses the following terms:
         Layer DiffID
     </dt>
     <dd>
-	A layer DiffID is a SHA256 digest over the layer's uncompressed tar archive and serialized in the descriptor digest format, e.g., <code>sha256:a9561eb1b190625c9adb5a9513e72c4dedafc1cb2d4c5236c9a6957ec7dfd5a9</code>.
-	Layers must be packed and unpacked reproducibly to avoid changing the layer ID, for example by using tar-split to save the tar headers.
+	A layer DiffID is a SHA256 digest over the layer's uncompressed pax archive and serialized in the descriptor digest format, e.g., <code>sha256:a9561eb1b190625c9adb5a9513e72c4dedafc1cb2d4c5236c9a6957ec7dfd5a9</code>.
+	Layers must be packed and unpacked reproducibly to avoid changing the layer ID, for example by using tar-split to save the pax headers.
 	NOTE: the DiffID is different than the digest in the manifest list because the manifest digest is taken over the gzipped layer for <code>application/vnd.oci.image.layer.tar+gzip</code> types.
     </dd>
     <dt>

image-layout.md

@@ -103,7 +103,7 @@ $ cat ./blobs/sha256/5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a43335
 ```
 ```
 $ cat ./blobs/sha256/e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f
-[tar stream]
+[pax stream]
 ```
 
 [descriptors]: ./descriptor.md

layer.md

@@ -6,8 +6,13 @@ This document will use a concrete example to illustrate how to create and consum
 
 ## Distributable Format
 
-Layer Changesets for the [mediatype](./media-types.md) `application/vnd.oci.image.layer.tar+gzip` MUST be packaged in [tar archive][tar-archive].
-Layer Changesets for the [mediatype](./media-types.md) `application/vnd.oci.image.layer.tar+gzip` MUST NOT include duplicate entries for file paths in the resulting [tar archive][tar-archive].
+Layer changesets have the [media type](media-types.md) `application/vnd.oci.image.layer.tar+gzip`.
+Layer changesets MUST be packaged in the pax interchange format, as [specified by IEEE Std 1003.1-2013][pax], and compressed using gzip, as [specified by RFC 1952][rfc1952].
+Layer changesets MUST NOT include duplicate entries for target paths (`path` from the pax extended header or the value computed from `prefix` and `name` in the ustar header).
+
+Portable layers SHOULD NOT use features that [IEEE Std 1003.1-2013][pax] leaves unspecified, undefined, or implementation-defined.
+For example, pax reserves `typeflag` values A through Z for custom implementations, so support for unpacking such entries may be mixed.
+[Sparse files](https://en.wikipedia.org/wiki/Sparse_file) SHOULD NOT be used because they [are a GNU extension][tar.5-gnu].
 
 ## Change Types
 
@@ -17,37 +22,10 @@ Types of changes that can occur in a changeset are:
 * Modifications
 * Removals
 
-Additions and Modifications are represented the same in the changeset tar archive.
+Additions and Modifications are have the same representation in the pax archive.
 
 Removals are represented using "[whiteout](#whiteouts)" file entries (See [Representing Changes](#representing-changes)).
 
-### File Types
-
-Throughout this document section, the use of word "files" or "entries" includes:
-
-* regular files
-* directories
-* sockets
-* symbolic links
-* block devices
-* character devices
-* FIFOs
-
-### File Attributes
-
-Where supported, MUST include file attributes for Additions and Modifications include:
-
-* Modification Time (`mtime`)
-* User ID (`uid`)
-    * User Name (`uname`) *secondary to `uid`*
-* Group ID (`gid `)
-    * Group Name (`gname`) *secondary to `gid`*
-* Mode (`mode`)
-* Extended Attributes (`xattrs`)
-* Symlink reference (`linkname`)
-
-[Sparse files](https://en.wikipedia.org/wiki/Sparse_file) SHOULD NOT be used because they lack consistent support across tar implementations.
-
 ## Creating
 
 ### Initial Root Filesystem
@@ -76,7 +54,7 @@ rootfs-c9d-v1/
         my-app-tools
 ```
 
-The `rootfs-c9d-v1` directory is then created as a plain [tar archive][tar-archive] with relative path to `rootfs-c9d-v1`.
+The `rootfs-c9d-v1` directory is then created as a plain [pax archive][pax] with paths relative to `rootfs-c9d-v1`.
 Entries for the following files:
 
 ```
@@ -91,7 +69,7 @@ Entries for the following files:
 ### Populate a Comparison Filesystem
 
 Create a new directory and initialize it with a copy or snapshot of the prior root filesystem.
-Example commands that can preserve [file attributes](#file-attributes) to make this copy are:
+Example commands that can preserve [file attributes][pax] to make this copy are:
 * [cp(1)](http://linux.die.net/man/1/cp): `cp -a rootfs-c9d-v1/ rootfs-c9d-v1.s1/`
 * [rsync(1)](http://linux.die.net/man/1/rsync):  `rsync -aHAX rootfs-c9d-v1/ rootfs-c9d-v1.s1/`
 * [tar(1)](http://linux.die.net/man/1/tar): `mkdir rootfs-c9d-v1.s1 && tar --acls --xattrs -C rootfs-c9d-v1/ -c . | tar -C rootfs-c9d-v1.s1/ --acls --xattrs -x` (including `--selinux` where supported)
@@ -152,12 +130,12 @@ This reflects the removal of `/etc/my-app-config` and creation of a file and dir
 
 ### Representing Changes
 
-A [tar archive][tar-archive] is then created which contains *only* this changeset:
+A [pax archive][pax] is then created which contains *only* this changeset:
 
 - Added and modified files and directories in their entirety
 - Deleted files or directories marked with a [whiteout file](#whiteouts)
 
-The resulting tar archive for `rootfs-c9d-v1.s1` has the following entries:
+The resulting pax archive for `rootfs-c9d-v1.s1` has the following entries:
 
 ```
 ./etc/my-app.d/
@@ -170,10 +148,10 @@ Where the basename name of `./etc/my-app-config` is now prefixed with `.wh.`, an
 
 ## Applying
 
-Layer Changesets of [mediatype](./media-types.md) `application/vnd.oci.image.layer.tar+gzip` are applied rather than strictly extracted in normal fashion for tar archives.
+Layer Changesets of [mediatype](./media-types.md) `application/vnd.oci.image.layer.tar+gzip` are applied rather than strictly extracted in normal fashion for pax archives.
 
 Applying a layer changeset requires consideration for the [whiteout](#whiteouts) files.
-In the absence of any [whiteout](#whiteouts) files in a layer changeset, the archive is extracted like a regular tar archive.
+In the absence of any [whiteout](#whiteouts) files in a layer changeset, the archive is extracted like a regular pax archive.
 
 
 ### Changeset over existing files
@@ -268,7 +246,7 @@ Note that this opaque file will apply to _all_ children, including sub-directori
 
 Implementations SHOULD generate layers using _explicit whiteout_ files, but MUST accept both.
 
-Any given image is likely to be composed of several of these Image Filesystem Changeset tar archives.
+Any given image is likely to be composed of several of these layers.
 
 # Non-Distributable Layers
 
@@ -279,4 +257,6 @@ Layers that have these restrictions SHOULD be tagged with an alternative mediaty
 [Descriptors](descriptor.md) referencing these layers MAY include `urls` for downloading these layers.
 It is implementation-defined whether or not implementations upload layers tagged with this media type.
 
-[tar-archive]: https://en.wikipedia.org/wiki/Tar_(computing)
+[pax]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html#tag_20_92_13
+[rfc1952]: https://tools.ietf.org/html/rfc1952
+[tar.5-gnu]: https://github.com/libarchive/libarchive/wiki/ManPageTar5#gnu-tar-archives

media-types.md

@@ -6,8 +6,8 @@ The following media types identify the formats described here and their referenc
 - `application/vnd.oci.image.manifest.list.v1+json`: [Manifest list](manifest.md#manifest-list)
 - `application/vnd.oci.image.manifest.v1+json`: [Image manifest format](manifest.md#image-manifest)
 - `application/vnd.oci.image.config.v1+json`: [Container config JSON](config.md)
-- `application/vnd.oci.image.layer.tar+gzip`: ["Layer", as a gzipped tar archive](layer.md)
-- `application/vnd.oci.image.layer.nondistributable.tar+gzip`: ["Layer", as a gzipped tar that has distribution restrictions](layer.md#non-distributable-layers)
+- `application/vnd.oci.image.layer.tar+gzip`: ["Layer", as a gzipped pax archive](layer.md)
+- `application/vnd.oci.image.layer.nondistributable.tar+gzip`: ["Layer", as a gzipped pax that has distribution restrictions](layer.md#non-distributable-layers)
 
 ## Compatibility Matrix