Introduction

This PR is mainly a rebase of a very old PR from the containers/image repository by @alexlarsson: containers/image#902. It also adds a small amount of extra debug logging to make the optimizations of this PR easily observable.

Rebased PR Info

The matter of network-efficient container updates has been gaining attention from developers in edge computing as container-based technologies to manage edge device applications have gained traction (e.g. Microshift, flightctl):

• https://issues.redhat.com/browse/OCTOET-1052
• https://issues.redhat.com/browse/OCPSTRAT-2356
• https://issues.redhat.com/browse/RHELBU-3151
• https://issues.redhat.com/browse/RHELBU-3333

So, I was investigating how we might support this need for efficient updates, when I came across Alex's PR to containers/image. While it's far from a perfect solution (garbage collection and race conditions were two big concerns that came up in the discussion), the "best" solution would likely be an extension of the OCI spec, which would also likely require much stronger proven demand to justify. After some discussion with engineers from these BU mentioned above as well as @mheon, we decided that rebasing Alex's PR may be a suitable first step.

As all of the logic needed for this PR was already implemented, just in a different codebase, this seemed like an appropriate task for an LLM. In accordance with Red Hat's AI policy, I've annotated the commits as AI generated. While I've done my best to review the generated code, the low-level details of this repository are unfamiliar to me, so I can't conclusively say on my own whether or not this PR is doing all the right things. However, according to my testing everything seems to be working as expected. A demo of the layer delta patch using Skopeo can be found in this repository: https://github.com/vraiti/oci-deltas.

Original PR Info

NOTE: This was mostly copied from containers/image#902. Only change was to the sample manifest media type and layer annotation names.

Deltas are a way to avoid downloading a full copy if a layer tar file
if you have a previous version of the layer available locally. In
testing these deltas have been shown to be around 10x to 100x smaller
than the .tar.gz files for typical Linux base images.

In the typical client-side case we have some previous version of the
image stored in container-storage somewhere, which means that we have
an uncompressed files available, but not the actual tarball
(compressed or not).

This means we can use github.com/alexlarsson/tar-diff which takes
two tar files and produces a delta file which when applied on the
untar:ed content of the first tarfile produces the (bitwise identical)
content of the uncompressed second tarfile. It just happens that the
uncompressed tarfile is exactly what we need to reproduce, because that
is how the layers are refered to in the image config (the DiffIDs).

How this works is that we use OCI artifacts to store, for each regular
image a manifest with information about the available deltas for the
image. This image looks like a regular manifest, except each layer
contains a tar-diff (as a blob) an uses the existing annotations key
to record which DiffIDs the layer applies to.

For example, a manifest would look like this:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356",
    "size": 3
  },
  "layers": [
    {
      "mediaType": "application/vnd.tar-diff",
      "digest":
"sha256:49402288de20a465616174a38aca4746f46be2c3f9519fe4d14fc7f83f44a32a",
      "size": 7059734,
      "annotations": {
          "io.github.containers.delta.from":
"sha256:b9137868142acd7ce4d62216e2b03e63e9800e2b647bf682492d3e9c5e66277c",
          "io.github.containers.delta.to":
"sha256:c88d2d437799c2879fded33ee358429e1eb954968a25f3153e2e0e26fef7ef28"
      }
    }
  ]
}

The config blob is just an json file containing "{}". Ideally it
should not be of type application/vnd.oci.image.config.v1+json,
because that is reserved for docker-style images. However, as
explained in oras-project/oras#129, docker hub
doesn't currently support any other type. For registries that support
OCI artifacts we should instead use some other type so that tooling
can know that this is not a regular image.

The way we attach the delta manifest to the image is that we store it
in the same repo and a tag name based on the manifest digest like
"delta-${shortid}".

The delta layers record which DiffID they apply to, which is what we
want to use to look up the pre-existing layers to use as delta source
material, and it is what the delta apply will generate. This means
however that using the deltas only works if we're allowed to
substitute blobs, but this doesn't seem to be an issue in the typical
case.