Support image relocation for docker and oci image types

[glyn]

Image relocation

Before installing a bundle, the user may wish to relocate the images referenced by the bundle to point at a suitable registry.

A highly desirable property of image relocation is that the image digest of the relocated images are the same as those of the original images. This gives the user confidence that the relocated images consist of the same bits as the original images.

Using your own registry has other advantages:

• You can control when those images are updated or deleted.
• The registry can be hosted on a private network for security or other reasons.

Restriction

This feature is restricted to images with type "docker" and "oci".

Relocating image names

An image name consists of a domain name (with optional port) and a path. The image name may also contain a tag and/or a digest. The domain name determines the network location of a registry. The path consists of one or more components separated by forward slashes. The first component is, by convention, a user name providing access control to the image.

Let’s look at some examples:

• The image name docker.io/istio/proxyv2 refers to an image with user name istio residing in the docker hub registry at docker.io.
• The image name projectriff/builder:v1 is short-hand for docker.io/projectriff/builder:v1 which refers to an image with user name projectriff also residing at docker.io.
• The image name gcr.io/cf-elafros/knative-releases/github.com/knative/serving/cmd/autoscaler@sha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef refers to an image with user name cf-elafros residing at gcr.io.

When an image is relocated to a registry, the domain name is set to that of the registry and the path modified so that it starts with the user name that will own the image.

Draft externals

duffle relocate [INPUT-BUNDLE] [OUTPUT-BUNDLE] --repository-prefix [string]

Relocates the images referenced by a bundle and creates a new bundle with an updated image map. Relocation is restricted to images with type "docker" and "oci". Images of other types are not relocated.

The --repository-prefix flag determines the repositories for the relocated images. Each image is tagged with a name starting with the given prefix and pushed to the repository.

For example, if the repository prefix is example.com/user, the image istio/proxyv2 is relocated
to a name starting with example.com/user/ and pushed to a repository hosted by example.com.

Implementation notes

The path of a relocated image may:

• Include the original user name for readability
• Be “flattened” to accommodate registries which do not support hierarchical paths with more than two components
• End with a hash of the image name (to avoid collisions)
• Preserve any tag in the original image name
• Preserve any digest in the original image name.

For instance, when relocated to a registry at example.com with user name user, the above image names might become something like this:

• example.com/user/istio-proxyv2-f93a2cacc6cafa0474a2d6990a4dd1a0
• example.com/user/projectriff-builder-a4a25a99d48adad8310050be267a10ce:v1
• example.com/user/cf-elafros-knative-releases-github.com-knative-serving-cmd-autoscaler-c74d62dc488234d6d1aaa38808898140@sha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef

The hash added to the end of the relocated image path should not depend on any tag and/or digest in
the original image name. This ensures a one-to-one mapping between repositories. In other words, if:

x maps to y

where x and y are image names without tags or digests, then

x:t maps to y:t (for all tags t)

and

x@d maps to y@d (for all digests d).

The intention is to use https://github.com/google/go-containerregistry for accessing OCI image layouts and remote repositories.

[silvin-lubecki]

I think the use case you described is exactly what https://github.com/docker/cnab-to-oci does (@simonferquel please jump in if I'm wrong).
You can pull the bundle from a registry, then push it to another (private or not) registry, and cnab-to-oci will update the image references. It even support image promotion pattern.

With @simonferquel we planned to integrate this library to duffle (it's already integrated in docker-app), had no time yet to do it.

[glyn]

I think the use case you described is exactly what https://github.com/docker/cnab-to-oci does (@simonferquel please jump in if I'm wrong).
You can pull the bundle from a registry, then push it to another (private or not) registry, and cnab-to-oci will update the image references. It even support image promotion pattern.

With @simonferquel we planned to integrate this library to duffle (it's already integrated in docker-app), had no time yet to do it.

We have a use case where the bundle is not stored in a registry. Will that also be supported by cnab-to-oci?

[silvin-lubecki]

We have a use case where the bundle is not stored in a registry. Will that also be supported by cnab-to-oci?

I don't understand the use case described earlier then, it says you want to relocate to another registry.
What do you want to relocate then, if it is not towards another registry?
Or maybe you mean you will relocate the images, but not the bundle itself?
That's not how cnab-to-oci works. See it more as pushing the thick bundle (bundle + the images) to a registry, as a whole.

[glyn]

Or maybe you mean you will relocate the images, but not the bundle itself?

Correct! So it seems that cnab-to-oci won't support our use case. Agreed?

[simonferquel]

Yes, the goal of cnab-to-oci is to be able to publish bundles with guaranteed immutability from the bundle metadata down to the actual component and invocation images.
It also guarantees that if you have pull rights for the bundle, you also have pull rights for the invocation image and component images.

[silvin-lubecki]

We have a use case where the bundle is not stored in a registry.

@glyn could you describe this use case?
It's interesting for cnab-to-oci as we may have all the building blocks to do it, and just need to add a top-level function to handle this relocation.

Maybe you can play with FixupBundle (which already relocates the images and patches the bundle itself with the digested references) and tell me if maybe it fits your need?

There's also a command line implementing this feature, described here.

Depending your feedback, we may just need to integrate all the cnab-to-oci features into duffle.

[glyn]

Thanks @simonferquel. I agree that goal is laudable and maybe our use case will change in the future to align with that goal.

@glyn could you describe this use case?

@silvin-lubecki We currently ship distributions of our software as a zipped archive plus a CLI program. Users relocate images to a (typically) private registry using the CLI. There are a number of steps involved and the process is a little complex to document and for the user to follow.

In the future, we are considering shipping a CNAB instead, either as a thin bundle for users with a DMZ, from which they can relocate images (but not the CNAB itself -- see below) from public repositories to a (typically) private registry, or as a fat bundle for users with no DMZ, in which case they can relocate from an OCI image layout in the bundle to a (typically) private registry.

We expect the user to use a standard CNAB runtime, such as duffle, to perform image relocation as well as installation. (The purpose of the current issue is to enable duffle to support this style of relocation.)

We don't currently have a registry in which to host a CNAB and, as CNAB is still being developed, we don't want to have the user push the CNAB to their private registry. From a registry perspective, it should just look very similar to today: images are pushed to a private registry and our software pulls the images from the private registry.

[glyn]

Maybe you can play with FixupBundle (which already relocates the images and patches the bundle itself with the digested references) and tell me if maybe it fits your need?

I took a look at FixupBundle and the associated command line. These perform a many-one mapping of repositories, whereas we currently use a one-one mapping and would like duffle to do the same (see "Implementation Notes" above).

The main downside of a many-one mapping is that the mapped image names are effectively indistinguishable from each other from a human readability point of view and certainly not easy to relate to the corresponding original image names. In our case, when we relocate kubernetes configuration files containing a number of image names, it's important to distinguish between mapped image names and preserve a readable connection to the original image names. This ensures that the relocated configuration files remain readable by users.

Note that the mapping we currently employ ensures the same authentication/authorisation property as mapping to a single repository: the mapped repositories are authenticated with, and authorised to, a single user.