OCI export should only put the tag into org.opencontainers.image.ref.name in index.json

[ChristianWolf2]

I noticed the following issue while trying to use the oci exporter to export an image directly into a filesystem folder, which is used by the zot registry.

The exported index.json contains "org.opencontainers.image.ref.name":"foo/bar:1.0.0", whereas I expect it to be "org.opencontainers.image.ref.name":"1.0.0".

The property "org.opencontainers.image.ref.name" in the index.json contains the whole image name together with the tag. The zot registry expects that only the tag should be in that property, and does not accept the generated index.json.
According to the OCI spec, this property should only hold the tag, and not the complete image name together with a tag, see https://github.com/opencontainers/image-spec/blob/main/image-layout.md#indexjson-file

[AkihiroSuda]

Is this a recent regression?

[ChristianWolf2]

I don't know. I would expect that it was always this way.

I think it should be at least possible to set "org.opencontainers.image.ref.name" to something like a tag only, e.g., "1.0.0", if the user intends it that way, e.g., by using the "name" property.
https://docs.docker.com/build/exporters/oci-docker/

[thaJeztah]

Looking at moby/moby#47232, I see that moby uses two separate annotations;

      "annotations": {
        "io.containerd.image.name": "docker.io/library/hello-world:latest",
        "org.opencontainers.image.ref.name": "latest"
      },

I wonder if those got confused for being equivalent 🤔 /cc @jedevc (ISTR you worked on OCI export)

[jedevc]

Looks like this was originally added in https://github.com/moby/buildkit/pull/615/files#diff-d77a9a4e3d44aebce1412917bb17e4cdf95b7cfcb8bebca87b68aea48eded183R31

I remember this being significant, see:

buildkit/client/ociindex/ociindex.go

Lines 128 to 132 in d6e1426

	for _, m := range idx.Manifests {
	if t, ok := m.Annotations[ocispecs.AnnotationRefName]; ok && t == tag {
	return &m, nil
	}
	}

And maybe also docker/buildx#1456.

This is the field used to do lookups in this file - at least for cache input lookups. The default field is "latest". And based on the above buildx PR, we want it to be a tag (and use the full containerd image.name for the full name).

This behavior of exporting the full name here instead of the tag definitely seems incorrect
(the cache importing/exporting and oci-layout loading looks correct). That said - I also can't see where buildkit sets this explicitly.

If someone can figure out where the wrong label comes from, would definitely be happy to take a PR for this.

[ChristianWolf2]

The wrong label comes from ParseNormalizedNamed if I am not mistaken: https://github.com/moby/containerd/blob/96c5ae04b6784e180aaeee50fba715ac448ddb0d/reference/docker/reference.go#L609,

[tianon]

https://github.com/opencontainers/image-spec/blob/5325ec48851022d6ded604199a3566254e72842a/annotations.md#:~:text=org.opencontainers.image.ref.name 👀

(org.opencontainers.image.ref.name is intended to support a "full" reference, but it is not intended to support a list of references -- in other words, both raw tag and full reference are fine, comma-separated list of either is not)

[tianon]

In other words, it's a bug in Zot to assume org.opencontainers.image.ref.name contains only the tag portion, and should be updated to accept a full reference in this annotation (per the OCI spec).

[andaaron]

Hi @tianon, per OCI dist spec the reference term represents either a tag or a digest: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests

This part of the OCI image spec mentions the use case of tags https://github.com/opencontainers/image-spec/blob/main/image-layout.md?plain=1#L153, and doesn't mention anything about the other folders in path being part of that annotation.

Can you point me to the part of the OCI specs (dist or image spec) where they define this full reference, as opposed to the reference defined in the dist spec link I mentioned above?
Does the OCI image spec define the reference as including a repository name? Anywhere.

[tianon]

The canonical reference that defines what org.opencontainers.image.ref.name means / what valid values are is https://github.com/opencontainers/image-spec/blob/v1.1.0/annotations.md#pre-defined-annotation-keys

Quote:

• org.opencontainers.image.ref.name Name of the reference for a target (string).

  • SHOULD only be considered valid when on descriptors on index.json within image layout.

  • Character set of the value SHOULD conform to alphanum of A-Za-z0-9 and separator set of -._:@/+

  • The reference must match the following grammar:

    ref       ::= component ("/" component)*
    component ::= alphanum (separator alphanum)*
    alphanum  ::= [A-Za-z0-9]+
    separator ::= [-._:@+] | "--"

(note especially that eBNF grammar which is defined explicitly and intentionally to support "full" image references here as in example.com/foo/bar:baz@sha256:xxx)

See also opencontainers/tob#114, which is a related proposal (although only tangentially related).

[andaaron]

Still, that section of the OCI spec does not define in https://github.com/opencontainers/image-spec/blob/v1.1.0/annotations.md#pre-defined-annotation-keys what the components represent.

The role of a spec is to standardize so different implementations to be compatible. It is supposed to be a contract so the two sides involved in writing to and reading from the OCI spec can build compatible software. Now it is not.

We could implement in zot picking up the tag baz out of example.com/foo/bar:baz@sha256:xxx for the API to serve it, it wouldn't be a big deal, but then someone else would want zot to pick up baz out of example.com/foo/bar:baz, another person would want zot to pick it from oci:foo/bar:baz, another person would want zot to pick it from foo/bar:baz or bar:baz, pretty much any string which matches the grammar mentioned above. And I don't see these standardized in the spec.

Thinking about cases such as:

• example.com/foo/bar:baz/sha256:xxx - the tag is not in the last component, it is in the second to last
  or
• example.com/foo/bar@sha256:xxx - the tag is missing entirely
  What is the server implementing the OCI dist spec supposed to use as a tag in these cases?

In short, anyone can come up with their own component(s) according to the grammar, and claim everyone else is not compliant with their interpretation of the spec. So I am very reluctant to implement something not explicitly mentioned in the spec.

Using the tag alone as org.opencontainers.image.ref.name is explicitly mentioned in the spec at https://github.com/opencontainers/image-spec/blob/main/image-layout.md?plain=1#L157, and it is not subject to interpretation. Why do we need to complicate every piece of software implementing the OCI spec by diverging from this?

[sudo-bmitch]

Still, that section of the OCI spec does not define in https://github.com/opencontainers/image-spec/blob/v1.1.0/annotations.md#pre-defined-annotation-keys what the components represent.

The role of a spec is to standardize so different implementations to be compatible. It is supposed to be a contract so the two sides involved in writing to and reading from the OCI spec can build compatible software. Now it is not.

Unfortunately, the concept of a reference isn't standardized, yet. So there's nothing within OCI to point to. I don't think there are enough people actively maintaining the specs to do that right now.

opencontainers/tob#114