Podman pull does not prefer zstd:chunked images

[p5]

According to RHDan's forum post, the current container tools are smart enough to favour pulling zstd:chunked images from registries when in a manifest alongside gzipped images. Even if the gzipped image appears first.

Over at Universal Blue, we have started looking to push gzip and zstd:chunked alongside eachother, however are facing some strange behaviour when pulling or inspecting the images.

Our images:

• Manifest: ghcr.io/ublue-os/bluefin:40-testing
• Consists of:
  • Gzip: ghcr.io/ublue-os/bluefin@sha256:a78fe8009beeadb657bad6e76aa47e99b5b5945e618c3192b0ab4f77a06345ef
  • Zstd:chunked: ghcr.io/ublue-os/bluefin@sha256:86d32d18a0eb7329293142bb0948b03e9a3896cde004f6faef0c60c3baac9375

When we target the manifest tag for during a podman pull, we receive the gzipped image where we would have expected the zstd image.
When we target the manifest digest during a skopeo inspect, we receive information about the zstd:chunked image, which is expected.

How can we ensure that users pull the chunked image when running with podman, while keeping compatibility with older tooling which does not support zstd:chunked?

We're not quite sure what we're missing, so any assistance would be greatly appreciated.

Thank you
Rob

[vrothberg]

I think we missed mentioning that the zstd-compressed image needs to be marked with annotation in the OCI index/manifest list:

{
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
    "size": 772,
    "annotations": {
        "io.github.containers.compression.zstd": "true"
    },
    "platform": {
        "architecture": "amd64",
        "os": "linux"
    }
},

[queeup]

podman manifest push?

[p5]

podman manifest push?

Yes. I am running podman manifest create, podman manifest add --annotation ... and podman manifest push.
The annotations are not visible on the pushed manifest when running podman manifest inspect, but they are visible on the local manifest.

Version: 5.1.1

[giuseppe]

it works fine for me. I've created the manifest with the following commands:

podman manifest create quay.io/giuseppe/zstd-chunked:fedora-manifest
podman manifest add quay.io/giuseppe/zstd-chunked:fedora-manifest quay.io/giuseppe/zstd-chunked:fedora-gzip
podman manifest add --annotation io.github.containers.compression.zstd=true quay.io/giuseppe/zstd-chunked:fedora-manifest quay.io/giuseppe/zstd-chunked:fedora

where quay.io/giuseppe/zstd-chunked:fedora-gzip is a gzip compressed image, and quay.io/giuseppe/zstd-chunked:fedora is using zstd:chunked.

and then I've:

$ skopeo inspect --raw docker://quay.io/giuseppe/zstd-chunked:fedora-manifest | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:8754d19a0335a0279efeac7621f9fc52ce094763fa7ed620ef98a48be3441a2c",
      "size": 615,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:9383f2cde72625a421683a1d75fc5b1067473479c119f36791f0b203eebaa2f3",
      "size": 922,
      "annotations": {
        "io.github.containers.compression.zstd": "true"
      },
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    }
  ]
}

[p5]

The possible issue in question was with podman manifest inspect, which I would expect to show the same information as skopeo, but doesn't.

Apologies if the thread didn't get the problem across well. The manifests are created perfectly, but confusion happens when you inspect those manifests after pushing them.

➜  ~ podman manifest inspect quay.io/giuseppe/zstd-chunked:fedora-manifest
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 615,
            "digest": "sha256:8754d19a0335a0279efeac7621f9fc52ce094763fa7ed620ef98a48be3441a2c",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 922,
            "digest": "sha256:9383f2cde72625a421683a1d75fc5b1067473479c119f36791f0b203eebaa2f3",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    ]
}

[p5]

Have created an issue which hopefully explains this better
#23163

[p5]

Will close this since Podman does prefer zstd:chunked images when the required annotations are added to the manifest