Potential regression: v0.12.x OCI exporter creates images which cannot be loaded in Docker

[Emzi0767]

I have recently encountered a problem when building images using Docker BuildKit. When using --output=type=oci,dest=test-image.tar, the exporter creates images which I could not import on any Docker version available to me.

Here is a minimal reproducible example:

Dockerfile:

FROM alpine:3.18.3
RUN echo "This is a successful test" > /something.txt
CMD cat /something.txt

Create a buildx builder like so:
docker buildx create --use --bootstrap --name=regression --driver-opt=image=moby/buildkit:v0.12.1-rootless

Build and attempt to load the image:

docker buildx build --no-cache --provenance=false --tag=local-regression:latest --platform=linux/amd64 --pull --output=type=oci,dest=local-regression.tar .
docker image load -i local-regression.tar

Expected outcome:
The image gets loaded and is visible under docker images.

Actual outcome:

> docker image load -i local-regression.tar
open /var/lib/docker/tmp/docker-import-3510229865/blobs/json: no such file or directory

Tested building on:

• Docker Desktop version 4.22.0 (117440) under Windows 11 Pro, x64
• Docker Community version 24.0.2 (build cb74dfc) under Debian GNU/Linux 12 (Bookworm), x64

Tested image loading on:

• Docker Desktop version 4.22.0 (117440) under Windows 11 Pro, x64
• Docker Community version 24.0.2 (build cb74dfc) under Debian GNU/Linux 12 (Bookworm), x64
• Docker Community version 24.0.5 (build ced0996) under Debian GNU/Linux 11 (Bullseye), x64
• Docker Community version 24.0.5 (build ced0996) under Ubuntu GNU/Linux 22.04.2 LTS (Jammy), ARM64

Affected versions tested:

• moby/buildkit:v0.12.1-rootless
• moby/buildkit:v0.12.0-rootless

Working versions tested:

• moby/buildkit:v0.11.6-rootless

Workaround:
When creating a builder, explicitly pin the moby/buildkit image tag to version 0.11.6 or earlier, rootless variants included., for example:

> docker buildx create --use --bootstrap --name=regression --driver-opt=image=moby/buildkit:v0.11.6-rootless
... lots of output ...

> docker buildx build --no-cache --provenance=false --tag=local-regression:latest --platform=linux/amd64 --pull --output=type=oci,dest=local-regression.tar .
... lots of output ...

> docker image load -i
... lots of output ...
Loaded image ID: sha256:<some-sha>

> docker tag <the-above-sha> local-regression:latest
> docker run --rm local-regression:latest
This is a successful test

[thaJeztah]

/cc @jedevc ISTR you once had a ticket for this. I'm wondering if it previously ignored the "oci" option and automatically fell back to other formats.

Possibly related;

• --load with moby containerd store should use the oci exporter docker/buildx#1813

[jedevc]

Ha, found it. #1949, which was fixed by containerd/containerd#8213 - TL;DR the manifest.json file that docker uses to import the image is not actually valid according to the OCI spec.

The oci exporter was not designed to export images that can be loaded into docker.

If you're loading into "normal" docker, you want type=docker,oci-mediatypes=true, if you have the containerd store enabled, then the import should work using type=oci?

[Emzi0767]

I have not explicitly enabled containerd store on any of the hosts, frankly didn't know this was a feature. I was using this as part of local testing, and previously as a workaround for some provenance bugs which resulted in empty images being created. I also previously used this to create images importable by specific embedded container runtime implementations.

So tl;dr is that pre-v0.12.0 behaviour is a bug? Also what is the difference between type=oci, type=docker,oci-mediatypes=true, and plain type=docker?

[jedevc]

type=oci is for outputting OCI image layouts according to https://github.com/opencontainers/image-spec/blob/main/image-layout.md.

type=docker outputs an image to be docker loaded. It creates the manifest.json that the daemon needs to load it, and uses docker media types.

type=docker,oci-mediatypes=true outputs an image similar to the above. However, it uses the oci media types, if you so desire them - in all likelihood, if you're not explicitly sure you need them, you don't need to set this option.

[Emzi0767]

So if I want to create images that are usable by any container runtime, I should create OCI images, but use the aforementioned containerd image store feature?

[thaJeztah]

@jedevc is there an easy way to find what layout a tarball uses from its content? (thinking if we can produce a better error message for the "non-containerd-store" case)

[thaJeztah]

@Emzi0767 "docker" images are compatible with any OCI runtime, but the docker load (and docker save) commands (for backward compatibility) use a different format.

If your intent is to produce images to be used by other installs / runtimes, you can consider to use the --push to push the image to a registry (to skip the intermediary .tar export), and pull the image from other runtimes.

[Emzi0767]

Splendid.

I will close this issue now, since it appears the original problem is not, in fact, a bug.

Thank you both for your input, I really appreciate your time.