Exec error for multi-platform builds of feature layers

[MartinLoeper]

I already reported the issue in the ci repository, but recognized that it is cli issue, see: devcontainers/ci#305

I just enabled the multi-platform build feature on my machine which runs on x86. I encountered an exec format error during a feature layer build for arm64 as follows:

Start: Run: docker buildx build --platform linux/amd64,linux/arm64 --push --build-context dev_containers_feature_content_source=/tmp/devcontainercli-mloeper/container-features/0.72.0-1737468359115 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/typescript-node:1-22-bookworm --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -f /tmp/devcontainercli-mloeper/container-features/0.72.0-1737468359115/Dockerfile.extended -t mloeper/test /tmp/devcontainercli-mloeper/empty-folder
[+] Building 32.2s (23/23) FINISHED          docker-container:container-builder
<skipped>
 => CACHED [linux/amd64 dev_containers_feature_content_normalize 1/3] FRO  0.0s
 => => resolve mcr.microsoft.com/devcontainers/typescript-node:1-22-bookw  0.0s
 => [linux/amd64 dev_containers_target_stage 2/5] RUN mkdir -p /tmp/dev-c  0.2s
 => [linux/amd64 dev_containers_feature_content_normalize 2/3] COPY --fro  0.1s
 => [linux/amd64 dev_containers_feature_content_normalize 3/3] RUN chmod   0.1s
 => [linux/amd64 dev_containers_target_stage 3/5] COPY --from=dev_contain  0.1s
 => [linux/amd64 dev_containers_target_stage 4/5] RUN echo "_CONTAINER_US  0.1s
 => CANCELED [linux/amd64 dev_containers_target_stage 5/5] RUN --mount=t  29.5s
 => [linux/arm64 dev_containers_target_stage 2/5] RUN mkdir -p /tmp/dev-c  0.3s
 => [linux/arm64 dev_containers_feature_content_normalize 2/3] COPY --fro  0.3s
 => [linux/arm64 dev_containers_feature_content_normalize 3/3] RUN chmod   0.1s
 => [linux/arm64 dev_containers_target_stage 3/5] COPY --from=dev_contain  0.0s
 => [linux/arm64 dev_containers_target_stage 4/5] RUN echo "_CONTAINER_US  0.1s
 => ERROR [linux/arm64 dev_containers_target_stage 5/5] RUN --mount=type=  0.2s
------
 > [linux/arm64 dev_containers_target_stage 5/5] RUN --mount=type=bind,from=dev_containers_feature_content_source,source=cosign_0,target=/tmp/build-features-src/cosign_0     cp -ar /tmp/build-features-src/cosign_0 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/cosign_0  && cd /tmp/dev-container-features/cosign_0  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/cosign_0:
0.139 ===========================================================================
0.139 Feature       : Cosign (via Github Releases)
0.139 Description   : Cosign is a command line utility that can sign and verify software artifact, such as container images and blobs.
0.139 Id            : ghcr.io/devcontainers-extra/features/cosign
0.139 Version       : 1.0.9
0.139 Documentation : http://github.com/devcontainers-contrib/features/tree/main/src/cosign
0.139 Options       :
0.139     VERSION="latest"
0.139 ===========================================================================
0.155 Error while loading /tmp/dev-container-features/cosign_0/./install.sh: Exec format error
0.156 ERROR: Feature "Cosign (via Github Releases)" (ghcr.io/devcontainers-extra/features/cosign) failed to install! Look at the documentation at http://github.com/devcontainers-contrib/features/tree/main/src/cosign for help troubleshooting this error.
------
Dockerfile.extended:24
--------------------
  23 |
  24 | >>> RUN --mount=type=bind,from=dev_containers_feature_content_source,source=cosign_0,target=/tmp/build-features-src/cosign_0 \
  25 | >>>     cp -ar /tmp/build-features-src/cosign_0 /tmp/dev-container-features \
  26 | >>>  && chmod -R 0755 /tmp/dev-container-features/cosign_0 \
  27 | >>>  && cd /tmp/dev-container-features/cosign_0 \
  28 | >>>  && chmod +x ./devcontainer-features-install.sh \
  29 | >>>  && ./devcontainer-features-install.sh \
  30 | >>>  && rm -rf /tmp/dev-container-features/cosign_0
  31 |
--------------------
ERROR: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c cp -ar /tmp/build-features-src/cosign_0 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/cosign_0  && cd /tmp/dev-container-features/cosign_0  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/cosign_0" did not complete successfully: exit code: 1
{"outcome":"error","message":"Command failed: docker buildx build --platform linux/amd64,linux/arm64 --push --build-context dev_containers_feature_content_source=/tmp/devcontainercli-mloeper/container-features/0.72.0-1737468359115 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/typescript-node:1-22-bookworm --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -f /tmp/devcontainercli-mloeper/container-features/0.72.0-1737468359115/Dockerfile.extended -t mloeper/test /tmp/devcontainercli-mloeper/empty-folder","description":"An error occurred building the container."}

Minimal Reproducible Example

build.sh:

devcontainer build --platform linux/amd64,linux/arm64 --workspace-folder . --push --image-name "yourrepo/test"

.devcontainer/devcontainer.json:

{
	"name": "MRE",
	"image": "mcr.microsoft.com/devcontainers/typescript-node:1-22-bookworm",
	"features": {
		"ghcr.io/devcontainers-extra/features/cosign:1": {}
	}
}

[MartinLoeper]

❯ docker info
Client:
 Version:    27.4.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /nix/store/f91j9299d003r86zyh7fyar93818a67g-docker-plugins/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.32.1
    Path:     /nix/store/f91j9299d003r86zyh7fyar93818a67g-docker-plugins/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 89
  Running: 15
  Paused: 0
  Stopped: 74
 Images: 246
 Server Version: 27.4.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: refs/tags/v2.0.1
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.69
 Operating System: NixOS 25.05 (Warbler)
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 62.56GiB
 Name: nesto-office-mloeper
 ID: 431195dc-76a1-4216-947c-07d36480c180
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

[MartinLoeper]

I also ran the following before executing the build: docker buildx create --use --platform linux/amd64,linux/arm64

[Kaniska244]

Hi @MartinLoeper ,

Thanks for the reproducible example, I will check & update you on the proceedings.

With Regards,
Kaniska

[Kaniska244]

Hi @MartinLoeper ,

One small question regarding the reproducible example, did u setup qemu or any similar tool to emulate the arm64 CPU architecture as you created the docker builder instance for both amd64(x86_64) & arm64 platforms while running the build on x86 architecture based system.

With Regards,
Kaniska

[MartinLoeper]

Hi @Kaniska244, you are MVP!! That was indeed the case! Since I build on x86 NixOS, I have to add a special option to my config.

For everybody as a reference: boot.binfmt.emulatedSystems

Example to cross-build arm64:

boot.binfmt.emulatedSystems = [
  "aarch64-linux"
];

Thanks so much!