Multiarch emulation not working with AlmaLinux 10 & Fedora 42 for rootless containers

[der-eismann]

Issue Description

Hey everyone, we've been using the docker.io/tonistiigi/binfmt image for years on our CentOS Stream 9 machines with kernel 5.14.0 & podman 5.6.0 to setup multiarch emulation which has worked flawlessly so far.

Now we tried to upgrade to AlmaLinux 10 with kernel 6.12.0 & podman 5.4.0 and unfortunately this solution has stopped working. I could replicate the same on my personal Fedora 42 machine with kernel 6.16.3 & podman 5.6.0.

Install looks successful, but emulators are empty when checking:

$ podman run --privileged --rm tonistiigi/binfmt --install arm64
installing: arm64 OK
{
  "supported": [
    "linux/amd64",
    "linux/amd64/v2",
    "linux/amd64/v3",
    "linux/amd64/v4",
    "linux/arm64",
    "linux/386"
  ],
  "emulators": [
    "qemu-aarch64"
  ]
}

$ podman run --privileged --rm tonistiigi/binfmt                
{
  "supported": [
    "linux/amd64",
    "linux/amd64/v2",
    "linux/amd64/v3",
    "linux/amd64/v4",
    "linux/386"
  ],
  "emulators": null
}

$ ls -la /proc/sys/fs/binfmt_misc/                              
insgesamt 0
drwxr-xr-x. 2 root root 0  2. Sep 16:33 .
dr-xr-xr-x. 1 root root 0  1. Sep 09:09 ..
--w-------. 1 root root 0  1. Sep 09:10 register
-rw-r--r--. 1 root root 0  1. Sep 09:10 status

However when executed as sudo things start to work again, but it stays broken without sudo:

$ sudo podman run --privileged --rm tonistiigi/binfmt --install arm64
installing: arm64 OK
{
  "supported": [
    "linux/amd64",
    "linux/amd64/v2",
    "linux/amd64/v3",
    "linux/amd64/v4",
    "linux/arm64",
    "linux/386"
  ],
  "emulators": [
    "qemu-aarch64"
  ]
}

$ sudo podman run --privileged --rm tonistiigi/binfmt                
{
  "supported": [
    "linux/amd64",
    "linux/amd64/v2",
    "linux/amd64/v3",
    "linux/amd64/v4",
    "linux/arm64",
    "linux/386"
  ],
  "emulators": [
    "qemu-aarch64"
  ]
}

$ ls -la /proc/sys/fs/binfmt_misc/
insgesamt 0
drwxr-xr-x. 2 root root 0  2. Sep 16:33 .
dr-xr-xr-x. 1 root root 0  1. Sep 09:09 ..
-rw-r--r--. 1 root root 0  2. Sep 16:34 qemu-aarch64
--w-------. 1 root root 0  1. Sep 09:10 register
-rw-r--r--. 1 root root 0  1. Sep 09:10 status

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 
enabled
interpreter /usr/bin/qemu-aarch64
flags: POCF
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

Now I really wonder why this has stopped working, is anybody aware of a kernel change or anything in that regard? We really wanna stick with unprivileged containers. Interestingly checking the status without root didn't work in the past, but does now:

[CentOS Stream 9]
$ podman run --privileged --rm tonistiigi/binfmt
error: operation not permitted
cannot mount binfmt_misc filesystem at /proc/sys/fs/binfmt_misc
main.run
        /src/cmd/binfmt/main.go:204
main.main
        /src/cmd/binfmt/main.go:191
runtime.main
        /usr/local/go/src/runtime/proc.go:272
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1700

[AlmaLinux 10]
$ podman run --privileged --rm tonistiigi/binfmt 
{
  "supported": [
    "linux/amd64",
    "linux/amd64/v2",
    "linux/amd64/v3",
    "linux/amd64/v4",
    "linux/386"
  ],
  "emulators": null
}

I know that this is probably not a podman issue, but I'm thankful for any hint or help I can get!

Steps to reproduce the issue

Steps to reproduce the issue

1. sudo podman run --privileged --rm tonistiigi/binfmt --install arm64
2. check with podman run --privileged --rm tonistiigi/binfmt
3. check with podman run --rm --platform linux/arm64 alpine uname -m - will fail on both Fedora 42 & AlmaLinux 10 with

{"msg":"exec container process `/bin/uname`: Exec format error","level":"error","time":"2025-09-02T15:15:40.446015Z"}

Describe the results you received

Emulation simply doesn't work

Describe the results you expected

Emulation should keep working like it was in the past

podman info output

host:
  arch: amd64
  buildahVersion: 1.41.3
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 81.13
    systemPercent: 1.88
    userPercent: 16.99
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: kde
    version: "42"
  emulatedArchitectures:
  - linux/arm
  - linux/arm64
  - linux/arm64be
  - linux/mips
  - linux/mips
  - linux/mips64
  - linux/ppc
  - linux/ppc64
  - linux/ppc64le
  - linux/riscv32
  - linux/riscv64
  - linux/s390x
  eventLogger: journald
  freeLocks: 2044
  hostname: eduard-laser
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.16.3-200.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2161643520
  memTotal: 32731648000
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250805.g309eefd-2.fc42.x86_64
    version: |
      pasta 0^20250805.g309eefd-2.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 12793278464
  swapTotal: 42949664768
  uptime: 32h 6m 15.00s (Approximately 1.33 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/sysop/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 1
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/sysop/.local/share/containers/storage
  graphRootAllocated: 940422725632
  graphRootUsed: 589281284096
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 498
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/sysop/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.0
  BuildOrigin: Fedora Project
  Built: 1755216000
  BuiltTime: Fri Aug 15 02:00:00 2025
  GitCommit: da671ef6cfa3fc9ac6225c18f1dd0a70a951e43f
  GoVersion: go1.24.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

SELinux is disabled or permissive, happens on AWS and locally on my machine.

Additional information

No response

[Luap99]

I cannot see how installing emulator was ever allowed as rootless. Permission wise this seems extremely dangerous if it were allowed so the restriction that this only work when run as root seem expected to me.

Adding an emulator there is done by writing to /proc/sys/fs/binfmt_misc/register and if I check the permission on my system it is write only for root, --w-------. 1 root root 0 Sep 2 11:07 /proc/sys/fs/binfmt_misc/register

I am not sure if the binfmt_misc is mount namespace aware and from the error it seems the container is trying to mount its own binfmt_misc mount. And if I try that locally it seems to allow me to do that in a rootless container so I guess it must be namespace aware now?

A google search seem to indicate that this was added in 6.7 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d82c0a37d431ada0d1dae9a2665fcfe17b0f9e14

[der-eismann]

Hey @Luap99, thanks for the quick reply! I forgot to add that on CentOS Stream 9 I had to register the emulator with sudo as well, but it worked without sudo afterwards.

[Luap99]

yeah this is normal kernel behavior as rootless podman runs in its own mounts and then the runtime mounts a fresh /proc

compare

$ unshare -r --mount --pid --mount-proc --fork ls /proc/sys/fs/binfmt_misc/

$ unshare -r --mount --pid --fork ls /proc/sys/fs/binfmt_misc/
qemu-aarch64	 qemu-arm ...

AFAICT you can still use the emulators from the root namespace so it should still work just the listing might be off.
But when you register the emulator rootless it will only be in its userns and thus not visible to the other container I guess.

[der-eismann]

That's an interesting point! As I mentioned in the comment above I tried to register the emulators as root as well and I can see them as root, but not as non-root.
I can actually mount /proc manually and then it shows all registered emulators correctly, but unfortunately emulating still doesn't work.

$ podman run --privileged --rm -v /proc:/proc tonistiigi/binfmt        
{
  "supported": [
    "linux/amd64",
    "linux/amd64/v2",
    "linux/amd64/v3",
    "linux/amd64/v4",
    "linux/386"
  ],
  "emulators": [
    "qemu-aarch64"
  ]
}

$ podman run --privileged --rm tonistiigi/binfmt      
{
  "supported": [
    "linux/amd64",
    "linux/amd64/v2",
    "linux/amd64/v3",
    "linux/amd64/v4",
    "linux/386"
  ],
  "emulators": null
}

$ podman run --rm --platform linux/arm64 -v /proc:/proc alpine uname -m
{"msg":"exec container process `/bin/uname`: Exec format error","level":"error","time":"2025-09-02T17:21:04.488394Z"}

$ podman run --rm --platform linux/arm64 alpine uname -m      
{"msg":"exec container process `/bin/uname`: Exec format error","level":"error","time":"2025-09-02T17:21:11.427606Z"}

So yeah, the namespace seems to be the culprit of everything. Now I just need to find out how to get it working without root again :/