podman exec fails with Error: crun: open /dev/ptmx ...

[mwopfner]

I'm running podman on Yocto Linux (where podman is installed via the meta-virtualization layer) and in general everything works normally. After trying to open a shell in an already running container with podman exec -it -l /bin/sh I got:

Error: crun: open /dev/ptmx: No such file or directory: OCI runtime attempted to invoke a command that was not found

and if I try podman exec -l ls I get:

Error: crun: error opening file `/proc/self/setgroups`: No such file or directory: OCI runtime attempted to invoke a command that was not found

The container I want to exec the command in was started with podman run --rm -d alpine:latest sleep 6000.
On my desktop pc this works without any problems, so it must be related to the setup on the machine.
I also tried it with runc on the Yocto Linux but got the same error.

I guess there is something not setup correctly or a privilege is missing but I could not find any useful resource where to start.

podman info gives:

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: 4c41ec2a215c502ad6c38ea255ac1082fa1d381d'
  cpuUtilization:
    idlePercent: 99.65
    systemPercent: 0.06
    userPercent: 0.29
  cpus: 2
  distribution:
    distribution: poky
    version: 4.0.5
  eventLogger: journald
  hostname: qemux86-64
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.72-yocto-standard
  linkmode: dynamic
  logDriver: journald
  memFree: 1791418368
  memTotal: 2077868032
  networkBackend: cni
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 1.4.3.4-3b30-dirty
      commit: 3b3061afdeb49d691cfbbeaca09c0a0f45694cb2
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.0-beta.0+dev
      commit: unknown
      libslirp: 4.6.1.55-888d
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 2h 20m 25.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /app/containers/storage
  graphRootAllocated: 2147463168
  graphRootUsed: 11083776
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /app/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668026638
  BuiltTime: Wed Nov  9 21:43:58 2022
  GitCommit: 814b7b003cc630bf6ab188274706c383f9fb9915-dirty
  GoVersion: go1.17.13
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

[mheon]

@giuseppe PTAL

[mwopfner]

Starting the exec command with log-level trace gives:

root@qemux86-64:~# podman --log-level trace exec -l ls
INFO[0000] podman filtering at log level trace          
DEBU[0000] Called exec.PersistentPreRunE(podman --log-level trace exec -l ls) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /app/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /app/containers/storage     
DEBU[0000] Using run root /run/containers/storage       
DEBU[0000] Using static dir /app/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/libpod                    
DEBU[0000] Using volume path /app/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is being used 
DEBU[0000] Cached value indicated that native-diff is not being used 
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
TRAC[0000] found runtime "/usr/bin/crun"                
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 7              
INFO[0000] Created exec session 4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586 in container 8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50 
INFO[0000] Going to start container 8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50 exec session 4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586 and attach to it 
DEBU[0000] Set user to                                  
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50 -u 4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586 -r /usr/bin/crun -b /app/containers/storage/overlay-containers/8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50/userdata/4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586 -p /app/containers/storage/overlay-containers/8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50/userdata/4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586/exec_pid -n sweet_joliot --exit-dir /app/containers/storage/overlay-containers/8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50/userdata/4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586/exit --full-attach -s -l none --log-level trace --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/app/containers/storage/overlay-containers/8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50/userdata/4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586/oci-log -e --exec-attach --exec-process-spec /app/containers/storage/overlay-containers/8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50/userdata/4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586/exec-process-2292052398 --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /app/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg trace --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /app/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --exec --exit-command-arg 4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586 --exit-command-arg 8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50.scope 
DEBU[0000] Attaching to container 8f634ad9bde170b129934ddaad36991811e242247e90f72fcd17fce03827be50 exec session 4b97714d91e361217b0a1497109659ea302325a08200860b94983ea84d925586 
DEBU[0000] Received: 0                                  
DEBU[0000] Received: -1                                 
Error: crun: error opening file `/proc/self/setgroups`: No such file or directory: OCI runtime attempted to invoke a command that was not found

[giuseppe]

what kernel version are you using?

Why is the /proc/self/setgroups file missing?

Can you please share the content of /proc/self/mountinfo?

[mwopfner]

The Kernel is 5.15.72-yocto-standard which was build with yocto and podman was installed via the meta-virtualization layer which has a recipe for podman 4.3.1.

On the host linux I have a /proc/self/setgroups

root@qemux86-64:~# cat /proc/self/setgroups 
allow

If I start a container with podman run --rm -it alpine:latest /bin/sh I also have the setgroups file.
If I now want to run podman exec -l cat /proc/self/setgroups on this container I get

Error: crun: error opening file `/proc/self/setgroups`: No such file or directory: OCI runtime attempted to invoke a command that was not found

The content of /proc/self/mountinfo inside the container (from the first command) returns

757 723 0:55 / / rw,relatime - overlay overlay rw,lowerdir=/app/containers/storage/overlay/l/JMLHP6CBCXYIMDNHKU4JA6R24D,upperdir=/app/containers/storage/overlay/fc59390ab68613ecb104c34f6dbfcc7828b9e7482000fa83d655e0ca003a9010/diff,workdir=/app/containers/storage/overlay/fc59390ab68613ecb104c34f6dbfcc7828b9e7482000fa83d655e0ca003a9010/work,volatile
758 757 0:58 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
759 757 0:59 / /dev rw,nosuid,noexec - tmpfs tmpfs rw,size=65536k,mode=755
760 757 0:60 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs rw
761 759 0:61 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
762 759 0:57 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw
763 757 0:24 /containers/storage/overlay-containers/511a392deaf9d5e56250b027eaa793d79def38ea2265933a5c88bc63b6f56177/userdata/resolv.conf /etc/resolv.conf rw - tmpfs tmpfs rw,size=405836k,nr_inodes=819200,mode=755
764 757 0:24 /containers/storage/overlay-containers/511a392deaf9d5e56250b027eaa793d79def38ea2265933a5c88bc63b6f56177/userdata/hosts /etc/hosts rw - tmpfs tmpfs rw,size=405836k,nr_inodes=819200,mode=755
765 759 0:54 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=64000k
766 757 0:24 /containers/storage/overlay-containers/511a392deaf9d5e56250b027eaa793d79def38ea2265933a5c88bc63b6f56177/userdata/.containerenv /run/.containerenv rw - tmpfs tmpfs rw,size=405836k,nr_inodes=819200,mode=755
767 757 0:24 /containers/storage/overlay-containers/511a392deaf9d5e56250b027eaa793d79def38ea2265933a5c88bc63b6f56177/userdata/hostname /etc/hostname rw - tmpfs tmpfs rw,size=405836k,nr_inodes=819200,mode=755
768 760 0:25 / /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
769 758 0:62 / /proc/acpi ro,relatime - tmpfs tmpfs rw,size=0k
770 758 0:5 /null /proc/kcore rw,relatime - devtmpfs none rw,size=1002040k,nr_inodes=250510,mode=755
771 758 0:5 /null /proc/keys rw,relatime - devtmpfs none rw,size=1002040k,nr_inodes=250510,mode=755
772 758 0:5 /null /proc/timer_list rw,relatime - devtmpfs none rw,size=1002040k,nr_inodes=250510,mode=755
773 758 0:63 / /proc/scsi ro,relatime - tmpfs tmpfs rw,size=0k
774 760 0:64 / /sys/firmware ro,relatime - tmpfs tmpfs rw,size=0k
775 760 0:65 / /sys/dev/block ro,relatime - tmpfs tmpfs rw,size=0k
776 758 0:58 /bus /proc/bus ro,relatime - proc proc rw
777 758 0:58 /fs /proc/fs ro,relatime - proc proc rw
778 758 0:58 /irq /proc/irq ro,relatime - proc proc rw
779 758 0:58 /sys /proc/sys ro,relatime - proc proc rw
780 758 0:58 /sysrq-trigger /proc/sysrq-trigger ro,relatime - proc proc rw
724 759 0:61 /0 /dev/console rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666

It is really strange how this gets lost, because I thought exec executes it inside the running container.

[mwopfner]

And for completeness, the content of /proc/self/mountinfo on the linux host is:

root@qemux86-64:~# cat /proc/self/mountinfo 
17 24 0:16 / /proc rw,relatime shared:4 - proc proc rw
18 24 0:17 / /sys rw,relatime shared:5 - sysfs sysfs rw
19 24 0:5 / /dev rw,relatime shared:8 - devtmpfs none rw,size=1002040k,nr_inodes=250510,mode=755
20 24 8:4 / /media/rfs/ro ro,noatime,nodiratime shared:2 - ext4 /dev/sda4 ro
21 24 0:18 / /media/rfs/rw rw,noatime shared:3 - tmpfs tmpfs rw,mode=755
24 1 0:19 / / rw,relatime shared:1 - overlay overlay rw,lowerdir=/media/rfs/ro,upperdir=/media/rfs/rw/upperdir,workdir=/media/rfs/rw/work
25 19 0:22 / /dev/shm rw,nosuid,nodev shared:9 - tmpfs tmpfs rw
26 19 0:23 / /dev/pts rw,relatime shared:10 - devpts devpts rw,gid=5,mode=620,ptmxmode=666
27 24 0:24 / /run rw,nosuid,nodev shared:11 - tmpfs tmpfs rw,size=405836k,nr_inodes=819200,mode=755
28 18 0:25 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:6 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
29 18 0:26 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:7 - bpf bpf rw,mode=700
30 18 0:6 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:12 - debugfs debugfs rw
31 18 0:10 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:13 - tracefs tracefs rw
32 19 0:14 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:14 - mqueue mqueue rw
33 24 0:27 / /tmp rw,nosuid,nodev shared:15 - tmpfs tmpfs rw,size=1014584k,nr_inodes=1048576
34 18 0:28 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:16 - fusectl fusectl rw
35 18 0:29 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:17 - configfs configfs rw
55 27 0:30 / /run/credentials/systemd-sysusers.service ro,nosuid,nodev,noexec,relatime shared:18 - ramfs none rw,mode=700
77 24 0:31 / /var/volatile rw,relatime shared:38 - tmpfs tmpfs rw
80 24 8:3 / /srv/rauc rw,relatime shared:41 - ext4 /dev/sda3 rw
84 24 0:32 / /app rw,relatime shared:43 - btrfs /dev/sda7 rw,space_cache,subvolid=5,subvol=/
83 24 8:1 / /srv/barebox ro,relatime shared:45 - vfat /dev/sda1 ro,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
494 27 0:24 /netns /run/netns rw,nosuid,nodev shared:11 - tmpfs tmpfs rw,size=405836k,nr_inodes=819200,mode=755
475 84 0:32 /containers/storage/overlay /app/containers/storage/overlay rw,relatime - btrfs /dev/sda7 rw,space_cache,subvolid=5,subvol=/
496 84 0:40 / /app/containers/storage/overlay-containers/af63fc3956acd9dc36326f2832847005f13bc97ad17c89727e25fcab42ba5df2/userdata/shm rw,nosuid,nodev,noexec,relatime shared:259 - tmpfs shm rw,size=64000k
534 475 0:41 / /app/containers/storage/overlay/39536c219868abe3396c3befa1ba65cb5416f298f23648773854301b437a9c84/merged rw,nodev,relatime - overlay overlay rw,lowerdir=/app/containers/storage/overlay/l/JMLHP6CBCXYIMDNHKU4JA6R24D,upperdir=/app/containers/storage/overlay/39536c219868abe3396c3befa1ba65cb5416f298f23648773854301b437a9c84/diff,workdir=/app/containers/storage/overlay/39536c219868abe3396c3befa1ba65cb5416f298f23648773854301b437a9c84/work,volatile
535 494 0:4 net:[4026532229] /run/netns/netns-5b306a86-95cd-04c5-5830-b8bc34664d3a rw shared:268 - nsfs nsfs rw
536 27 0:4 net:[4026532229] /run/netns/netns-5b306a86-95cd-04c5-5830-b8bc34664d3a rw shared:268 - nsfs nsfs rw

[giuseppe]

it looks like procfs inside the container is behaving differently than on the host.

What other files are available for /proc/self?

[mwopfner]

I run ls -l /proc/self in the container which gives:

# ls -l /proc/self/
total 0
-r--r--r--    1 root     root             0 Mar  8 12:20 arch_status
dr-xr-xr-x    2 root     root             0 Mar  8 12:20 attr
-r--------    1 root     root             0 Mar  8 12:20 auxv
-r--r--r--    1 root     root             0 Mar  8 12:20 cgroup
--w-------    1 root     root             0 Mar  8 12:20 clear_refs
-r--r--r--    1 root     root             0 Mar  8 12:20 cmdline
-rw-r--r--    1 root     root             0 Mar  8 12:20 comm
-rw-r--r--    1 root     root             0 Mar  8 12:20 coredump_filter
-r--r--r--    1 root     root             0 Mar  8 12:20 cpuset
lrwxrwxrwx    1 root     root             0 Mar  8 12:20 cwd -> /root
-r--------    1 root     root             0 Mar  8 12:20 environ
lrwxrwxrwx    1 root     root             0 Mar  8 12:20 exe -> /bin/busybox
dr-x------    2 root     root             0 Mar  8 12:20 fd
dr-xr-xr-x    2 root     root             0 Mar  8 12:20 fdinfo
-rw-r--r--    1 root     root             0 Mar  8 12:20 gid_map
-r--r--r--    1 root     root             0 Mar  8 12:20 limits
dr-x------    2 root     root             0 Mar  8 12:20 map_files
-r--r--r--    1 root     root             0 Mar  8 12:20 maps
-rw-------    1 root     root             0 Mar  8 12:20 mem
-r--r--r--    1 root     root             0 Mar  8 12:20 mountinfo
-r--r--r--    1 root     root             0 Mar  8 12:20 mounts
-r--------    1 root     root             0 Mar  8 12:20 mountstats
dr-xr-xr-x   54 root     root             0 Mar  8 12:20 net
dr-x--x--x    2 root     root             0 Mar  8 12:20 ns
-rw-r--r--    1 root     root             0 Mar  8 12:20 oom_adj
-r--r--r--    1 root     root             0 Mar  8 12:20 oom_score
-rw-r--r--    1 root     root             0 Mar  8 12:20 oom_score_adj
-r--------    1 root     root             0 Mar  8 12:20 pagemap
-r--------    1 root     root             0 Mar  8 12:20 personality
-rw-r--r--    1 root     root             0 Mar  8 12:20 projid_map
lrwxrwxrwx    1 root     root             0 Mar  8 12:20 root -> /
-rw-r--r--    1 root     root             0 Mar  8 12:20 setgroups
-r--r--r--    1 root     root             0 Mar  8 12:20 smaps
-r--r--r--    1 root     root             0 Mar  8 12:20 smaps_rollup
-r--------    1 root     root             0 Mar  8 12:20 stack
-r--r--r--    1 root     root             0 Mar  8 12:20 stat
-r--r--r--    1 root     root             0 Mar  8 12:20 statm
-r--r--r--    1 root     root             0 Mar  8 12:20 status
-r--------    1 root     root             0 Mar  8 12:20 syscall
dr-xr-xr-x    3 root     root             0 Mar  8 12:20 task
-rw-r--r--    1 root     root             0 Mar  8 12:20 timens_offsets
-r--r--r--    1 root     root             0 Mar  8 12:20 timers
-rw-rw-rw-    1 root     root             0 Mar  8 12:20 timerslack_ns
-rw-r--r--    1 root     root             0 Mar  8 12:20 uid_map
-r--r--r--    1 root     root             0 Mar  8 12:20 wchan

[mwopfner]

Ok, I will prepare it and sent you a mail containing the link to download the disk image.

[mwopfner]

@giuseppe I just sent you the link via mail. Hope you can download and start it. If there are any problems just let me know.

[giuseppe]

thanks, I've received and looked at it.

It looks like a kernel issue to me, I can easily reproduce without using podman/crun:

# unshare -p -f --mount-proc=/proc -m sleep 100 &
[1] 2701
# nsenter -p -m -t $! ls /proc/

while on another box:

# unshare -p -f --mount-proc=/proc -m sleep 100 &
[1] 3234926
# nsenter -p -m -t $! ls /proc/
1	    devices	   ioports	  loadavg	pressure       thread-self
acpi	    diskstats	   irq		  locks		schedstat      timer_list
asound	    dma		   kallsyms	  mdstat	scsi	       tty
bootconfig  driver	   kcore	  meminfo	self	       uptime
buddyinfo   dynamic_debug  keys		  misc		slabinfo       version
bus	    execdomains    key-users	  modules	softirqs       vmallocinfo
cgroups     fb		   kmsg		  mounts	stat	       vmstat
cmdline     filesystems    kpagecgroup	  mtrr		swaps	       zoneinfo
consoles    fs		   kpagecount	  net		sys
cpuinfo     interrupts	   kpageflags	  pagetypeinfo	sysrq-trigger
crypto	    iomem	   latency_stats  partitions	sysvipc

what is weird is that the second process doesn't even see the correct root directory

[mwopfner]

Thanks for looking into it. So the Linux namespaces are broken in some way.
Now, at least I know where to look. 👍

[mwopfner]

I had some more time to investigate this problem and it is related to how I have setup the rootfs.
To get a readonly rootfs I have (the script I used) mounted an overlayfs over the complete rootfs and then move the /sys , /proc, /dev mounts into the new overlay rootfs. Finally I chroot into the new rootfs.
If I deactivate this overlayfs everything works as expected.

[mwopfner]

After some digging with the help of @giuseppe (thanks a lot) I could solve the problem.

Problem:
The problem was NOT podman or some Kernel configs. The real problem was that I have used an intitramfs with an init script that created a tmpfs overlay over the rootfs which used chroot to switch to the newly setuped rootfs. Unfortunately chroot does not work properly with linux mount namespaces. Thus the later errors.

Solution:
I just had to replace chroot with switch_root and everything worked as expected.

Some background info can be found here:

• https://linuxlink.timesys.com/docs/classic/change_root
• https://man7.org/linux/man-pages/man8/switch_root.8.html
• https://home.robusta.dev/blog/containers-dont-use-chroot
• https://web.archive.org/web/20160730094856/http://wiki.sourcemage.org/HowTo(2f)Initramfs.html