"failed to reexec: Permission denied" on every rootless Podman command

[amirgol70]

I'm using Debian 13, Kernel 6.12.27-amd64 and Podman 5.4.2. Every Podman command I run on a rootless user returns "failed to reexec: Permission denied", then "Error: cannot re-exec process to join the existing user namespace" on consecutive attempts, until I delete the pause.pid, then it's back to "failed to reexec: Permission denied". This happens on any non-root user I tried.

Any ideas?

P.S. Not using SELinux and AppArmor is disabled.

containers@Server:~$ podman --log-level=debug info
INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman --log-level=debug info)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/containers/.local/share/containers/storage
DEBU[0000] Using run root /run/user/999/containers
DEBU[0000] Using static dir /home/containers/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/999/libpod/tmp
DEBU[0000] Using volume path /home/containers/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Invalid systemd user session for current user
failed to reexec: Permission denied

host:
  arch: amd64
  buildahVersion: 1.39.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon: null
  cpuUtilization:
    idlePercent: 99.83
    systemPercent: 0.1
    userPercent: 0.07
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: trixie
    distribution: debian
    version: "13"
  eventLogger: journald
  freeLocks: 2048
  hostname: Server
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.12.27-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 25935671296
  memTotal: 33391992832
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.14.0-3_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark_1.14.0-2_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.14.0
  ociRuntime: null
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20250503.587980c-2_amd64
    version: ""
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SET
PCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1.1_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.6.0
  swapFree: 32208056320
  swapTotal: 32208056320
  uptime: 22h 51m 23.00s (Approximately 0.92 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 215407505408
  graphRootUsed: 591183872
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /tmp/user/0
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.4.2
  BuildOrigin: Debian
  Built: 1748111104
  BuiltTime: Sat May 24 21:25:04 2025
  GitCommit: ""
  GoVersion: go1.24.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.2

[eriksjolund]

I see that you are logged in as the user containers

containers@Server

The username containers has a special meaning when running podman:

https://github.com/containers/podman/blob/main/troubleshooting.md#44-sudo-podman-run---usernsauto-fails-with-cannot-find-mappings-for-user-containers

Does it work from another user account?

edit

This happens on any non-root user I tried.

I see you already provided this information in the original post.

[eriksjolund]

In the output you provided:

DEBU[0000] Invalid systemd user session for current user

Maybe it could be a problem with switching users with sudo or su?

To get a proper systemd user session you could use

sudo machinectl shell --uid=myuser

[amirgol70]

This happens also when I SSH to that user (and other non-root users).

[amirgol70]

No, you're right after all. Podman doesn't work when su-ing and SSH-ing to the user, but it does when machinectl-ing! But why is that? I've been su-ing to the containers user and using Podman there for a few years now and only recently had had this problem.

[eriksjolund]

Podman doesn't work when su-ing and SSH-ing to the user

Normally it should work when SSH-ing to the user.

but it does when machinectl-ing! But why is that?

I wrote some stackexchange answers before related to this

https://unix.stackexchange.com/a/743664/9360
https://serverfault.com/a/1114780/90881

Copy-pasting parts of the second answer that I wrote in November 2022:

---

sudo -u username -i does not create a systemd login session

su is "not a tool for opening a completely new login session"
according to systemd/systemd#7451 (comment) (I assume sudo is related to su. Just running sudo to become another user will no start that user's systemd daemon)

I tried out a few examples on a Fedora 36 with systemd 250:

Example "sudo -u username -i"

[myuser@laptop ~]$ sudo useradd test1
[myuser@laptop ~]$ sudo -u test1 -i
[test1@laptop ~]$ pgrep -u $USER -l
25217 bash
[test1@laptop ~]$ env | grep XDG
XDG_DATA_DIRS=/home/test1/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
[test1@laptop ~]$ 

Example "sudo machinectl shell username@"

[myuser@laptop ~]$ sudo useradd test2
[myuser@laptop ~]$ sudo machinectl shell test2@
Connected to the local host. Press ^] three times within 1s to exit session.
[test2@laptop ~]$ pgrep -u $USER -l
25086 bash
25091 systemd
25100 (sd-pam)
25115 (sd-pam)
[test2@laptop ~]$ env | grep XDG
XDG_SESSION_TYPE=tty
XDG_SESSION_CLASS=user
XDG_SESSION_ID=51
XDG_RUNTIME_DIR=/run/user/1494
XDG_DATA_DIRS=/home/test2/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/
[test2@laptop ~]$ ls -l $XDG_RUNTIME_DIR/bus
srw-rw-rw-. 1 test2 test2 0 Nov  4 08:42 /run/user/1494/bus
[test2@laptop ~]$ 

[amirgol70]

Thanks.

[giuseppe]

@amirgol70 does the suggestion fix your problem? Can we close this discussion?