podman run fails inside a container with Error: crun: creating cgroup directory "/sys/fs/cgroup/devices/libpod_parent/***"

[tchowice]

Issue Description

I have a Debian 12.5 container with podman 4.3.1 installed. The host is a Redhat 8.5 running podman 4.6.1.

Within the container, when I execute podman run, I get the following error:

Error: crun: creating cgroup directory /sys/fs/cgroup/devices/libpod_parent/libpod-a0442b6cb250349aaca3435df7ef3ca0dfca4f88e885ed68575a0380554cc90d: No such file or directory: OCI runtime attempted to invoke a command that was not found

Steps to reproduce the issue

Steps to reproduce the issue

1. Start container on host with the following arguments: --sysctl net.ipv4.conf.default.route_localnet=1 --security-opt seccomp=unconfined --security-opt label=disable --security-opt=unmask=/proc/* --security-opt=unmask=/sys/fs/cgroup --systemd=always --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --cap-add NET_ADMIN --cap-add SYS_CHROOT --device /dev/fuse --mount=type=tmpfs,destination=/var/lib/containers --user=root -v /usr/lib/systemd/system/podman.socket:/usr/lib/systemd/system/podman.socket -v /run/systemd/journal/socket:/run/systemd/journal/socket
2. Within the container, execute podman run --rm -d --name test_container --hostname test_container docker.io/redis:alpine

Describe the results you received

This is the output:

Trying to pull docker.io/library/redis:alpine...
Getting image source signatures
Copying blob sha256:f76326fd8e6b93c5ddd86c0795b0a04c186faf08ce032c102aa9b3671276019a
Copying blob sha256:5c3180d102093de53ebc54b965de6754cbbb344a30e2bf2f5d17cbc3ac1d50b5
Copying blob sha256:034c076ba1e793bb3f31ae4f06a1737a362a34ce9e843c6c15022cfbec955bb9
Copying blob sha256:dffcad17539bc6497d8dd4bd24f6628013eb413b988050010925cb2ce4382291
Copying blob sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
Copying blob sha256:5913474e0f39b23ca3d952a08c0008364c774a07984efaf8ad3a5ba8e04d31f6
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:cc6fccbbefa3f40f40b56bb79e12ea0d1712b5b2421e04b1e0086c222e43da52
Copying config sha256:435993df2c8d3a1508114cea2dd12ef4d6cbab5c7238bb8e587f20b18982c834
Writing manifest to image destination
Storing signatures
failed to connect to syslog: Initialization
time="2024-04-03T21:44:59-07:00" level=warning msg="Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup for pids: mkdir /sys/fs/cgroup/pids/libpod_parent: read-only file system"
Error: crun: creating cgroup directory `/sys/fs/cgroup/devices/libpod_parent/libpod-a0442b6cb250349aaca3435df7ef3ca0dfca4f88e885ed68575a0380554cc90d`: No such file or directory: OCI runtime attempted to invoke a command that was not found

Describe the results you expected

The redis container should successfully launch on the host.

podman info output

Within the container:

host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 99.64
    systemPercent: 0.1
    userPercent: 0.26
  cpus: 4
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: file
  hostname: 695ee1fe277f
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-348.20.1.el8_5.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1551605760
  memTotal: 6232244224
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 8355713024
  swapTotal: 8485072896
  uptime: 2537h 54m 49.00s (Approximately 105.71 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs_1.10-1_amd64
      Version: |-
        fusermount3 version: 3.14.0
        fuse-overlayfs: version 1.10
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 3116122112
  graphRootUsed: 32768
  graphStatus:
    Backing Filesystem: tmpfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

On the host:

host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 21e9be52d14128789284e1bbf54d8c25b4820215'
  cpuUtilization:
    idlePercent: 99.64
    systemPercent: 0.1
    userPercent: 0.26
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.5"
  eventLogger: file
  freeLocks: 2046
  hostname: eq1vjenbldl3001
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1302400513
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000854465
    uidmap:
    - container_id: 0
      host_id: 1302439623
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000854465
  kernel: 4.18.0-348.20.1.el8_5.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3952402432
  memTotal: 6232244224
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.1.9
      spec: 1.0.2-dev
      go: go1.20.6
      libseccomp: 2.5.1
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/1302439623/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 8355713024
  swapTotal: 8485072896
  uptime: 2550h 59m 57.00s (Approximately 106.25 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/svc_jenkins/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/svc_jenkins/.local/share/containers/storage
  graphRootAllocated: 123469824000
  graphRootUsed: 4865560576
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 11
  runRoot: /run/user/1302439623/containers
  transientStore: false
  volumePath: /home/svc_jenkins/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1696868155
  BuiltTime: Mon Oct  9 09:15:55 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1
+ docker info
host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 21e9be52d14128789284e1bbf54d8c25b4820215'
  cpuUtilization:
    idlePercent: 99.64
    systemPercent: 0.1
    userPercent: 0.26
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.5"
  eventLogger: file
  freeLocks: 2046
  hostname: eq1vjenbldl3001
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1302400513
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000854465
    uidmap:
    - container_id: 0
      host_id: 1302439623
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000854465
  kernel: 4.18.0-348.20.1.el8_5.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3952402432
  memTotal: 6232244224
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.1.9
      spec: 1.0.2-dev
      go: go1.20.6
      libseccomp: 2.5.1
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/1302439623/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 8355713024
  swapTotal: 8485072896
  uptime: 2550h 59m 57.00s (Approximately 106.25 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/svc_jenkins/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/svc_jenkins/.local/share/containers/storage
  graphRootAllocated: 123469824000
  graphRootUsed: 4865560576
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 11
  runRoot: /run/user/1302439623/containers
  transientStore: false
  volumePath: /home/svc_jenkins/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1696868155
  BuiltTime: Mon Oct  9 09:15:55 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[tchowice]

Anyone?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[giuseppe]

sorry for the late reply.

There is no safe delegation for cgroup v1 and you are already running with privileges, so you could just bind mount /sys/fs/cgroup from the host and achieve the same level of safety: -v /sys/fs/cgroup:/sys/fs/cgroup. Does that make any difference?

[bijianpeng]

I have also encountered the same problem, I would like to know if this issue has been resolved?