Podman Quadlet /sys/fs/cgroup permission denied

[vpetrigo]

I followed this Red Hat guide on how to use Quadlet to improve systemd container management and met the following issue:

• when running this myservice.service via systemctl I got the following error:

Oct 10 15:00:14 localhost.localdomain myservice[24989]: Pulling image registry.access.redhat.com/ubi9-minimal:latest inside systemd: setting pull timeout to 5m0s            
Oct 10 15:00:23 localhost.localdomain myservice[24989]: Getting image source signatures
Oct 10 15:00:23 localhost.localdomain myservice[24989]: Checking if image destination supports signatures
Oct 10 15:00:23 localhost.localdomain myservice[24989]: Copying blob sha256:516f7562f02267e05bc0f8b175363c70768471b977360d0f0ab5711a8a6d25ab
Oct 10 15:00:30 localhost.localdomain myservice[24989]: Copying config sha256:f69b56cb053f31dbd9c7bf0380c6e607b198a2597436c557904ad94de0817d39
Oct 10 15:00:30 localhost.localdomain myservice[24989]: Writing manifest to image destination
Oct 10 15:00:30 localhost.localdomain myservice[24989]: Storing signatures
Oct 10 15:00:30 localhost.localdomain podman[24989]: 2023-10-10 15:00:30.410660709 +0300 MSK m=+15.657647765 container create ebd60dd4a8afabd872d444f19d2e26fca971ef7857982f>
Oct 10 15:00:30 localhost.localdomain podman[24989]: 2023-10-10 15:00:30.428341105 +0300 MSK m=+15.675328160 container remove ebd60dd4a8afabd872d444f19d2e26fca971ef7857982f>
Oct 10 15:00:30 localhost.localdomain podman[24989]: 2023-10-10 15:00:30.400951074 +0300 MSK m=+15.647938119 image pull f69b56cb053f31dbd9c7bf0380c6e607b198a2597436c557904a>
Oct 10 15:00:30 localhost.localdomain myservice[24989]: Error: mkdir /sys/fs/cgroup/memory/user.slice/user-1000.slice/[email protected]/runtime: permission denied
Oct 10 15:00:30 localhost.localdomain systemd[989]: myservice.service: Main process exited, code=exited, status=126/n/a
Oct 10 15:00:30 localhost.localdomain systemd[989]: myservice.service: Killing process 25126 (slirp4netns) with signal SIGKILL.
Oct 10 15:00:30 localhost.localdomain systemd[989]: myservice.service: Failed with result 'exit-code'.
Oct 10 15:00:30 localhost.localdomain systemd[989]: Failed to start myservice.service - The sleep container.

The content of the myservice.container:

[Unit]
Description=The sleep container
After=local-fs.target

[Container]
Image=registry.access.redhat.com/ubi9-minimal:latest
Exec=sleep 1000

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

Qualet service output:

---myservice.service---                                                                                                                                                      
[Unit]                                                                                                                                                                       
Description=The sleep container                                                                                                                                              
After=local-fs.target                                                                                                                                                        
SourcePath=/home/vpetrigo/.config/containers/systemd/myservice.container                                                                                                     
RequiresMountsFor=%t/containers                                                                                                                                              
                                                                                                                                                                             
[X-Container]                       
Image=registry.access.redhat.com/ubi9-minimal:latest                                  
Exec=sleep 1000                                                                       
                                           
[Install]                                                                             
# Start by default on boot                                                            
WantedBy=multi-user.target default.target                                             
                                                                                                                                                                             
[Service]                                                                                                                                                                    
Environment=PODMAN_SYSTEMD_UNIT=%n                                                    
KillMode=mixed                                                                        
ExecStop=/usr/bin/podman rm -f -i --cidfile=%t/%N.cid                                 
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid                            
Delegate=yes                                                                          
Type=notify                                                                                                                                                                  
NotifyAccess=all                                                                                                                                                             
SyslogIdentifier=%N                                                                                                                                                          ExecStart=/usr/bin/podman run --name=systemd-%N --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d registry.access.redhat.com/ubi9-minimal:latest sleep 
1000

container_manage_cgroup is set to on:

$ getsebool container_manage_cgroup                   
container_manage_cgroup --> on

What helps is adding the following line into the myservice.container:

PodmanArgs=--cgroups=disabled
# OR
PodmanArgs=--cgroups=enabled

My system info:

$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="38 (Workstation Edition)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f38/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Podman version:

$ podman --version
podman version 4.7.0

Podman info output:

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 92.58
    systemPercent: 2.2
    userPercent: 5.22
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "38"
  eventLogger: journald
  freeLocks: 2046
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.5.6-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 208687104
  memTotal: 8273944576
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.9.2-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.9.2
      commit: 35274d346d2e9ffeacb22cc11590b0266a23d634
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.x86_64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 12521562112
  swapTotal: 12522086400
  uptime: 2h 5m 16.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/vpetrigo/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/vpetrigo/.local/share/containers/storage
  graphRootAllocated: 100109275136
  graphRootUsed: 78713954304
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 62
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/vpetrigo/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.0
  Built: 1695839078
  BuiltTime: Wed Sep 27 21:24:38 2023
  GitCommit: ""
  GoVersion: go1.20.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0

[rhatdan]

@mheon @giuseppe PTAL

[vrothberg]

Quadlet doesn't run on cgroups v1 AFAIR but we should make sure it's documented (or/and a proper error being thrown).

[vpetrigo]

Seems like I found out why this happened:

• I got cgroup v1 fallback enabled as described here with this command:

grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

After I removed that with the following line everything works as expected:

grubby --update-kernel=ALL --remove-args "systemd.unified_cgroup_hierarchy=0"