Is that possible to set resource limit in rootless mode when using cgroupfs? #19158

m0n203 · 2023-07-07T09:56:17Z

m0n203
Jul 7, 2023

English is not my first language. If I came across as rude, that was not my intention.

basic infomation:

key	value
cgroup manager	cgroupfs
cgroup version	v2
podman version	v4.5.0 on gentoo (openrc)
^	v4.2.2 on ubuntu jammy (systemd)

I have a gentoo box, which have podman installed.
I am trying to set memory limit to a container like this:

$ podman run --security-opt=proc-opts=hidepid=2 -it --rm -m 10m leap

$ pstree -aTp 9401
podman,9401 run --security-opt=proc-opts=hidepid=2 -it --rm -m 10m leap
  └─slirp4netns,9451 --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path/run/user/1000/netns/

$ podman stats -l
ID            NAME            CPU %       MEM USAGE / LIMIT  MEM %       NET IO       BLOCK IO           PIDS        CPU TIME           AVG CPU %
d1a8e9094e0f  relaxed_swartz  422.56%     2.931GB / 10.49MB  27953.87%   110B / 430B  836.5GB / 262.8GB  2835        113h29m44.142349s  399225.64%

No errors, but the mem usage is much larger than the limit, and stress-ng can use more than 10M memory.

I have checked the current cgroup version is v2, and these process is in the default user's cgroup (/sys/fs/cgroup/2), which is no limit set on.

$ mount | grep cgroup
none on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)

$ pstree -aTp 9401 | grep -oP '(?<=,)\d+' | xargs -i cat /proc/{}/cgroup
0::/2
0::/2

$ grep '' /sys/fs/cgroup/2/memory.{max,min,high,low}
/sys/fs/cgroup/2/memory.max:max
/sys/fs/cgroup/2/memory.min:0
/sys/fs/cgroup/2/memory.high:max
/sys/fs/cgroup/2/memory.low:0

And then I change the log level to info, it's seems the current user doesn't have the permission to create a new cgroup.

$ podman --log-level info run --security-opt=proc-opts=hidepid=2 -m 10m -it --rm leap
INFO[0000] podman filtering at log level info           
INFO[0000] Setting parallel job count to 145            
INFO[0000] Received shutdown.Stop(), terminating!        PID=1307582
INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path conmon: open /sys/fs/cgroup/cgroup.subtree_control: permission denied 
INFO[0000] Got Conmon PID as 1307613                    
e9ed0df57c6d:/ #

I create a new cgroup manually, chown it to the current user, and using --cgroup-parent to specify that path, but now i can not even run the contianer.

$ podman --log-level info run --security-opt=proc-opts=hidepid=2 -m 10m --cgroup-parent podman -it --rm leap                                    
INFO[0000] podman filtering at log level info                                                                                                                
INFO[0000] Setting parallel job count to 145                                                                                                                 
INFO[0000] Received shutdown.Stop(), terminating!        PID=1307703                                                                                         
INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path podman/conmon: open /sys/fs/cgroup/cgroup.subtree_control: permission denied
                                                                                                                                                             
Error: crun: writing file `/sys/fs/cgroup/podman/libpod-47f129fa2782afbce364b553255edea1de9cfa9dfff45dc5f2e5963220f9dfaa/cgroup.procs`: Permission denied: OC
I permission denied 
$ stat /sys/fs/cgroup/podman/libpod-47f129fa2782afbce364b553255edea1de9cfa9dfff45dc5f2e5963220f9dfaa/cgroup.procs
  File: /sys/fs/cgroup/podman/libpod-47f129fa2782afbce364b553255edea1de9cfa9dfff45dc5f2e5963220f9dfaa/cgroup.procs
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 1dh/29d Inode: 199545      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/      op)   Gid: ( 1000/      op)

Also, I tried to grant write permission to the current user for /sys/fs/cgroup/cgroup.subtree_control, and create /sys/fs/cgroup/conmon manually, now it's run w/o errors, but still, the memory limit doesn't work.

$ podman --log-level info run --security-opt=proc-opts=hidepid=2 -m 10m -it --rm leap                                                           
INFO[0000] podman filtering at log level info                                                                                                                
INFO[0000] Setting parallel job count to 145                                                                                                                 
INFO[0000] Received shutdown.Stop(), terminating!        PID=1309109                                                                                         
INFO[0000] Got Conmon PID as 1309140                                                                                                                                                                       
60a8ef30e12f:/ # cat /sys/fs/cgroup/memory.max                                                                                                               
max                                                                                                                                                          
60a8ef30e12f:/

I tried run this w/ root user on the same gentoo box, it's works.

$ sudo podman run --security-opt=proc-opts=hidepid=2 -it --rm -m 10m leap
2cffea05c26a:/ # cat /sys/fs/cgroup/memory.
memory.current        memory.high           memory.min            memory.peak           memory.stat           memory.swap.high      memory.zswap.max
memory.events         memory.low            memory.numa_stat      memory.pressure       memory.swap.current   memory.swap.max       
memory.events.local   memory.max            memory.oom.group      memory.reclaim        memory.swap.events    memory.zswap.current  
2cffea05c26a:/ # cat /sys/fs/cgroup/memory.max 
10485760
2cffea05c26a:/ #

And I run the same command with --cgroup-manager systemd on ubuntu, it's also works.

podman --log-level info --cgroup-manager systemd run --security-opt=proc-opts=hidepid=2 -m 10M -it --rm leap
INFO[0000] podman filtering at log level info           
INFO[0000] Setting parallel job count to 145            
INFO[0000] Received shutdown.Stop(), terminating!        PID=1309684
INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-819a5b817903a0d2ffddae06dd396559f8e76eaba32675cf77b362fcef187c04.scope 
INFO[0000] Got Conmon PID as 1309721                    
819a5b817903:/ # cat /sys/fs/cgroup/memory.max 
10485760
819a5b817903:/ #

Am I doing it right? Does that need extra setup or something else?
Is that mean set resource limit in rootless mode when using cgroupfs is not supported?

rhatdan · 2023-07-10T20:13:44Z

rhatdan
Jul 10, 2023
Maintainer

@giuseppe PTAL

0 replies

giuseppe · 2023-07-10T20:40:35Z

giuseppe
Jul 10, 2023
Maintainer

You need to recreate the cgroup, chown it and force podman to use it with --cgroup-parent

1 reply

m0n203 Jul 19, 2023
Author

Should i mount a new cgroupfs and then chown the new root cgroup?
I have create a new cgroup subdirectory inside the current /sys/fs/cgroup(mkdir /sys/fs/cgroup/demo),
and chown it (chown -R uid:gid /path/to/new_cgroup/),
then checked the directory to make sure all the files belongs to my current user, but still not work.

Error: crun: writing file `/sys/fs/cgroup/demo/libpod-3e94bda2312a1e8c70432031952238c06a30d4f2fa31e46887e1c800e7c52282/cgroup.procs`: Permission denied: OCI permission denied

I also tried to write pid to this file, got denied:

$ echo $$ > /sys/fs/cgroup/demo/libpod-3e94bda2312a1e8c70432031952238c06a30d4f2fa31e46887e1c800e7c52282/cgroup.procs 
echo: write error: permission denied

ikke-t · 2023-10-10T04:07:17Z

ikke-t
Oct 10, 2023

I also experience this. My environment is fresh Ansible EDA 2.4.2 install on RHEL 9.2. Somehow mysteriously the containers won't get started for EDA service as eda user. When I start them by sudo su - eda, podman run... the same container starts. I don't see selinux denials for one.

@giuseppe any ideas?

[eda@rh-ansibleeda-01 ~]$ id
uid=988(eda) gid=987(eda) groups=987(eda),988(nginx),989(redis) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-1.el9_2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: 606c693de21bcbab87e31002e46663c5f2dc8a9b'                                                                                    
  cpuUtilization:
    idlePercent: 99.38
    systemPercent: 0.24
    userPercent: 0.37
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "9.2"
  eventLogger: journald
  hostname: rh-ansibleeda-01.cool.lab
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 987
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 988
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.14.0-284.30.1.el9_2.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 14116360192
  memTotal: 16492187648
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.4-1.el9_2.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/user/988/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/988/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-3.el9.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 3221221376
  swapTotal: 3221221376
  uptime: 13h 60m 48.00s (Approximately 0.54 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  cool-quay-quay-quay.apps.ocpmgt.cool.lab:
    Blocked: false
    Insecure: true
    Location: cool-quay-quay-quay.apps.ocpmgt.cool.lab
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: cool-quay-quay-quay.apps.ocpmgt.cool.lab
    PullFromMirror: ""
store:
  configFile: /var/lib/ansible-automation-platform/eda/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 0
    stopped: 7
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/ansible-automation-platform/eda/.local/share/containers/storage
  graphRootAllocated: 27899465728
  graphRootUsed: 4010336256
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /tmp/ansible.kzaio7ez/containers
  transientStore: false
  volumePath: /var/lib/ansible-automation-platform/eda/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1692279033
  BuiltTime: Thu Aug 17 16:30:33 2023
  GitCommit: ""
  GoVersion: go1.19.10
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

[eda@rh-ansibleeda-01 ~]$ rpm -q podman kernel 
podman-4.4.1-13.el9_2.x86_64
kernel-5.14.0-162.6.1.el9_1.x86_64
kernel-5.14.0-284.30.1.el9_2.x86_64

[root@rh-ansibleeda-01 ~]# ls -laZ /sys/fs/cgroup/
total 0
dr-xr-xr-x. 12 root root system_u:object_r:cgroup_t:s0 0 Oct 10 00:32 .
drwxr-xr-x.  8 root root system_u:object_r:sysfs_t:s0  0 Oct  9 17:02 ..
-r--r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 cgroup.controllers
-rw-r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 cgroup.max.depth
-rw-r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 cgroup.max.descendants
-rw-r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 cgroup.procs
-r--r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 cgroup.stat
-rw-r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 cgroup.subtree_control
-rw-r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 cgroup.threads
-rw-r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 cpu.pressure
-r--r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 cpuset.cpus.effective
-r--r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 cpuset.mems.effective
-r--r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 cpu.stat
drwxr-xr-x.  2 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 dev-hugepages.mount
drwxr-xr-x.  2 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 dev-mqueue.mount
drwxr-xr-x.  2 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 init.scope
-rw-r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 io.pressure
-r--r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 io.stat
-r--r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 memory.numa_stat
-rw-r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 memory.pressure
--w-------.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 memory.reclaim
-r--r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 memory.stat
-r--r--r--.  1 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:04 misc.capacity
drwxr-xr-x.  2 root root system_u:object_r:cgroup_t:s0 0 Oct 10 00:32 proc-sys-fs-binfmt_misc.mount
drwxr-xr-x.  2 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 sys-fs-fuse-connections.mount
drwxr-xr-x.  2 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 sys-kernel-config.mount
drwxr-xr-x.  2 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 sys-kernel-debug.mount
drwxr-xr-x.  2 root root system_u:object_r:cgroup_t:s0 0 Oct  9 17:02 sys-kernel-tracing.mount
drwxr-xr-x. 40 root root system_u:object_r:cgroup_t:s0 0 Oct 10 07:02 system.slice
drwxr-xr-x.  4 root root system_u:object_r:cgroup_t:s0 0 Oct  9 20:00 user.slice

1 reply

giuseppe Oct 10, 2023
Maintainer

you need to make sure you are using the systemd cgroup driver.

Why is it using cgroupfs? Is it a nested container?

ikke-t · 2023-10-10T10:05:00Z

ikke-t
Oct 10, 2023

@giuseppe I have installed ansible automation platform with event driven ansible. It installs and configures all podman stuff. But likely there is some error. No, this is not nested container. More info about it here, where your opinions would be welcome: https://issues.redhat.com/browse/AAP-16958

1 reply

ikke-t Oct 10, 2023

@giuseppe thanks, that fixed the cgroups error. It didn't fix my problem, but good enhancement to ansible team. thanks.

Oct 10 13:24:05 rh-ansibleeda-01 podman[1699]: time="2023-10-10T13:24:05+03:00" level=info msg="Running conmon under slice user.slice and unitName libpod-conmon-74fe20ed6
7f05287387cf8a030be9ece8cfb57b723178c4fa5241c5218406857.scope"
Oct 10 13:24:05 rh-ansibleeda-01 podman[1699]: time="2023-10-10T13:24:05+03:00" level=info msg="Got Conmon PID as 1815"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Is that possible to set resource limit in rootless mode when using cgroupfs? #19158

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Is that possible to set resource limit in rootless mode when using cgroupfs? #19158

Uh oh!

m0n203 Jul 7, 2023

Replies: 4 comments · 3 replies

Uh oh!

rhatdan Jul 10, 2023 Maintainer

Uh oh!

giuseppe Jul 10, 2023 Maintainer

Uh oh!

m0n203 Jul 19, 2023 Author

Uh oh!

Uh oh!

ikke-t Oct 10, 2023

Uh oh!

giuseppe Oct 10, 2023 Maintainer

Uh oh!

ikke-t Oct 10, 2023

Uh oh!

ikke-t Oct 10, 2023

m0n203
Jul 7, 2023

Replies: 4 comments 3 replies

rhatdan
Jul 10, 2023
Maintainer

giuseppe
Jul 10, 2023
Maintainer

m0n203 Jul 19, 2023
Author

ikke-t
Oct 10, 2023

giuseppe Oct 10, 2023
Maintainer

ikke-t
Oct 10, 2023