Unable to run a systemd container with ro /sys/fs/cgroup

[jay7x]

Description

I'm not sure this is right place to report though as I'm using nerdctl & containerd within a lima-vm (https://github.com/lima-vm/lima/). Feel free to move or close the issue if unrelated 🙇🏻

I was trying to run a systemd-enabled container with nerdctl in a lima-managed virtual machine on my m1-based macbook pro.

I tried 2 different images w/o success:

• https://hub.docker.com/r/jrei/systemd-ubuntu
• https://hub.docker.com/r/dhoppeit/docker-ubuntu-systemd

With systemd debug enabled I got the following errors:

    Failed to create /init.scope control group: Read-only file system
    Failed to allocate manager object: Read-only file system
    [!!!!!!] Failed to allocate manager object.

I found the issue is described in moby repo: moby/moby#42275. Though I'm really surprised as nerdctl is not using moby as far as I understand.. 🤔

Steps to reproduce the issue

1. Provision the VM with limactl start default
2. SSH into the VM with limactl shell default

3.  jay@lima-default:~$ sudo nerdctl run -it --tmpfs /tmp --tmpfs /run --tmpfs /run/lock --cap-add SYS_ADMIN --name ubuntu-20.04 --volume /sys/fs/cgroup:/sys/fs/cgroup:ro dhoppeit/docker-ubuntu-systemd:20.04 /lib/systemd/systemd --log-level=debug --log-target=console
    systemd 245.4-4ubuntu3.19 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
    Detected virtualization docker.
    Detected architecture arm64.
    
    Welcome to Ubuntu 20.04.5 LTS!
    
    Set hostname to <e950a4b16a5e>.
    Failed to add address 127.0.0.1 to loopback interface: Operation not permitted
    Failed to add address ::1 to loopback interface: Operation not permitted
    Failed to bring loopback interface up: Operation not permitted
    Failed to bump AF_UNIX datagram queue length, ignoring: Read-only file system
    Setting 'fs/file-max' to '9223372036854775807'.
    Failed to bump fs.file-max, ignoring: Read-only file system
    Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
    Unified cgroup hierarchy is located at /sys/fs/cgroup.
    Failed to create /init.scope control group: Read-only file system
    Failed to allocate manager object: Read-only file system
    [!!!!!!] Failed to allocate manager object.
    Exiting PID 1...
   

Describe the results you received and expected

Got:

    Failed to create /init.scope control group: Read-only file system
    Failed to allocate manager object: Read-only file system
    [!!!!!!] Failed to allocate manager object.

Want: systemd-enabled container is up and running w/o any issues.

What version of nerdctl are you using?

jay@lima-default:~$ /usr/local/bin/nerdctl --version
nerdctl version 1.0.0
jay@lima-default:~$ /usr/local/bin/containerd --version
containerd github.com/containerd/containerd v1.6.8 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

Lima

Host information

jay@lima-default:~$ nerdctl info
Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: v1.6.8
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: native overlayfs stargz fuse-overlayfs
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
  rootless
 Kernel Version: 5.19.0-26-generic
 Operating System: Ubuntu 22.10
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.813GiB
 Name: lima-default
 ID: 6272f224-2be7-43b4-8579-1fa018a88f1a

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[AkihiroSuda]

Running systemd inside a container basically requires full --privileged

[jay7x]

The --privileged option doesn't help actually. I even tried to remove the :ro option from cgroup volume mapping.

jay@lima-default:~$ sudo nerdctl run -it --tmpfs /tmp --tmpfs /run --tmpfs /run/lock --privileged --name ubuntu-20.04 --volume /sys/fs/cgroup:/sys/fs/cgroup dhoppeit/docker-ubuntu-systemd:20.04 /lib/systemd/systemd --log-level=debug --log-target=console
systemd 245.4-4ubuntu3.19 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture arm64.

Welcome to Ubuntu 20.04.5 LTS!

Set hostname to <c39d4e655087>.
Failed to add address 127.0.0.1 to loopback interface: File exists
Failed to add address ::1 to loopback interface: File exists
Successfully brought loopback interface up
Setting 'fs/file-max' to '9223372036854775807'.
Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
Unified cgroup hierarchy is located at /sys/fs/cgroup.
Failed to create /init.scope control group: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...

JFYI, it works in podman with :ro and w/o --privileged (though it's Fedora 36 with kernel 5.17.5-300.fc36.aarch64 when ubuntu kernel is 5.19.0-26-generic):

[jay@lima-podman jay]$ podman run -it --tmpfs /tmp --tmpfs /run --tmpfs /run/lock --cap-add SYS_ADMIN --name ubuntu-20.04 --volume /sys/fs/cgroup:/sys/fs/cgroup:ro dhoppeit/docker-ubuntu-systemd:20.04 /lib/systemd/systemd 
systemd 245.4-4ubuntu3.19 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture arm64.

Welcome to Ubuntu 20.04.5 LTS!

Set hostname to <778ef425a7b4>.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
         Starting Remount Root and Kernel File Systems...
         Starting Create Static Device Nodes in /dev...
         Starting Journal Service...
[  OK  ] Finished Remount Root and Kernel File Systems.
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
[  OK  ] Started Journal Service.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Create Volatile Files and Directories.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.

According to https://lists.freedesktop.org/archives/systemd-devel/2018-November/041644.html both OSes are using unified hierarchy (cgroup2fs is mounted under /sys/fs/cgroup).

[LewisGaul]

Why are you passing --volume /sys/fs/cgroup:/sys/fs/cgroup? The cgroup mount should be created by containerd inside the container, using a private cgroup namespace by default on cgroups v2, so overwriting that with the host's cgroupfs is a recipe for causing problems.

I think this just happens to work with podman because in systemd mode it remounts the cgroupfs read-write after doing volume mounts:

root@ubuntu:~# podman run --rm --volume /sys/fs/cgroup:/sys/fs/cgroup:ro --systemd always ubuntu:20.04 grep cgroup /proc/mounts
cgroup2 /sys/fs/cgroup cgroup2 rw,relatime,nsdelegate,memory_recursiveprot 0 0
root@ubuntu:~# podman run --rm --volume /sys/fs/cgroup:/sys/fs/cgroup:ro ubuntu:20.04 grep cgroup /proc/mounts
cgroup2 /sys/fs/cgroup cgroup2 ro,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0

But you shouldn't be passing the volume mount:

root@ubuntu:~# podman run --rm --systemd always ubuntu:20.04 grep cgroup /proc/mounts
cgroup2 /sys/fs/cgroup cgroup2 rw,relatime,nsdelegate,memory_recursiveprot 0 0