Testing systemd in containers: https://github.com/LewisGaul/systemd-containers

[LewisGaul]

I have written some tests exercising systemd in containers using python-on-whales: https://github.com/LewisGaul/systemd-containers

Some relevant bits that use python-on-whales:

• Creating container client (supporting docker/podman, and remote host):
  • tests/conftest.py#L83
  • tests/utils.py#L54
• Building images from a Dockerfile passed as text:
  • tests/utils.py#L215
• Starting a systemd container, setting required args:
  • tests/conftest.py#L432
• Running container with modified image's entrypoint:
  • tests/test_exec_procs.py#L20
• Spamming container exec processes:
  • tests/test_exec_procs.py#L82

Example usage:

(venv) systemd-containers/(main)$ pytest --log-cli-level=debug --container-exe=docker --setup-mode=unmount --cgroup-mode=hybrid --cgroupns=host -k "test_non_priv or test_cgroup"
=============================================== test session starts ================================================
platform linux -- Python 3.9.5, pytest-7.4.0, pluggy-1.2.0
rootdir: /mnt/c/Users/legaul/Documents/personal/systemd-containers
configfile: pytest.ini
testpaths: tests/
collecting 240 items
----------------------------------------------- live log collection ------------------------------------------------
09:32:54:DEBUG[tests.conftest:170] Removing 296 test parameterisations that don't apply
09:32:54:DEBUG[tests.conftest:193] Removing 269 test parameterisations due to CLI args
collected 576 items / 5 deselected / 571 selected

tests/test_boot_success.py::test_non_priv[host-hybrid-unmount]
-------------------------------------------------- live log setup --------------------------------------------------
09:32:54:INFO[tests.conftest:234] Using container manager docker, see debug logs for detailed info
09:32:54:DEBUG[tests.utils:99] Running command: 'docker info'
09:32:54:DEBUG[tests.utils:149] Command stdout:
... <snip> ...
09:32:54:DEBUG[tests.utils:229] Building image using Dockerfile:
FROM ubuntu:20.04
RUN apt-get update -y \
    && apt-get install -y systemd \
    && ln -s /lib/systemd/systemd /sbin/init \
    && systemctl mask systemd-resolved.service \
    && systemctl set-default multi-user.target
RUN echo 'root:root' | chpasswd
STOPSIGNAL SIGRTMIN+3
ENTRYPOINT ["/sbin/init"]
09:32:54:DEBUG[tests.utils:229] Building image using Dockerfile:
FROM ubuntu-systemd:20.04
COPY init_script.sh /init_script.sh
ENTRYPOINT ["/init_script.sh"]
-------------------------------------------------- live log call ---------------------------------------------------
09:32:55:INFO[tests.conftest:474] Running container image ubuntu-systemd-unmount:20.04 with args: cap_add=['sys_admin'], tmpfs=['/run'], envs={'container': 'docker'}, tty=True, interactive=True, detach=True, remove=False, cgroupns=host, name=systemd-tests-1695717175.27
09:32:56:DEBUG[tests.conftest:514] Init script logs:
2023-09-26 08:32:55,772: Detected cgroups v1
2023-09-26 08:32:55,773: Unmounting all /sys/fs/cgroup mounts (allow systemd to recreate)
09:32:56:DEBUG[tests.conftest:515] Container boot logs:
systemd 245.4-4ubuntu3.22 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization wsl.
Detected architecture x86-64.

Welcome to Ubuntu 20.04.6 LTS!

Set hostname to <63f53fa2322b>.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Created slice system-modprobe.slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password ▒ts to Console Directory Watch.
[  OK  ] Started Forward Password R▒uests to Wall Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
         Mounting Huge Pages File System...
systemd-journald.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/unified/docker/63f53fa2322b2edeb2f796759783575e1a81e3b0beac3cb04f3f8aa94e8578ea/system.slice/systemd-journald.service failed: Invalid argument
         Starting Journal Service...
         Mounting FUSE Control File System...
         Starting Remount Root and Kernel File Systems...
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Starting Create System Users...
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Finished Flush Journal to Persistent Storage.
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Create Volatile Files and Directories.
[  OK  ] Reached target System Time Set.
[  OK  ] Reached target System Time Synchronized.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Finished Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily apt download activities.
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Started Periodic ext4 Onli▒ata Check for All Filesystems.
[  OK  ] Started Message of the Day.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
[  OK  ] Started D-Bus System Message Bus.
         Starting Dispatcher daemon for systemd-networkd...
         Starting Login Service...
         Starting Permit User Sessions...
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Login Service.
[  OK  ] Started Dispatcher daemon for systemd-networkd.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.

09:32:57:WARNING[tests.test_boot_success:90] Unexpected boot log lines:
systemd-journald.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/unified/docker/63f53fa2322b2edeb2f796759783575e1a81e3b0beac3cb04f3f8aa94e8578ea/system.slice/systemd-journald.service failed: Invalid argument
PASSED                                                                                                       [ 16%]
tests/test_boot_success.py::test_non_priv_systemd_mode[host-hybrid-unmount] SKIPPED (Systemd mode not su...) [ 33%]
tests/test_cgroups.py::test_cgroup_dir[host-hybrid-unmount]
-------------------------------------------------- live log call ---------------------------------------------------
09:32:58:INFO[tests.conftest:474] Running container image ubuntu-systemd-unmount:20.04 with args: tmpfs=['/run'], envs={'container': 'docker'}, cap_add=['sys_admin'], tty=True, interactive=True, detach=True, remove=False, cgroupns=host, name=systemd-tests-1695717178.02
09:32:59:DEBUG[tests.test_cgroups:19] Contents of /sys/fs/cgroup/:
total 0
dr-xr-xr-x 3 root root  0 Sep 24 14:05 blkio
lrwxrwxrwx 1 root root 11 Sep 26 08:32 cpu -> cpu,cpuacct
drwxr-xr-x 2 root root 40 Sep 26 08:32 cpu,cpuacct
lrwxrwxrwx 1 root root 11 Sep 26 08:32 cpuacct -> cpu,cpuacct
dr-xr-xr-x 3 root root  0 Sep 24 14:05 cpuset
dr-xr-xr-x 5 root root  0 Sep 24 14:05 devices
dr-xr-xr-x 3 root root  0 Sep 24 14:05 freezer
dr-xr-xr-x 3 root root  0 Sep 24 14:05 hugetlb
dr-xr-xr-x 6 root root  0 Sep 24 14:05 memory
lrwxrwxrwx 1 root root 16 Sep 26 08:32 net_cls -> net_cls,net_prio
drwxr-xr-x 2 root root 40 Sep 26 08:32 net_cls,net_prio
lrwxrwxrwx 1 root root 16 Sep 26 08:32 net_prio -> net_cls,net_prio
dr-xr-xr-x 3 root root  0 Sep 24 14:05 perf_event
dr-xr-xr-x 5 root root  0 Sep 24 14:05 pids
dr-xr-xr-x 3 root root  0 Sep 24 14:05 rdma
dr-xr-xr-x 6 root root  0 Sep 25 16:01 systemd
dr-xr-xr-x 3 root root  0 Sep 24 14:05 unified
PASSED                                                                                                       [ 50%]
tests/test_cgroups.py::test_cgroup_mounts[host-hybrid-unmount]
-------------------------------------------------- live log call ---------------------------------------------------
09:33:00:INFO[tests.conftest:474] Running container image ubuntu-systemd-unmount:20.04 with args: tmpfs=['/run'], envs={'container': 'docker'}, cap_add=['sys_admin'], tty=True, interactive=True, detach=True, remove=False, cgroupns=host, name=systemd-tests-1695717180.15
09:33:01:DEBUG[tests.test_cgroups:29] Cgroup mounts:
TARGET                      SOURCE  FSTYPE  OPTIONS
/sys/fs/cgroup              tmpfs   tmpfs   ro,nosuid,nodev,noexec,mode=755
|-/sys/fs/cgroup/unified    cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate
|-/sys/fs/cgroup/systemd    cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,xattr,name=systemd
|-/sys/fs/cgroup/rdma       cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,rdma
|-/sys/fs/cgroup/blkio      cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,blkio
|-/sys/fs/cgroup/cpuset     cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,cpuset
|-/sys/fs/cgroup/memory     cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,memory
|-/sys/fs/cgroup/hugetlb    cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,hugetlb
|-/sys/fs/cgroup/pids       cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,pids
|-/sys/fs/cgroup/freezer    cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,freezer
|-/sys/fs/cgroup/perf_event cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,perf_event
`-/sys/fs/cgroup/devices    cgroup  cgroup  rw,nosuid,nodev,noexec,relatime,devices
PASSED                                                                                                       [ 66%]
tests/test_cgroups.py::test_cgroup_paths[host-hybrid-unmount]
-------------------------------------------------- live log call ---------------------------------------------------
09:33:02:INFO[tests.conftest:474] Running container image ubuntu-systemd-unmount:20.04 with args: tmpfs=['/run'], envs={'container': 'docker'}, cap_add=['sys_admin'], tty=True, interactive=True, detach=True, remove=False, cgroupns=host, name=systemd-tests-1695717182.69
09:33:04:DEBUG[tests.test_cgroups:43] Got PID 1 cgroups:
14:name=systemd:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c/init.scope
13:rdma:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
12:pids:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
11:hugetlb:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
10:net_prio:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
9:perf_event:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
8:net_cls:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
7:freezer:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
6:devices:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
5:blkio:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
4:cpuacct:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
3:cpu:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
2:cpuset:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
1:memory:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
0::/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c/init.scope
09:33:04:DEBUG[tests.test_cgroups:46] Got systemd-journald (PID 25) cgroups:
14:name=systemd:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c/system.slice/systemd-journald.service
13:rdma:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
12:pids:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c/system.slice/systemd-journald.service
11:hugetlb:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
10:net_prio:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
9:perf_event:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
8:net_cls:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
7:freezer:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
6:devices:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c/system.slice/systemd-journald.service
5:blkio:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
4:cpuacct:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
3:cpu:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
2:cpuset:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c
1:memory:/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c/system.slice/systemd-journald.service
0::/docker/97ecda794d72e6e641306916fbb2fc5e8738ddc1e6ceeadf36217b5716b0390c/system.slice/systemd-journald.service
PASSED                                                                                                       [ 83%]
tests/test_cgroups.py::test_cgroup_controllers[host-hybrid-unmount]
-------------------------------------------------- live log call ---------------------------------------------------
09:33:05:INFO[tests.conftest:474] Running container image ubuntu-systemd-unmount:20.04 with args: tmpfs=['/run'], envs={'container': 'docker'}, cap_add=['sys_admin'], tty=True, interactive=True, detach=True, remove=False, cgroupns=host, name=systemd-tests-1695717185.28
09:33:06:DEBUG[tests.test_cgroups:59] Enabled controllers: {'memory', 'pids', 'devices'}
PASSED                                                                                                       [100%]

=================================== 5 passed, 1 skipped, 5 deselected in 14.14s ====================================