Utility process autorestart

[CrossRoast]

When a container or a pod is launched via systemd, the only process systemd seems to track and restart is conmon. Conmon, in turn, watches the container. However, it seems, if utility processes, such as aardvark-dns, slirp4netns/pasta or rootlessport are killed, they are not restarted. Should conmon, perhaps, restart those too? Is there a way to add resilience to podman?

[baude]

examples of what you are currently trying might help this conversation .

[CrossRoast]

Sure. I've tested it with quadlets, podman generate systemd and podman-compose. Testing this on Debian 13, so the podman version is 5.4.2. I'm running an nginx container proxying to a hello-world container. Here are the processes running on the system via quadlets:

$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.3  2.9  23168 14068 ?        Ss   22:23   0:02 /sbin/init
root           2  0.0  0.0      0     0 ?        S    22:23   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        S    22:23   0:00 [pool_workqueue_release]
root           4  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-kvfree_rcu_reclaim]
root           5  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-rcu_gp]
root           6  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-sync_wq]
root           7  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-slub_flushwq]
root           8  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-netns]
root           9  0.0  0.0      0     0 ?        I    22:23   0:00 [kworker/0:0-cgroup_bpf_destroy]
root          11  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/0:0H-events_highpri]
root          13  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-mm_percpu_wq]
root          14  0.0  0.0      0     0 ?        I    22:23   0:00 [rcu_tasks_kthread]
root          15  0.0  0.0      0     0 ?        I    22:23   0:00 [rcu_tasks_rude_kthread]
root          16  0.0  0.0      0     0 ?        I    22:23   0:00 [rcu_tasks_trace_kthread]
root          17  0.0  0.0      0     0 ?        S    22:23   0:00 [ksoftirqd/0]
root          18  0.0  0.0      0     0 ?        R    22:23   0:00 [rcu_preempt]
root          19  0.0  0.0      0     0 ?        S    22:23   0:00 [rcu_exp_par_gp_kthread_worker/0]
root          20  0.0  0.0      0     0 ?        S    22:23   0:00 [rcu_exp_gp_kthread_worker]
root          21  0.0  0.0      0     0 ?        S    22:23   0:00 [migration/0]
root          22  0.0  0.0      0     0 ?        S    22:23   0:00 [idle_inject/0]
root          23  0.0  0.0      0     0 ?        S    22:23   0:00 [cpuhp/0]
root          25  0.0  0.0      0     0 ?        S    22:23   0:00 [kdevtmpfs]
root          26  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-inet_frag_wq]
root          27  0.0  0.0      0     0 ?        S    22:23   0:00 [kauditd]
root          28  0.0  0.0      0     0 ?        S    22:23   0:00 [khungtaskd]
root          29  0.0  0.0      0     0 ?        I    22:23   0:00 [kworker/u60:1-events_unbound]
root          30  0.0  0.0      0     0 ?        S    22:23   0:00 [oom_reaper]
root          31  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-writeback]
root          32  0.0  0.0      0     0 ?        S    22:23   0:00 [kcompactd0]
root          33  0.0  0.0      0     0 ?        SN   22:23   0:00 [ksmd]
root          34  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-kintegrityd]
root          35  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-kblockd]
root          36  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-blkcg_punt_bio]
root          37  0.0  0.0      0     0 ?        S    22:23   0:00 [irq/9-acpi]
root          38  0.0  0.0      0     0 ?        S    22:23   0:00 [xen-balloon]
root          39  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-tpm_dev_wq]
root          40  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/0:1H-kblockd]
root          41  0.0  0.0      0     0 ?        S    22:23   0:00 [kswapd0]
root          45  0.0  0.0      0     0 ?        I    22:23   0:00 [kworker/u60:3-events_unbound]
root          50  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-kthrotld]
root          54  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-acpi_thermal_pm]
root          55  0.0  0.0      0     0 ?        S    22:23   0:00 [xenbus]
root          56  0.0  0.0      0     0 ?        S    22:23   0:00 [xenwatch]
root          57  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-nvme-wq]
root          58  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-nvme-reset-wq]
root          59  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-nvme-delete-wq]
root          60  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-nvme-auth-wq]
root          61  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-mld]
root          62  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-ipv6_addrconf]
root          67  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-kstrp]
root          69  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/u61:0]
root         173  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-ata_sff]
root         174  0.0  0.0      0     0 ?        S    22:23   0:00 [scsi_eh_0]
root         175  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-scsi_tmf_0]
root         176  0.0  0.0      0     0 ?        S    22:23   0:00 [scsi_eh_1]
root         177  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-scsi_tmf_1]
root         196  0.0  0.0      0     0 ?        S    22:23   0:00 [jbd2/xvda1-8]
root         197  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-ext4-rsv-conversion]
root         272  0.0  3.2  59584 15112 ?        Ss   22:23   0:00 /usr/lib/systemd/systemd-journald
systemd+     318  0.0  3.1  23512 14724 ?        Ss   22:23   0:00 /usr/lib/systemd/systemd-resolved
systemd+     319  0.0  1.6 165604  7984 ?        Ssl  22:23   0:00 /usr/lib/systemd/systemd-timesyncd
root         328  0.0  2.1  32872 10308 ?        Ss   22:23   0:00 /usr/lib/systemd/systemd-udevd
root         329  0.0  0.0      0     0 ?        S    22:23   0:00 [psimon]
root         448  0.0  0.0      0     0 ?        I<   22:23   0:00 [kworker/R-cryptd]
root         494  0.0  0.0      0     0 ?        I    22:23   0:00 [kworker/u60:4-events_unbound]
root         544  0.0  0.0      0     0 ?        I    22:23   0:00 [kworker/0:3-cgroup_destroy]
systemd+     576  0.0  2.5  21736 11924 ?        Ss   22:23   0:00 /usr/lib/systemd/systemd-networkd
message+     625  0.0  0.9   8052  4584 ?        Ss   22:23   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         636  0.0  1.9  18664  9176 ?        Ss   22:23   0:00 /usr/lib/systemd/systemd-logind
root         648  0.0  1.6  11768  7960 ?        Ss   22:23   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root         656  0.0  0.5   5168  2564 tty1     Ss+  22:23   0:00 /sbin/agetty -o -- \u --noreset --noclear - linux
root         663  0.0  0.5   5212  2740 ttyS0    Ss+  22:23   0:00 /sbin/agetty -o -- \u --noreset --noclear --keep-baud 115200,57600,38400,9600 - vt220
root         670  0.0  4.5  36852 21480 ?        Ss   22:23   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root         676  0.0  0.0      0     0 ?        S    22:23   0:00 [psimon]
polkitd      677  0.0  1.7 380436  8108 ?        Ssl  22:23   0:00 /usr/lib/polkit-1/polkitd --no-debug --log-level=notice
testuser     679  0.0  2.7  23872 13148 ?        Ss   22:23   0:00 /usr/lib/systemd/systemd --user
testuser     681  0.0  0.7  22384  3620 ?        S    22:23   0:00 (sd-pam)
root         715  0.0  2.6  19340 12556 ?        Ss   22:23   0:00 sshd-session: testuser [priv]
testuser     723  0.0  1.5  19460  7084 ?        S    22:23   0:00 sshd-session: testuser@pts/0
testuser     724  0.0  1.1   6260  5540 pts/0    Ss   22:23   0:00 -bash
testuser     748  0.0  0.1   1032   732 ?        S    22:24   0:00 catatonit -P
testuser     775  0.0  0.8   7768  4172 ?        Ss   22:24   0:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root        1106  0.0  0.0      0     0 ?        I    22:28   0:00 [kworker/0:1-rcu_gp]
root        1245  0.0  0.0      0     0 ?        I    22:30   0:00 [kworker/u60:0-events_unbound]
root        1246  0.0  0.0      0     0 ?        I    22:30   0:00 [kworker/u60:2-events_unbound]
root        1266  0.0  0.0      0     0 ?        I    22:30   0:00 [kworker/0:2-events]
testuser    1294  0.0  0.8  74044  3872 ?        Ssl  22:30   0:00 /usr/lib/podman/aardvark-dns --config /run/user/1001/containers/networks/aardvark-dns -p 53 run
testuser    1297  0.0  0.4   7380  2344 ?        Ss   22:30   0:00 /usr/bin/conmon --api-version 1 -c cdafac2310e397d0b7e891c4d06260a190023b07d478d6cc4392d12cfb7d31fa -u cdafac2310e397d0b7e891c4d06260a190023b07d478d
testuser    1300  0.0  0.1   1324   672 ?        Ss   22:30   0:00 /bin/sh -c echo "httpd started" && trap "exit 0;" TERM INT; httpd -v -p $PORT -h /www -f & wait
testuser    1303  0.0  0.1   1320   572 ?        S    22:30   0:00 httpd -v -p 8000 -h /www -f
testuser    1428  0.0  3.5 206272 16556 ?        Ss   22:32   0:00 /usr/bin/pasta --config-net --pid /run/user/1001/containers/networks/rootless-netns/rootless-netns-conn.pid --dns-forward 169.254.1.1 -t none -u non
testuser    1462  0.0  1.0 1597732 5100 ?        Sl   22:32   0:00 rootlessport
testuser    1468  0.0  1.0 1524000 4872 ?        Sl   22:32   0:00 rootlessport-child
testuser    1476  0.0  0.4   7380  2080 ?        Ss   22:32   0:00 /usr/bin/conmon --api-version 1 -c d4d2cb5696f1b6dc0f7e648c365538787dc410bac39ac312f4895f2fe8319652 -u d4d2cb5696f1b6dc0f7e648c365538787dc410bac39ac
testuser    1479  0.0  0.3   2580  1604 ?        Ss   22:32   0:00 /bin/sh -c while :; do sleep 6h & wait ${!}; /usr/sbin/nginx -s reload; done & /usr/sbin/nginx -c /etc/nginx/nginx.conf -g "daemon off;"
testuser    1481  0.0  0.2   2580  1060 ?        S    22:32   0:00 /bin/sh -c while :; do sleep 6h & wait ${!}; /usr/sbin/nginx -s reload; done & /usr/sbin/nginx -c /etc/nginx/nginx.conf -g "daemon off;"
testuser    1482  0.0  0.3   2488  1448 ?        S    22:32   0:00 sleep 6h
testuser    1483  0.0  1.6  32032  7852 ?        S    22:32   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf -g daemon off;
165636      1484  0.0  0.6  32480  3184 ?        S    22:32   0:00 nginx: worker process
testuser    1501  0.0  0.8   6848  4084 pts/0    R+   22:32   0:00 ps aux

If I run kill -9 1297 or kill -9 1476, killing either one of the conmon processes, they get restarted by systemd.
If I run kill -9 1294, kill -9 1428 or kill -9 1462, killing aardvark-dns, pasta or rootlessport respectively, they do not get restarted and the system keeps running in a broken state. I can no longer curl localhost and get a hello world response.

It would be great if all the processes required for podman to function would get restarted. Those processes could crash or could be killed by oom_reaper. Is it possible to configure that or would it, perhaps, make sense to add that functionality?