Systemd service shutdown order for pods

[mwopfner]

Hi, I run a rootfull podman (v3.4.1) on a PC and start the pod/containers with podman kube play.
This creates systemd slices under machine.slice

CGroup: /
           ├─init.scope
           │ └─ 1 /sbin/init
           ├─machine.slice
           │ ├─machine-libpod_pod_5c64cc348d80d9f78e3b794aacfb03e88e8906b92e223671b0686a4c0a0ebbef.slice
           │ │ ├─libpod-8d87174d3a3b74a1415c821516e28a8e04818bcec90c598deb48fea54e5e49fc.scope
           │ │ │ ├─  497 sh /usr/bin/start_tinc -D -U nobody

If I run podman kube down before I reboot/poweroff the reboot works without a problem, but if reboot without stopping the pod first I get timeout errors for the pod slices and the system shutdown hangs until the timeouts happen.
I tried to setup a proper shutdown sequence with systemd After but it seams that pod slices gets directly stopped after the shutdown.target

Is there any way to disable the adding of pod slices to the machine.slice or to change them to another slice.
Unfortunately I could not find any useful information for this. The only thing was (https://discussion.fedoraproject.org/t/podman-systemd-services-shutdown-order/23181)

[vrothberg]

Thanks for reaching out, @mwopfner!

How are you running kube play? Are you running in a systemd unit?

[giuseppe]

the pod slice is somehow expected since that is what Kubernetes does.

We should try if DefaultDependencies could be applied to the slice and if it makes any difference

[mwopfner]

@giuseppe I can try this if you can give me a hint on where to put it. (kube.yaml file?)

[mwopfner]

Just for some more info i looked into the generated pod slice:

$ cat /run/systemd/transient/machine-libpod_pod_3458a7be508ea734838eae09f2b30cb3752037b120c9a85b49d72ba51fcf1e28.slice
# This is a transient unit file, created programmatically via the systemd API. Do not edit.
[Unit]
Description=cgroup machine-libpod_pod_3458a7be508ea734838eae09f2b30cb3752037b120c9a85b49d72ba51fcf1e28.slice
Wants=machine.slice
DefaultDependencies=no

[Slice]
MemoryAccounting=yes
CPUAccounting=yes
IOAccounting=yes

I guess it is this DefaultDependencies, but how to get another value into it?

[giuseppe]

you can use systemctl edit $SLICE_NAME

[mwopfner]

So, now I have finally time again to look into this problem.
It turns out that the REST API and podman kube play produces the same .slice file. Which in both cases have DefaultDependencies=no. What is the reason to have no here, because https://www.freedesktop.org/software/systemd/man/systemd.slice.html states:

Only slice units involved with late system shutdown should disable DefaultDependencies= option.

I further found out that not every kube file causes this error. It seams that only pods/containers which need longer to be stopped causes this issue. I thing this also has to do with my specific setup.
I have my containers and volumes on a extra /app partition (my rootfs is readonly) and now on shutdown systemd tries to unmount /app before all my containers are stopped. This of course fails:

umount[2294]: umount: /app: target is busy.
systemd[1]: app.mount: Mount process exited, code=exited, status=32/n/a
systemd[1]: Failed unmounting /app.

Later in the log I get

systemd[1]: libpod-6f551c2e908868ca3dd4dd88b9bfbc444a98967bd44af782eccae55ec35590f9.scope: Stopping timed out. Killing.

which finally kills the hanging pod and the system shuts down.
Stopping the pod with podman kube down before I trigger poweroff works. So the pod shutdown works but needs some seconds for my application.