systemd ignores TimeoutStopSec when rebooting server and kills container

[ihatethecloud]

Issue Description

Systemd kills container with SIGKILL while systemctl stop waits for container to shutdown.

Steps to reproduce the issue

Steps to reproduce the issue

1. Generated systemd unit file with podman generate systemd
2. Added TimeoutStopSec=400
3. ExitStop= .. --ignore -t 350 ...

Describe the results you received

systemctl stop container-my app.service waits until container shuts down(~173 seconds)
reboot kills the container after 90 seconds.

Describe the results you expected

reboot waits until container shuts down

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.8-1.module_el8.9.0+3643+9234dc3b.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: d30adc73700e7871656d41a5afe7c9544ab83b9f'
  cpuUtilization:
    idlePercent: 79.8
    systemPercent: 4.9
    userPercent: 15.3
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: '"almalinux"'
    version: "8.9"
  eventLogger: file
  freeLocks: 2048
  hostname: server
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 200
      size: 1
    - container_id: 1
      host_id: 1048576
      size: 4096
    uidmap:
    - container_id: 0
      host_id: 200
      size: 1
    - container_id: 1
      host_id: 1048576
      size: 4096
  kernel: 4.18.0-513.9.1.el8_9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3273056256
  memTotal: 16512942080
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.6.1-4.module_el8.9.0+3643+9234dc3b.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.3.1
        commit: unknown
    package: containernetworking-plugins-1.3.0-4.module_el8.9.0+3643+9234dc3b.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: runc-1.1.9-1.module_el8.9.0+3643+9234dc3b.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.9
      spec: 1.0.2-dev
      go: go1.20.6
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /tmp/podman-run-200/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.module_el8.9.0+3643+9234dc3b.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 17179865088
  swapTotal: 17179865088
  uptime: 11h 14m 39.00s (Approximately 0.46 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 53660876800
  graphRootUsed: 21016059904
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /tmp/containers-user-200/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1697638007
  BuiltTime: Wed Oct 18 10:06:47 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[rhatdan]

Isn't this an issue with systemd, don't see how Podman can control what systemd is doing here?

[ihatethecloud]

Should I add something to what Podman systemd generate so that the behaviour of systemctl stop container.service also applies to reboot?

[rhatdan]

No idea, but I would be more interested on how this is handled with Quadlet, since we are encouraging people to use it rather the podman systemd generate.

[ihatethecloud]

It's the same thing also with Quadlet.... questions is, how is the proper way to stop a container which has systemd running inside it? I also tried this and it's the same outcome: PodmanArgs=--stop-signal SIGRTMIN+3

[ihatethecloud]

Also... it looks to me that it's trying to rm the container while systemd is still running:

             ├─ myservice.service
             │ ├─.control
             │ │ └─13479 /usr/bin/podman rm -f -i --cidfile=/run/myservice.cid
             │ ├─libpod-payload-ff3f9a3d1b4b042ac5c26cc315befa2b6bd875c2408e062e85f992916c0d3456
             │ │ ├─init.scope
             │ │ │ └─10888 /sbin/init
             │ │ └─system.slice

[Luap99]

If you run the containers rootless then they are part of the system user session. So I think it is likely that your user session runs into the default 90s timeout and then all user processes are killed AFAIK.
If you run journalctl -b-1 -r -u user@$UID then you should see what it did on the last shutdown.

[ihatethecloud]

I run them as root.... see below reply. I posted the system unit.

[ihatethecloud]

I have the following unit.

Description=Podman container-myservice.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=no
TimeoutStopSec=200
ExecStart=/usr/bin/podman run \
	--cidfile=%t/%n.ctr-id \
	--cgroups=no-conmon \
	--rm \
	--sdnotify=conmon \
	-d \
	--replace \
	--name=myservice \
	--security-opt seccomp=unconfined \
	--label io.podman.compose.config-hash=123 \
	--label io.podman.compose.project=myservice \
	--label io.podman.compose.version=0.0.1 \
	--label com.docker.compose.container-number=1 \
	--label com.docker.compose.service=myservice \
	--network host \
	--cap-add CAP_SYS_PTRACE \
	--cap-add CAP_NET_ADMIN \
	--cap-add SYS_RAWIO \
	-e "PS1=[\\u@\\h (myservice) \\W]\\$$ " \
	-v /mnt/data/myservice:/mnt/data 
	--add-host nginx:127.0.0.1 project/myimage
ExecStop=/usr/bin/podman stop \
	--ignore -t 200 \
	--cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
	-f \
	--ignore -t 200 \
	--cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target

Stopping with systemctl stop is fast.

$ sudo systemctl stop container-myservice.service
Feb 14 14:31:02 server systemd[1]: Stopping Podman container-myservice.service...
Feb 14 14:31:03 server podman[19658]: e5f5904ee9feb17f130f931be3269e7cec36ec47307417a0d952b5f863c4c52b
Feb 14 14:31:03 server podman[19749]: e5f5904ee9feb17f130f931be3269e7cec36ec47307417a0d952b5f863c4c52b
Feb 14 14:31:03 server systemd[1]: container-myservice.service: Succeeded.
Feb 14 14:31:03 server systemd[1]: Stopped Podman container-myservice.service.

Stopping with reboot

podman stop keeps running

 │ ├─container-myservice.service
           │ │ ├─19983 /usr/bin/conmon --api-version 1 -c 2d4e793d7b4552744ae051f61f5650b8924a6bdbe7bf49a9dfde9f508534c91d -u 2d4e793d7b4552744ae051f61f5650b8>
           │ │ └─20284 /usr/bin/podman stop --ignore -t 100 --cidfile=/run/container-myservice.service.ctr-id

after the reboot looking at the journal I see a 90 second timeout between stopping and killing which is consistent with DefaultTimeoutStopSec=90s.

Feb 14 14:34:14 server systemd[1]: Stopping Podman container-myservice.service...
Feb 14 14:35:24 server systemd[1]: container-myservice.service: Stopping timed out. Terminating.
Feb 14 14:35:44 server systemd[1]: container-myservice.service: Main process exited, code=exited, status=137/n/a
Feb 14 14:35:44 server systemd[1]: container-myservice.service: Failed with result 'timeout'.
Feb 14 14:35:44 server systemd[1]: Stopped Podman container-myservice.service.

Worth metioning the fact that the container has system running inside it. If I run the ExecStop command the container closes cleanly after a few seconds.

$ sudo /usr/bin/podman stop --ignore -t 100 --cidfile=/run/container-myservice.service.ctr-id
$ sudo podman inspect --format '{{.State.ExitCode}}' myservice
0

Somehow my container doesn't understand SIGTERM if the podman stop is sent by systemd.

[vrothberg]

But SIGKILL should terminate the container immediately, so I cannot put the pieces together at the moment.

[ihatethecloud]

It does send SIGRTMIN+3 and the container shuts down if I run systemctl stop but it doesn’t react the same way if I run reboot. It does run podman stop but no reaction from the container and then it’s killed with SIGKILL when the default systemd stop timeout expired.

Funny thing. I tested with quadlets as some suggested and it works just fine. Still scratching my head over this issue though.

[vrothberg]

To be extra sure: it works with Quadlet but does not with generate systemd?

[vrothberg]

I´d recommend using Quadlet in any case since generate systemd will not receive new features.

[ihatethecloud]

Yes. It's like that.

I settled with Quadlet. Works great so far.