Can't start rootless pod systemd service #17123

xxyzz · 2023-01-16T01:42:38Z

xxyzz
Jan 16, 2023

Steps to reproduce:

$ podman pod create alpine_pod
$ podman create --pod alpine_pod docker.io/alpine
$ cd .config/systemd/user
$ podman generate systemd -n alpine_pod -f
$ systemctl --user daemon-reload
$ systemctl --user start pod-alpine_pod

Generated systemd files:

[Unit]
Description=Podman pod-alpine_pod.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/run/user/1000/containers
Wants=container-wizardly_hofstadter.service
Before=container-wizardly_hofstadter.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStart=/usr/bin/podman start 4e6aac60a5cc-infra
ExecStop=/usr/bin/podman stop  \
        -t 10 4e6aac60a5cc-infra
ExecStopPost=/usr/bin/podman stop  \
        -t 10 4e6aac60a5cc-infra
PIDFile=/run/user/1000/containers/overlay-containers/b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2/userdata/conmon.pid
Type=forking

[Install]
WantedBy=default.target

[Unit]
Description=Podman container-wizardly_hofstadter.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/run/user/1000/containers
BindsTo=pod-alpine_pod.service
After=pod-alpine_pod.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStart=/usr/bin/podman start wizardly_hofstadter
ExecStop=/usr/bin/podman stop  \
        -t 10 wizardly_hofstadter
ExecStopPost=/usr/bin/podman stop  \
        -t 10 wizardly_hofstadter
PIDFile=/run/user/1000/containers/overlay-containers/7b6c59bcb8b68c8731e07818d1b2c83f10f305a3bdea7b7cd0392efea2c3c615/userdata/conmon.pid
Type=forking

[Install]
WantedBy=default.target

systemd log:

Jan 16 08:38:06 archlinux systemd[814]: Starting Podman pod-alpine_pod.service...
Jan 16 08:38:06 archlinux systemd[814]: Started podman-7145.scope.
Jan 16 08:38:06 archlinux systemd[814]: Started libcrun container.
Jan 16 08:38:06 archlinux podman[7145]: 2023-01-16 08:38:06.844104438 +0800 CST m=+0.140241590 container init b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, pod_id=4e6aac60a5cc653c56bbc1dcb65167b52f99403b33b4dc5a0a22690412a8bd52, io.buildah.version=1.28.0)
Jan 16 08:38:06 archlinux podman[7145]: 2023-01-16 08:38:06.850811508 +0800 CST m=+0.146948660 container start b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, pod_id=4e6aac60a5cc653c56bbc1dcb65167b52f99403b33b4dc5a0a22690412a8bd52, io.buildah.version=1.28.0)
Jan 16 08:38:06 archlinux podman[7145]: 4e6aac60a5cc-infra
Jan 16 08:38:06 archlinux systemd[814]: pod-alpine_pod.service: New main PID 7165 does not belong to service, and PID file is not owned by root. Refusing.
Jan 16 08:38:06 archlinux systemd[814]: pod-alpine_pod.service: New main PID 7165 does not belong to service, and PID file is not owned by root. Refusing.
Jan 16 08:38:06 archlinux systemd[814]: Started podman-7169.scope.
Jan 16 08:38:07 archlinux systemd[814]: Started podman-7180.scope.
Jan 16 08:38:07 archlinux podman[7180]: 2023-01-16 08:38:07.020440843 +0800 CST m=+0.073514537 container died b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, io.buildah.version=1.28.0)
Jan 16 08:38:07 archlinux podman[7180]: 2023-01-16 08:38:07.046541596 +0800 CST m=+0.099615267 container cleanup b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, pod_id=4e6aac60a5cc653c56bbc1dcb65167b52f99403b33b4dc5a0a22690412a8bd52, io.buildah.version=1.28.0)
Jan 16 08:38:07 archlinux podman[7169]: 2023-01-16 08:38:07.048283086 +0800 CST m=+0.187521694 container cleanup b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, pod_id=4e6aac60a5cc653c56bbc1dcb65167b52f99403b33b4dc5a0a22690412a8bd52, io.buildah.version=1.28.0)
Jan 16 08:38:07 archlinux podman[7169]: 4e6aac60a5cc-infra
Jan 16 08:38:07 archlinux systemd[814]: pod-alpine_pod.service: Failed with result 'protocol'.
Jan 16 08:38:07 archlinux systemd[814]: Failed to start Podman pod-alpine_pod.service.
Jan 16 08:38:07 archlinux systemd[814]: Dependency failed for Podman container-wizardly_hofstadter.service.
Jan 16 08:38:07 archlinux systemd[814]: container-wizardly_hofstadter.service: Job container-wizardly_hofstadter.service/start failed with result 'dependency'.

I removed the PIDFile line in .service files but the infra container still died:

Jan 16 08:50:40 archlinux systemd[814]: Starting Podman pod-alpine_pod.service...
Jan 16 08:50:40 archlinux systemd[814]: Started podman-7666.scope.
Jan 16 08:50:40 archlinux systemd[814]: Started libcrun container.
Jan 16 08:50:40 archlinux podman[7666]: 2023-01-16 08:50:40.869940054 +0800 CST m=+0.095698022 container init b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, pod_id=4e6aac60a5cc653c56bbc1dcb65167b52f99403b33b4dc5a0a22690412a8bd52, io.buildah.version=1.28.0)
Jan 16 08:50:40 archlinux podman[7666]: 2023-01-16 08:50:40.877032954 +0800 CST m=+0.102790921 container start b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, pod_id=4e6aac60a5cc653c56bbc1dcb65167b52f99403b33b4dc5a0a22690412a8bd52, io.buildah.version=1.28.0)
Jan 16 08:50:40 archlinux podman[7666]: 4e6aac60a5cc-infra
Jan 16 08:50:40 archlinux systemd[814]: Started podman-7690.scope.
Jan 16 08:50:41 archlinux systemd[814]: Started podman-7702.scope.
Jan 16 08:50:41 archlinux podman[7702]: 2023-01-16 08:50:41.03949197 +0800 CST m=+0.061724197 container died b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, io.buildah.version=1.28.0)
Jan 16 08:50:41 archlinux podman[7702]: 2023-01-16 08:50:41.069971396 +0800 CST m=+0.092203633 container cleanup b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, pod_id=4e6aac60a5cc653c56bbc1dcb65167b52f99403b33b4dc5a0a22690412a8bd52, io.buildah.version=1.28.0)
Jan 16 08:50:41 archlinux podman[7690]: 2023-01-16 08:50:41.073052486 +0800 CST m=+0.186800090 container cleanup b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, pod_id=4e6aac60a5cc653c56bbc1dcb65167b52f99403b33b4dc5a0a22690412a8bd52, io.buildah.version=1.28.0)
Jan 16 08:50:41 archlinux podman[7690]: 4e6aac60a5cc-infra
Jan 16 08:50:41 archlinux systemd[814]: Started podman-7715.scope.
Jan 16 08:50:41 archlinux podman[7715]: 2023-01-16 08:50:41.168080301 +0800 CST m=+0.063809211 container cleanup b3200ec533d9687ce7b2b68f7e41c9a739d30a427ebacd151c427f642079d2a2 (image=localhost/podman-pause:4.3.1-1668983565, name=4e6aac60a5cc-infra, pod_id=4e6aac60a5cc653c56bbc1dcb65167b52f99403b33b4dc5a0a22690412a8bd52, io.buildah.version=1.28.0)
Jan 16 08:50:41 archlinux podman[7715]: 4e6aac60a5cc-infra
Jan 16 08:50:41 archlinux systemd[814]: Started Podman pod-alpine_pod.service.
Jan 16 08:50:41 archlinux systemd[814]: container-wizardly_hofstadter.service: Bound to unit pod-alpine_pod.service, but unit isn't active.
Jan 16 08:50:41 archlinux systemd[814]: Dependency failed for Podman container-wizardly_hofstadter.service.
Jan 16 08:50:41 archlinux systemd[814]: container-wizardly_hofstadter.service: Job container-wizardly_hofstadter.service/start failed with result 'dependency'.

podman info:

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.5-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: c9f7f19eb82d5b8151fc3ba7fbbccf03fdcd0325'
  cpuUtilization:
    idlePercent: 99.2
    systemPercent: 0.23
    userPercent: 0.57
  cpus: 4
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: archlinux
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.6-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 6254284800
  memTotal: 8233705472
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.7.2-1
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 4116705280
  swapTotal: 4116705280
  uptime: 1h 59m 47.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
    PullFromMirror: ""
  search:
  - docker.io
store:
  configFile: /home/lihua/.config/containers/storage.conf
  containerStore:
    number: 10
    paused: 0
    running: 0
    stopped: 10
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.10-1
      Version: |-
        fusermount3 version: 3.13.0
        fuse-overlayfs: version 1.10
        FUSE library version 3.13.0
        using FUSE kernel interface version 7.38
  graphRoot: /home/lihua/.local/share/containers/storage
  graphRootAllocated: 244452069376
  graphRootUsed: 122524651520
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 8
  runRoot: /run/user/1000/containers
  volumePath: /home/lihua/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668983565
  BuiltTime: Mon Nov 21 06:32:45 2022
  GitCommit: 814b7b003cc630bf6ab188274706c383f9fb9915-dirty
    stopped: 10
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.10-1
      Version: |-
        fusermount3 version: 3.13.0
        fuse-overlayfs: version 1.10
        FUSE library version 3.13.0
        using FUSE kernel interface version 7.38
  graphRoot: /home/lihua/.local/share/containers/storage
  graphRootAllocated: 244452069376
  graphRootUsed: 122524651520
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 8
  runRoot: /run/user/1000/containers
  volumePath: /home/lihua/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668983565
  BuiltTime: Mon Nov 21 06:32:45 2022
  GitCommit: 814b7b003cc630bf6ab188274706c383f9fb9915-dirty
  GoVersion: go1.19.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

vrothberg · 2023-01-16T09:37:07Z

vrothberg
Jan 16, 2023
Maintainer

Thanks for reaching out. What is the expected behavior? I can start the service without issues. Note that altering the generated files isn't supported.

4 replies

xxyzz Jan 16, 2023
Author

systemd log shows the New main PID 7165 does not belong to service, and PID file is not owned by root. Refusing. error message for the pod service, then the pod service failed to start. I was expecting the pod service could start and all containers running.

I could run podman pod start alpine_pod to start the pod but I want to enable the systemd service so it can start at boot.

vrothberg Jan 16, 2023
Maintainer

It works without issues on Fedora. Why does systemd expect the PID file to be owned by root? That doesn't make sense for a rootless service.

vrothberg Jan 16, 2023
Maintainer

Could it be a systemd bug in Arch Linux?

xxyzz Jan 17, 2023
Author

I guess it's because the pid in file conmon.id is not the service main pid: https://github.com/systemd/systemd/blob/841dfd3dc0dd370a21f190a5b7b870db1c95f7e6/src/core/service.c#L1024-L1044

And the conmon.id file in the systemd service file doesn't exist.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Can't start rootless pod systemd service #17123

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Can't start rootless pod systemd service #17123

Uh oh!

Uh oh!

xxyzz Jan 16, 2023

Replies: 1 comment · 4 replies

Uh oh!

vrothberg Jan 16, 2023 Maintainer

Uh oh!

xxyzz Jan 16, 2023 Author

Uh oh!

vrothberg Jan 16, 2023 Maintainer

Uh oh!

vrothberg Jan 16, 2023 Maintainer

Uh oh!

Uh oh!

xxyzz Jan 17, 2023 Author

xxyzz
Jan 16, 2023

Replies: 1 comment 4 replies

vrothberg
Jan 16, 2023
Maintainer

xxyzz Jan 16, 2023
Author

vrothberg Jan 16, 2023
Maintainer

vrothberg Jan 16, 2023
Maintainer

xxyzz Jan 17, 2023
Author