Stopping a service which has previously spawned Podman kills the containers

[swilton86]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Firstly, sorry if this is actually a bug in my configuration, systemd, Node.js or something else.

My setup is as follows:

• Docker Registry is running as a rootless Podman container (user www-data).
  • Created with --net slirp4netns:allow_host_loopback=true.
• Node.js HTTP server acting as a webhook target for Docker Registry.
  • This runs on the host system (user www-data), managed by systemd (e.g., systemctl --user start webhook.service).
  • When an application container image is pushed to the registry, the webhook application spawns Podman processes a few times to:
    • Pull the image to the host.
    • Stop the running application container.
    • Create and start a new application container.

Expected Result

When I stop the webhook application on the host the containerized applications are unaffected. The webhook application process has no children and therefore killing it kills no other processes.

Actual Result

When I stop the webhook service on the host several other processes are stopped:

• fuse-overlayfs
• slirp4netns
• rootlessport
• rootlessport-child
• conmon
• containerized application

This takes the containerized application offline and means further Podman commands fail with:

Error: container <id> conmon process missing, cannot retrieve exit code: conmon process killed

As expected, Podman starts and later exits when spawned by Node.js. The processes above are not children of the webhook service process (node), but are in the same cgroup.

Speculation

As far as I can tell, one of the following is true:

• Podman does not create a new cgroup for a container when starting it, but should do.
• Podman is unable to create a new cgroup for a container when started in a cgroup.
• systemd kills all processes in a service's cgroup when stopping the service, but shouldn't do.
• Node.js's spawn creates Podman processes in such a way that they are unable to create a new cgroup for a new container.

Versions

• podman version (built from source)
  Client: Podman Engine
  Version: 4.0.0-dev
  API Version: 4.0.0-dev
  Go Version: go1.17.6 (from Debian testing)
  Git Commit: ab4af50
  Built: Sun Feb 6 22:11:31 2022
  OS/Arch: linux/amd64

• runc --version (from Debian testing)
  runc version 1.0.3+ds1
  commit: 1.0.3+ds1-1
  spec: 1.0.2-dev
  go: go1.17.4
  libseccomp: 2.5.1

• crun --version (from Debian stable)
  crun version 0.17
  commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
  spec: 1.0.0
  +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

• systemd --version
  systemd 247 (247.3-6)
  +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified

• node --version
  v16.13.2

• podman info --debug

  host:
    arch: amd64
    buildahVersion: 1.24.1
    cgroupControllers:
    - cpuset
    - cpu
    - io
    - memory
    - hugetlb
    - pids
    - rdma
    cgroupManager: cgroupfs
    cgroupVersion: v2
    conmon:
      package: 'conmon: /usr/bin/conmon'
      path: /usr/bin/conmon
      version: 'conmon version 2.0.25, commit: unknown'
    cpus: 1
    distribution:
      codename: bullseye
      distribution: debian
      version: "11"
    eventLogger: file
    hostname: <removed>
    idMappings:
      gidmap: null
      uidmap: null
    kernel: 5.10.0-11-amd64
    linkmode: dynamic
    logDriver: k8s-file
    memFree: 70627328
    memTotal: 1023930368
    networkBackend: cni
    ociRuntime:
      name: crun
      package: 'crun: /usr/bin/crun'
      path: /usr/bin/crun
      version: |-
        crun version 0.17
        commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
        spec: 1.0.0
        +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
    os: linux
    remoteSocket:
      exists: true
      path: /run/podman/podman.sock
    security:
      apparmorEnabled: false
      capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
      rootless: false
      seccompEnabled: true
      seccompProfilePath: /usr/share/containers/seccomp.json
      selinuxEnabled: false
    serviceIsRemote: false
    slirp4netns:
      executable: /usr/bin/slirp4netns
      package: 'slirp4netns: /usr/bin/slirp4netns'
      version: |-
        slirp4netns version 1.0.1
        commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
        libslirp: 4.4.0
    swapFree: 448638976
    swapTotal: 536866816
    uptime: 115h 42m 34.73s (Approximately 4.79 days)
  plugins:
    log:
    - k8s-file
    - none
    - passthrough
    network:
    - bridge
    - macvlan
    - ipvlan
    volume:
    - local
  registries:
    localhost:10100:
      Blocked: false
      Insecure: true
      Location: localhost:10100
      MirrorByDigestOnly: false
      Mirrors: null
      Prefix: localhost:10100
  store:
    configFile: /usr/share/containers/storage.conf
    containerStore:
      number: 0
      paused: 0
      running: 0
      stopped: 0
    graphDriverName: overlay
    graphOptions: {}
    graphRoot: /var/lib/containers/storage
    graphStatus:
      Backing Filesystem: extfs
      Native Overlay Diff: "true"
      Supports d_type: "true"
      Using metacopy: "false"
    imageCopyTmpDir: /var/tmp
    imageStore:
      number: 0
    runRoot: /run/containers/storage
    volumePath: /var/lib/containers/storage/volumes
  version:
    APIVersion: 4.0.0-dev
    Built: 1644185491
    BuiltTime: Sun Feb  6 22:11:31 2022
    GitCommit: ab4af502b3f60f891192356eddaa13092f785612
    GoVersion: go1.17.6
    OsArch: linux/amd64
    Version: 4.0.0-dev

• Package info
  Podman not installed via package manager.

[giuseppe]

do you have any configuration that overrides the cgroup manager?

From the message above I see cgroupManager: cgroupfs so Podman doesn't create a new systemd scope.

The cgroupManager should be set to systemd

[swilton86]

Thanks Giuseppe. I haven't chosen cgroupfs intentionally.

grep cgroup /proc/filesystems lists cgroup and cgroup2, so I think kernel support is OK.

Issue #5443 mentions systemd linger. I do have that enabled for www-data. That said, I haven't seen any fallback warning messages in the system log.

I'm also logging in over SSH, not by switching user with su. systemctl --user commands do work, although they did not work when I used su before, so I think I have a proper login session ($DBUS_SESSION_BUS_ADDRESS is set).

/etc/containers/libpod.conf has the line cgroup_manager = "systemd". Is this file used by rootless Podman processes as well as rootfull? I don't have a ~/.config/containers/libpod.conf file that may override it. Does podman info --debug just print config that it has loaded from it's various config files, i.e., is there a config file that I'm missing?

Edit: podman --cgroup-manager=systemd info does show cgroupManager: systemd. I'll try recreating the containers while passing this flag to all podman commands.

Edit: If I create and start the container while passing --cgroup-manager=systemd info to each command, the output of podman inspect <container> does include "CgroupManager": "systemd",. That said, the output of systemctl --user status for the webhook application is the same and the new processes are still created in the same cgroup. Stopping the service stops conmon, the containerized process, etc.

[giuseppe]

do you have /etc/containers/containers.conf or ~/.config/containers/containers.conf?

[swilton86]

I didn't have one, no. Does this file replace libpod.conf in 4.x? I had podman 3.4.4 installed before via APT and the uninstallation process may not have removed everything.

I downloaded /vendor/github.com/containers/common/pkg/config/containers.conf to /etc/containers. podman info was still showing cgroupManager: cgroupfs, so I modified the containers.conf to explicitly use systemd. I also renamed podman.conf in case that was interfering. podman info does now show cgroupManager: systemd.

I removed and recreated all of my containers. systemctl status no longer shows the containerized process in the cgroup of the webhook service, but conmon, rootlessport, etc. are still in the same cgroup. Stopping the agent still kills the container.

[giuseppe]

what is the systemd version you are using?

[swilton86]

systemd --version
systemd 247 (247.3-6)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified

I think that release was around February or March last year.

I tried upgrading to systemd 250 (from Debian bullseye-backports) and rebooting:

systemd --version
systemd 250 (250.3-2~bpo11+1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

I now have two separate issues:

• The webhook application now fails to start the new container after creating it:

  Error: OCI runtime error: unable to start container "c83c38e9964e7446509e4499a72105ffde55b4c2c729b8ea3e1bdaac7f059610": runc: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF
  

• I can create and start containers manually, but network connections into the container time out. I've observed this with my own containerized application and Docker Registry. No warnings are printed when running the Podman commands. podman container stop also doesn't seem to work – I had to kill the processes manually.

  I tried starting the container with --log-level=debug. I get the following output. Is "vfs" the usual driver? Should I be using runc or should I make sure Podman can find crun?

  INFO[0000] podman filtering at log level debug
  DEBU[0000] Called start.PersistentPreRunE(podman --log-level=debug container start application-new)
  DEBU[0000] Cached value indicated that overlay is not supported
  DEBU[0000] Merged system config "/etc/containers/containers.conf"
  DEBU[0000] Cached value indicated that overlay is not supported
  DEBU[0000] Using conmon: "/usr/bin/conmon"
  DEBU[0000] Initializing boltdb state at /srv/.local/share/containers/storage/libpod/bolt_state.db
  DEBU[0000] systemd-logind: Unknown object '/'.
  DEBU[0000] Using graph driver vfs
  DEBU[0000] Using graph root /srv/.local/share/containers/storage
  DEBU[0000] Using run root /run/user/33/containers
  DEBU[0000] Using static dir /srv/.local/share/containers/storage/libpod
  DEBU[0000] Using tmp dir /run/user/33/libpod/tmp
  DEBU[0000] Using volume path /srv/.local/share/containers/storage/volumes
  DEBU[0000] Cached value indicated that overlay is not supported
  DEBU[0000] Set libpod namespace to ""
  DEBU[0000] [graphdriver] trying provided driver "vfs"
  DEBU[0000] Initializing event backend file
  DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
  DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument
  DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
  DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
  DEBU[0000] Using OCI runtime "/usr/bin/runc"
  INFO[0000] Setting parallel job count to 4
  DEBU[0000] [graphdriver] trying provided driver "vfs"
  DEBU[0000] Mounted container "76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514" at "/srv/.local/share/containers/storage/vfs/dir/6cc0816bb9b2961ec70e6820d787cecef5e493fe5868b73fe6b6fa212fb58777"
  DEBU[0000] Created root filesystem for container 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514 at /srv/.local/share/containers/storage/vfs/dir/6cc0816bb9b2961ec70e6820d787cecef5e493fe5868b73fe6b6fa212fb58777
  DEBU[0000] Made network namespace at /run/user/33/netns/netns-f9e59851-316a-5225-7605-c9a6f3a5b8ed for container 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514
  DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/33/netns/netns-f9e59851-316a-5225-7605-c9a6f3a5b8ed tap0
  DEBU[0000] rootlessport: time="2022-02-09T18:37:44Z" level=info msg="Starting parent driver"
  time="2022-02-09T18:37:44Z" level=info msg="opaque=map[builtin.readypipepath:/run/user/33/libpod/tmp/rootlessport2738659640/.bp-ready.pipe builtin.socketpath:/run/user/33/libpod/tmp/rootlessport2738659640/.bp.sock]"
  time="2022-02-09T18:37:44Z" level=info msg="Starting child driver in child netns (\"/proc/self/exe\" [rootlessport-child])"
  DEBU[0000] rootlessport: time="2022-02-09T18:37:44Z" level=info msg="Waiting for initComplete"
  DEBU[0000] rootlessport is ready
  DEBU[0000] Not modifying container 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514 /etc/passwd
  DEBU[0000] Not modifying container 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514 /etc/group
  DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
  DEBU[0000] Setting Cgroups for container 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514 to user.slice:libpod:76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514
  DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
  DEBU[0000] Workdir "/application/" resolved to host path "/srv/.local/share/containers/storage/vfs/dir/6cc0816bb9b2961ec70e6820d787cecef5e493fe5868b73fe6b6fa212fb58777/application"
  DEBU[0000] Created OCI spec for container 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514 at /srv/.local/share/containers/storage/vfs-containers/76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514/userdata/config.json
  DEBU[0000] /usr/bin/conmon messages will be logged to syslog
  DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514 -u 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514 -r /usr/bin/runc -b /srv/.local/share/containers/storage/vfs-containers/76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514/userdata -p /run/user/33/containers/vfs-containers/76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514/userdata/pidfile -n application-new --exit-dir /run/user/33/libpod/tmp/exits --full-attach -s -l k8s-file:/srv/.local/share/containers/storage/vfs-containers/76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/33/containers/vfs-containers/76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /srv/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/33/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/33/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg vfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514]"
  INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514.scope
  [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
  
  DEBU[0000] Received: 2064
  INFO[0000] Got Conmon PID as 2049
  DEBU[0000] Created container 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514 in OCI runtime
  DEBU[0000] Starting container 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514 with command [/entrypoint.sh]
  DEBU[0000] Started container 76710e0f263f92c3ef16fb2f75fb0e452e276b6524eadb20edf5d65a0d40f514
  application-new
  DEBU[0000] Called start.PersistentPostRunE(podman --log-level=debug container start application-new)
  

[giuseppe]

vfs is very slow. Probably your kernel doesn't support native overlay, in this case I suggest to install fuse-overlayfs.

Not sure if crun would help here, but we can give it a try.

If you run the container with crun (podman --runtime crun run ....) do you see any difference?

[swilton86]

• I must have accidentally uninstalled fuse-overlayfs. I've re-installed and Podman picks it up.
• I also specified crun in /etc/containers/containers.conf. Podman also picks that up
• I cleared out all CRI-related iptables rules, and now network traffic is reaching containers correctly again.

The issuue persists, and podman info --debug now looks like this:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 1
  distribution:
    codename: bullseye
    distribution: debian
    version: "11"
  eventLogger: file
  hostname: <removed>
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 33
      size: 1
    - container_id: 1
      host_id: 951968
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 33
      size: 1
    - container_id: 1
      host_id: 951968
      size: 65536
  kernel: 5.10.0-11-amd64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 111452160
  memTotal: 1023930368
  networkBackend: cni
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/33/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.4.0
  swapFree: 487596032
  swapTotal: 536866816
  uptime: 66h 14m 26.77s (Approximately 2.75 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:10100:
    Blocked: false
    Insecure: true
    Location: localhost:10100
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:10100
store:
  configFile: /srv/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.10.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /srv/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 16
  runRoot: /run/user/33/containers
  volumePath: /srv/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.0-dev
  Built: 1644185491
  BuiltTime: Sun Feb  6 22:11:31 2022
  GitCommit: ab4af502b3f60f891192356eddaa13092f785612
  GoVersion: go1.17.6
  OsArch: linux/amd64
  Version: 4.0.0-dev

Sorry for the long comment, but hopefully this helps.

Below are three outputs from systemctl --user status. The first is after login, the second after starting the Registry container manually, and the third is starting an application container with the webhook application.

Notice that the Registry container processes are created in user.slice/podman-13704.scope but the application container processes are created in the same cgroup as the webhook application, app.slice/webhook.service. I've tried setting the Slice property in the systemd unit file for the webhook service but Podman still creates processes in the same cgroup as the webhook process. Presumably disabling systemd's behavior of creating a cgroup for every service would get around the problem, but I can't find an option to do that. Is it even possible for Podman to "escape" the webhook application's cgroup without being privileged?

This still looks like a bug in Podman or at least a limitation of rootless containers.

• Login:

  ● <removed>
      State: running
      Jobs: 0 queued
    Failed: 0 units
      Since: Fri 2022-02-11 03:52:14 GMT; 2 days ago
    CGroup: /user.slice/user-33.slice/[email protected]
            ├─user.slice
            │ └─podman-pause-0a5990e6.scope
            │   └─1093 podman
            ├─app.slice
            │ ├─webhook.service
            │ │ └─13658 node /srv/build/webhook/index.js
            │ └─dbus.service
            │   └─1099 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
            └─init.scope
              ├─667 /lib/systemd/systemd --user
              └─674 (sd-pam)

• Start registry:

  ● <removed>
      State: running
      Jobs: 0 queued
    Failed: 0 units
      Since: Fri 2022-02-11 03:52:14 GMT; 2 days ago
    CGroup: /user.slice/user-33.slice/[email protected]
            ├─user.slice
            │ ├─podman-13704.scope
            │ │ ├─13713 /usr/bin/fuse-overlayfs -o ,lowerdir=/srv/.local/share/containers/storage/overlay/l/MLRGACZPBNW3XMZ7LO6Q6OI6J4:/srv/.local/share/containers/storage/overlay/l/QK22V5CQYOPRRJVDV32PGT5JYP:/srv/.local/share/containers/storage/overlay/l/JJTKQQUZ2ZUJSYCDDIC6FVI66S:/srv/.local/share/containers/storage/overlay/l/G3KQCBZSJKEKEQBR2R2OPFXADN:/srv/.local/share/containers/storage/overlay/l/NNELXSPXLLQLTV5MJ7A4UFHB2G,upperdir=/srv/.local/share/containers/storage/overlay/4ed1ddac7648fac91eaad9376b7f2518c38620d3f56679e6cc1f2423e7684d38/diff,workdir=/srv/.local/share/containers/storage/overlay/4ed1ddac7648fac91eaad9376b7f2518c38620d3f56679e6cc1f2423e7684d38/work /srv/.local/share/containers/storage/overlay/4ed1ddac7648fac91eaad9376b7f2518c38620d3f56679e6cc1f2423e7684d38/merged
            │ │ ├─13716 /usr/bin/slirp4netns --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/33/netns/netns-e4bf6f2e-2299-2ab8-9e44-b2c7c4244f53 tap0
            │ │ ├─13719 rootlessport
            │ │ └─13725 rootlessport-child
            │ ├─podman-pause-0a5990e6.scope
            │ │ └─1093 podman
            │ ├─libpod-conmon-e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0.scope
            │ │ └─13732 /usr/bin/conmon --api-version 1 -c e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0 -u e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0 -r /usr/bin/crun -b /srv/.local/share/containers/storage/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata -p /run/user/33/containers/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata/pidfile -n registry --exit-dir /run/user/33/libpod/tmp/exits --full-attach -s -l k8s-file:/srv/.local/share/containers/storage/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata/ctr.log --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/33/containers/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata/oci-log --conmon-pidfile /run/user/33/containers/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /srv/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/33/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/33/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0
            │ └─libpod-e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0.scope
            │   └─container
            │     └─13736 registry serve /etc/docker/registry/config.yml
            ├─app.slice
            │ ├─webhook.service
            │ │ └─13658 node /srv/build/webhook/index.js
            │ └─dbus.service
            │   └─1099 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
            └─init.scope
              ├─667 /lib/systemd/systemd --user
              └─674 (sd-pam)

• Create and start container with webhook application:

  ● <removed>
      State: running
      Jobs: 0 queued
    Failed: 0 units
      Since: Fri 2022-02-11 03:52:14 GMT; 2 days ago
    CGroup: /user.slice/user-33.slice/[email protected]
            ├─user.slice
            │ ├─podman-13704.scope
            │ │ ├─13713 /usr/bin/fuse-overlayfs -o ,lowerdir=/srv/.local/share/containers/storage/overlay/l/MLRGACZPBNW3XMZ7LO6Q6OI6J4:/srv/.local/share/containers/storage/overlay/l/QK22V5CQYOPRRJVDV32PGT5JYP:/srv/.local/share/containers/storage/overlay/l/JJTKQQUZ2ZUJSYCDDIC6FVI66S:/srv/.local/share/containers/storage/overlay/l/G3KQCBZSJKEKEQBR2R2OPFXADN:/srv/.local/share/containers/storage/overlay/l/NNELXSPXLLQLTV5MJ7A4UFHB2G,upperdir=/srv/.local/share/containers/storage/overlay/4ed1ddac7648fac91eaad9376b7f2518c38620d3f56679e6cc1f2423e7684d38/diff,workdir=/srv/.local/share/containers/storage/overlay/4ed1ddac7648fac91eaad9376b7f2518c38620d3f56679e6cc1f2423e7684d38/work /srv/.local/share/containers/storage/overlay/4ed1ddac7648fac91eaad9376b7f2518c38620d3f56679e6cc1f2423e7684d38/merged
            │ │ ├─13716 /usr/bin/slirp4netns --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/33/netns/netns-e4bf6f2e-2299-2ab8-9e44-b2c7c4244f53 tap0
            │ │ ├─13719 rootlessport
            │ │ └─13725 rootlessport-child
            │ ├─libpod-fcaa3f87e65b5b7f69b4086d4b0411fa32697123e9478d7c9af9aa84308d5a9e.scope
            │ │ └─container
            │ │   └─13838 /usr/bin/node /<removed>/src/server.js
            │ ├─podman-pause-0a5990e6.scope
            │ │ └─1093 podman
            │ ├─libpod-conmon-e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0.scope
            │ │ └─13732 /usr/bin/conmon --api-version 1 -c e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0 -u e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0 -r /usr/bin/crun -b /srv/.local/share/containers/storage/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata -p /run/user/33/containers/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata/pidfile -n registry --exit-dir /run/user/33/libpod/tmp/exits --full-attach -s -l k8s-file:/srv/.local/share/containers/storage/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata/ctr.log --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/33/containers/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata/oci-log --conmon-pidfile /run/user/33/containers/overlay-containers/e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /srv/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/33/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/33/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0
            │ └─libpod-e532dc657efcbae30a5de679a88668c167e7f7dd7f0cd695fee3d6d3408c2ac0.scope
            │   └─container
            │     └─13736 registry serve /etc/docker/registry/config.yml
            ├─app.slice
            │ ├─webhook.service
            │ │ ├─13658 node /srv/build/webhook/index.js
            │ │ ├─13816 /usr/bin/fuse-overlayfs -o ,lowerdir=/srv/.local/share/containers/storage/overlay/l/QN232BZ2FDSM6FQOE2PV7QL6O7:/srv/.local/share/containers/storage/overlay/l/MMYOJCKFH7VXW5VYGEUDBWTTCT:/srv/.local/share/containers/storage/overlay/l/BSU4XLXRUPKO46XRWUNMONJLV7:/srv/.local/share/containers/storage/overlay/l/U65ZTFFNNXCP4RIC6OBYDJQV6E:/srv/.local/share/containers/storage/overlay/l/NNELXSPXLLQLTV5MJ7A4UFHB2G,upperdir=/srv/.local/share/containers/storage/overlay/b3801d9c686e7c427333a260fa99782e0eec0e4708b91e0377b84767bc436c67/diff,workdir=/srv/.local/share/containers/storage/overlay/b3801d9c686e7c427333a260fa99782e0eec0e4708b91e0377b84767bc436c67/work /srv/.local/share/containers/storage/overlay/b3801d9c686e7c427333a260fa99782e0eec0e4708b91e0377b84767bc436c67/merged
            │ │ ├─13820 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/33/netns/netns-7357cd90-4302-ae7b-acd6-c733a4896a91 tap0
            │ │ ├─13822 rootlessport
            │ │ ├─13827 rootlessport-child
            │ │ └─13834 /usr/bin/conmon --api-version 1 -c fcaa3f87e65b5b7f69b4086d4b0411fa32697123e9478d7c9af9aa84308d5a9e -u fcaa3f87e65b5b7f69b4086d4b0411fa32697123e9478d7c9af9aa84308d5a9e -r /usr/bin/crun -b /srv/.local/share/containers/storage/overlay-containers/fcaa3f87e65b5b7f69b4086d4b0411fa32697123e9478d7c9af9aa84308d5a9e/userdata -p /run/user/33/containers/overlay-containers/fcaa3f87e65b5b7f69b4086d4b0411fa32697123e9478d7c9af9aa84308d5a9e/userdata/pidfile -n <removed>-new --exit-dir /run/user/33/libpod/tmp/exits --full-attach -s -l k8s-file:/srv/.local/share/containers/storage/overlay-containers/fcaa3f87e65b5b7f69b4086d4b0411fa32697123e9478d7c9af9aa84308d5a9e/userdata/ctr.log --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/33/containers/overlay-containers/fcaa3f87e65b5b7f69b4086d4b0411fa32697123e9478d7c9af9aa84308d5a9e/userdata/oci-log --conmon-pidfile /run/user/33/containers/overlay-containers/fcaa3f87e65b5b7f69b4086d4b0411fa32697123e9478d7c9af9aa84308d5a9e/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /srv/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/33/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/33/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg fcaa3f87e65b5b7f69b4086d4b0411fa32697123e9478d7c9af9aa84308d5a9e
            │ └─dbus.service
            │   └─1099 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
            └─init.scope
              ├─667 /lib/systemd/systemd --user
              └─674 (sd-pam)

[swilton86]

Possibly related: rootless-containers/rootlesscontaine.rs#32

[swilton86]

@giuseppe Sorry to bump, but do you have any other ideas? It's not a deal breaker for me, but I think there is a bug here.

[giuseppe]

I think that what is happening is that Podman automatically creates a new cgroup and moves itself there if the user doesn't own the current cgroup.

In your case though, since the cgroup is owned, Podman doesn't create a new cgroup (correctly!) and all the auxiliary processes are created there. You could get rid of fuse-overlayfs by moving to native overlay, but I think the issue would still persist with slirp4netns

[rhatdan]

Since this is a configuration question not an issue with Podman, moving to discussion.