(Starting in 4.8.0 - regression from 4.7.2) Quadlet fails to resolve user provided by FreeIPA in system quadlet

[mhjacks]

Issue Description

In Podman <= 4.7.2, having a User=podman stanza in my X-Container section for quadlet resolved the userid correctly and the container ran. Since the upgrade to podman 4.8.0 on December 1, I get the following error in my logs:
Dec 01 08:20:20 srv-hv-4.imladris.lan arm[964]: Error: unable to find user podman: no matching entries in passwd file. The exact ID setup worked previously with podman 4.7.2 and earlier versions.

Steps to reproduce the issue

Steps to reproduce the issue

1. Set up a user in freeipa, and add that user to /etc/group groups as well
2. Set up a quadlet to run (as root) specifying that user name
3. Start quadlet; observe errors in log

Describe the results you received

container fails to run on start, for example (this error seems to be coming from podman): Dec 01 08:20:20 srv-hv-4.imladris.lan arm[964]: Error: unable to find user podman: no matching entries in passwd file

Describe the results you expected

arm container starts and runs

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 97.63
    systemPercent: 0.34
    userPercent: 2.02
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: server
    version: "39"
  eventLogger: journald
  freeLocks: 2048
  hostname: srv-hv-4.imladris.lan
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.6.4-200.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 14414995456
  memTotal: 67348897792
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.12-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231119.g4f1709d-1.fc39.x86_64
    version: |
      pasta 0^20231119.g4f1709d-1.fc39.x86_64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 99h 12m 31.00s (Approximately 4.12 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1998694907904
  graphRootUsed: 117711495168
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 9
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.8.1
  Built: 1701777650
  BuiltTime: Tue Dec  5 06:00:50 2023
  GitCommit: ""
  GoVersion: go1.21.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

My quadlet:

Unit]
Description=Automatic Ripping Machine

[Install]
WantedBy=default.target

[Container]
Image=docker.io/automaticrippingmachine/automatic-ripping-machine
ContainerName=arm
PublishPort=8080
User=podman
Environment=ARM_UID=344800018
Environment=ARM_GID=344800018
Volume=/btrfs/sixtb-int/arm-home:/home/arm
Volume=/btrfs/sixtb-int/arm-home/Music:/home/arm/Music
Volume=/btrfs/sixtb-int/arm-home/logs:/home/arm/logs
Volume=/btrfs/sixtb-int/arm-home/media:/home/arm/media
Volume=/btrfs/sixtb-int/arm-home/config:/etc/arm/config
AddDevice=/dev/sr0
Network=host
Pull=newer
PodmanArgs=--privileged

[Service]
Restart=always
KillMode=control-group
TimeoutStopSec=10

[ygalblum]

@mhjacks thanks for reporting this.

I deleted my previous comment since it was wrong

[ygalblum]

I ran the .container file you provided with both v4.8.0 and v4.7.2 and found what changed.

In v4.7.2, Quadlet expected the value of the User key to be an unsigned integer and when it wasn't, it used a default value of 0 resulting in adding --user 0 to the podman run command.

In v4.8.0, Quadlet was changed to support any value as part of #20395. This resulted in adding --user podman to the podman run command.

Now, I'm not sure v4.7.2 worked as you expected either, did it?

[mhjacks]

It certainly worked before 4.8.0 (e.g. for 4.6.x for sure), but I don't remember whether it worked on 4.7.2 specifically.

When I specify the UID in the quadlet (344800018 in my case) the container starts, but I get a different error from inside the container:

Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711757]: 4e9d2276504ec0dbf44e4ab69bc4e95665bde1346a34f433da9e398fdcd9a913
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]: *** Killing all processes...
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]: Traceback (most recent call last):
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]:   File "/sbin/my_init", line 414, in <module>
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]:     main(args)
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]:   File "/sbin/my_init", line 330, in main
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]:     import_envvars(False, False)
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]:   File "/sbin/my_init", line 90, in import_envvars
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]:     for envfile in listdir("/etc/container_environment"):
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]:   File "/sbin/my_init", line 74, in listdir
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]:     return sorted(os.listdir(path))
Dec 12 08:58:06 srv-hv-4.imladris.lan arm[711775]: PermissionError: [Errno 13] Permission denied: '/etc/container_environment'

Previously, the various hostdirs that were mounted worked fine - I was able to write files to them, and they were owned by podman as I expected them to be. I wasn't looking too closely at the actual commmand line args the container was started with. This container is a little odd because it requires extra permissions to access video acceleration and physical media, but it definitely worked before.

[baude]

@ygalblum depending on the feedback from @mhjacks , can I assume you will handle this one given it looks like you did the legwork already ?

[ygalblum]

@baude I don't know yet since I'm still not sure if this is a Quadlet or Podman issue. I can keep track of it for now until we have a better view on where the issue is.

[mhjacks]

I should mention - I get the same errrors with podman run as I do with quadlet, so this does not appear to be quadlet-specific. From the command line:

/usr/bin/podman run --name=arm --replace --rm --cgroups=split --network=host --sdnotify=conmon -d --device=/dev/sr0 --user 344800018 -v /btrfs/sixtb-int/arm-home:/home/arm -v /btrfs/sixtb-int/arm-home/Music:/home/arm/Music -v /btrfs/sixtb-int/arm-home/logs:/home/arm/logs -v /btrfs/sixtb-int/arm-home/media:/home/arm/media -v /btrfs/sixtb-int/arm-home/config:/etc/arm/config --publish 8080 --env ARM_GID=344800018 --env ARM_UID=344800018 --pull newer --privileged docker.io/automaticrippingmachine/automatic-ripping-machine fails to start (without an error message), as does the same command with --user 0 -- I get a container ID but no output from podman ps or podman container ls --all:

# /usr/bin/podman run --name=arm --replace --rm --cgroups=split --network=host --sdnotify=conmon -d --device=/dev/sr0 --user 0 -v /btrfs/sixtb-int/arm-home:/home/arm -v /btrfs/sixtb-int/arm-home/Music:/home/arm/Music -v /btrfs/sixtb-int/arm-home/logs:/home/arm/logs -v /btrfs/sixtb-int/arm-home/media:/home/arm/media -v /btrfs/sixtb-int/arm-home/config:/etc/arm/config --publish 8080 --env ARM_GID=344800018 --env ARM_UID=344800018 --pull newer --privileged docker.io/automaticrippingmachine/automatic-ripping-machine
Port mappings have been discarded as one of the Host, Container, Pod, and None network modes are in use
4ee674dfadd3c974d4d4bad5f05053d45185dd8a6f8bedd8afe36949e64757b3

Using the podman user I get the reported error on the command line:

# /usr/bin/podman run --name=arm --replace --rm --cgroups=split --network=host --sdnotify=conmon -d --device=/dev/sr0 --user podman -v /btrfs/sixtb-int/arm-home:/home/arm -v /btrfs/sixtb-int/arm-home/Music:/home/arm/Music -v /btrfs/sixtb-int/arm-home/logs:/home/arm/logs -v /btrfs/sixtb-int/arm-home/media:/home/arm/media -v /btrfs/sixtb-int/arm-home/config:/etc/arm/config --publish 8080 --env ARM_GID=344800018 --env ARM_UID=344800018 --pull newer --privileged docker.io/automaticrippingmachine/automatic-ripping-machine
Port mappings have been discarded as one of the Host, Container, Pod, and None network modes are in use
Error: unable to find user podman: no matching entries in passwd file

[mhjacks]

Downgrading to podman 4.7.0 (which was the convenient thing to do for me, this is the ExecStart that it generates, and the container and service start, but fail pretty quickly upon running (this seems to be changes internal to the container and not obviously something to do with podman):

ExecStart=/usr/bin/podman run --name=arm --cidfile=%t/%N.cid --replace --rm --cgroups=split --network=host --sdnotify=conmon -d --device=/dev/sr0 --user 0 -v /btrfs/sixtb-int/arm-home:/home/arm -v /btrfs/sixtb-int/arm-home/Music:/home/arm/Music -v /btrfs/sixtb-int/arm-home/logs:/home/arm/logs -v /btrfs/sixtb-int/arm-home/media:/home/arm/media -v /btrfs/sixtb-int/arm-home/config:/etc/arm/config --publish 8080 --env ARM_GID=344800018 --env ARM_UID=344800018 --pull newer --privileged docker.io/automaticrippingmachine/automatic-ripping-machine

It appears to me that podman is trying to start the container (as it should) and there are further problems inside that I need to t/s further. But podman should NOT be failing to look up the user, I would think.

[rhatdan]

So this looks like the podman user does not exists in /etc/passwd in the image docker.io/automaticrippingmachine/automatic-ripping-machine

[mhjacks]

No, it doesn't. You're supposed to be able to inject your own notion of UID/GID via environment variable (e.g. ARM_UID, ARM_GID). There may be other problems with the image - but it certainly appears that podman is complaining about not finding podman in /etc/passwd, right?

[Luap99]

--user lookups the name in the image, never the hosts /etc/passwd so it is totally normal that podman errors as the user does not exists in the image.

[mhjacks]

Really? I did not realize that. I thought that was a host-side lookup. And the way I was using it seems to have only worked accidentally, based on what Ygal said above

[rhatdan]

Yes --user always searches the containers /etc/passwd, so that it sets the UID/GID map.

You could try playing with
--passwd
--passwd-entry=ENTRY

Although I am not sure which happens first.

[mhjacks]

I'm completely up and running now - I removed the User= key from the quadlet. The "infuence" of the host FS is handled by the environment vars. The issue was my understanding of what "user" meant to quadlet/podman - I thought it was a sort of "su" mechanism for the host when running rootfully.

Thanks for the explanation and advice!