Can't run rootless Podman with users managed by systemd-homed on a correct setup.

[IPlayZed]

Issue Description

Command podman run -v -dt -p 8080:80/tcp docker.io/library/httpd fails with:

Trying to pull docker.io/library/httpd:latest...
Getting image source signatures
Copying blob 8c226ac2053e done  
Copying blob 2832a695827e done  
Copying blob 360eba32fa65 done  
Copying blob 45a0ea29816d done  
Copying blob b57c1299d233 done  
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0": processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/gshadow): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /etc/gshadow: invalid argument): exit status 1

Steps to reproduce the issue

Steps to reproduce the issue

1. Use a systemd-homed managed user.
2. Setup rootless Podman.
3. Try to run a Podman container, like podman run -v -dt -p 8080:80/tcp docker.io/library/httpd.

Describe the results you received

Describe the results you received

Describe the results you expected

Container runs normally as it does launching it as a superuser.

podman info output

Podman info:

❯ podman info
host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.8-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
  cpuUtilization:
    idlePercent: 93.49
    systemPercent: 5.87
    userPercent: 0.64
  cpus: 32
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2048
  hostname: minefpc
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 60311
      size: 1
    uidmap:
    - container_id: 0
      host_id: 60311
      size: 1
  kernel: 6.4.15-hardened1-1-hardened
  linkmode: dynamic
  logDriver: journald
  memFree: 117821149184
  memTotal: 134976802816
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: /usr/lib/podman/netavark is owned by netavark 1.7.0-1
    path: /usr/lib/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.9-1
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/user/60311/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: /usr/bin/pasta is owned by passt 2023_09_08.05627dc-1
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    path: /run/user/60311/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 164024020992
  swapTotal: 164982943744
  uptime: 0h 37m 8.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/personal/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/personal/.local/share/containers/storage
  graphRootAllocated: 279153827840
  graphRootUsed: 108797673472
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/60311/containers
  transientStore: false
  volumePath: /home/personal/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1693343961
  BuiltTime: Tue Aug 29 23:19:21 2023
  GitCommit: 5db42e86862ef42c59304c38aa583732fd80f178-dirty
  GoVersion: go1.21.0
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman version:

❯ podman version
Client:       Podman Engine
Version:      4.6.2
API Version:  4.6.2
Go Version:   go1.21.0
Git Commit:   5db42e86862ef42c59304c38aa583732fd80f178-dirty
Built:        Tue Aug 29 23:19:21 2023
OS/Arch:      linux/amd64

Kernel:

❯ uname -a
Linux minefpc 6.4.15-hardened1-1-hardened #1 SMP PREEMPT_DYNAMIC Tue, 12 Sep 2023 17:08:22 +0000 x86_64 GNU/Linux

OS: Arch Linux (x64)

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

The output of userdbctl:

NAME                           DISPOSITION        UID   GID REALNAME                         HOME                     SHELL
   root                           intrinsic            0     0 -                                /root                    /bin/bash
┌─ ↓ begin system users ↓         system               1     - First system user                -                        -
   bin                            system               1     1 -                                /                        /usr/bin/nologin
   daemon                         system               2     2 -                                /                        /usr/bin/nologin
   mail                           system               8    12 -                                /var/spool/mail          /usr/bin/nologin
   ftp                            system              14    11 -                                /srv/ftp                 /usr/bin/nologin
   rpc                            system              32    32 Rpcbind Daemon                   /var/lib/rpcbind         /usr/bin/nologin
   http                           system              33    33 -                                /srv/http                /usr/bin/nologin
   clamav                         system              64    64 Clam AntiVirus                   /                        /usr/bin/nologin
   uuidd                          system              68    68 -                                /                        /usr/bin/nologin
   dbus                           system              81    81 System Message Bus               /                        /usr/bin/nologin
   polkitd                        system             102   102 PolicyKit daemon                 /                        /usr/bin/nologin
   gdm                            system             120   120 Gnome Display Manager            /var/lib/gdm             /usr/bin/nologin
   rtkit                          system             133   133 RealtimeKit                      /proc                    /usr/bin/nologin
   usbmux                         system             140   140 usbmux user                      /                        /usr/bin/nologin
   nvidia-persistenced            system             143   143 NVIDIA Persistence Daemon        /                        /usr/bin/nologin
   cups                           system             209   209 cups helper user                 /                        /usr/bin/nologin
   fwupd                          system             951   951 Firmware update daemon           /var/lib/fwupd           /usr/bin/nologin
   systemd-journal-upload         system             953   953 systemd Journal Upload           /                        /usr/bin/nologin
   saned                          system             955   955 SANE daemon user                 /                        /usr/bin/nologin
   mysql                          system             956   956 MariaDB                          /var/lib/mysql           /usr/bin/nologin
   sddm                           system             958   958 Simple Desktop Display Manager   /var/lib/sddm            /usr/bin/nologin
   nm-openconnect                 system             960   960 NetworkManager OpenConnect       /                        /usr/bin/nologin
   openvpn                        system             961   961 OpenVPN                          /                        /usr/bin/nologin
   nm-openvpn                     system             962   962 NetworkManager OpenVPN           /                        /usr/bin/nologin
   dnsmasq                        system             963   963 dnsmasq daemon                   /                        /usr/bin/nologin
   qemu                           system             964   964 QEMU user                        /                        /usr/bin/nologin
   libvirt-qemu                   system             965   965 Libvirt QEMU user                /                        /usr/bin/nologin
   gluster                        system             966   966 GlusterFS daemons                /var/run/gluster         /usr/bin/nologin
   git                            system             968   968 git daemon user                  /                        /usr/bin/git-shell
   geoclue                        system             969   969 Geoinformation service           /var/lib/geoclue         /usr/bin/nologin
   flatpak                        system             970   970 Flatpak system helper            /                        /usr/bin/nologin
   colord                         system             971   971 Color management daemon          /var/lib/colord          /usr/bin/nologin
   brltty                         system             972   972 Braille Device Daemon            /var/lib/brltty          /usr/bin/nologin
   avahi                          system             973   973 Avahi mDNS/DNS-SD daemon         /                        /usr/bin/nologin
   tss                            system             975   975 tss user for tpm2                /                        /usr/bin/nologin
   systemd-timesync               system             976   976 systemd Time Synchronization     /                        /usr/bin/nologin
   systemd-resolve                system             977   977 systemd Resolver                 /                        /usr/bin/nologin
   systemd-journal-remote         system             978   978 systemd Journal Remote           /                        /usr/bin/nologin
   systemd-oom                    system             979   979 systemd Userspace OOM Killer     /                        /usr/bin/nologin
   systemd-network                system             980   980 systemd Network Management       /                        /usr/bin/nologin
   systemd-coredump               system             981   981 systemd Core Dumper              /                        /usr/bin/nologin
└─ ↑ end system users ↑           system             999     - Last system user                 -                        -
   developer-no-homed             regular           1000  1000 developer (not Homed managed)    /home/developer-no-homed /bin/bash
┌─ ↓ begin systemd-homed users ↓  regular          60001     - First systemd-homed user         -                        -
   gaming                         regular          60197 60197 Börcsök Balázs Róbert (gaming)   /home/gaming             /bin/bash
   admin                          regular          60282 60282 Börcsök Balázs Róbert (admin)    /home/admin              /bin/bash
   personal                       regular          60311 60311 Börcsök Balázs Róbert (personal) /home/personal           /bin/bash
└─ ↑ end systemd-homed users ↑    regular          60513     - Last systemd-homed user          -                        -
┌─ ↓ begin mapped users ↓         regular          60514     - First mapped user                -                        -
└─ ↑ end mapped users ↑           regular          60577     - Last mapped user                 -                        -
┌─ ↓ begin dynamic system users ↓ dynamic          61184     - First dynamic system user        -                        -
└─ ↑ end dynamic system users ↑   dynamic          65519     - Last dynamic system user         -                        -
   nobody                         intrinsic        65534 65534 Kernel Overflow User             /                        /usr/bin/nologin
┌─ ↓ begin container users ↓      container       524288     - First container user             -                        -
└─ ↑ end container users ↑        container   1878982656     - Last container user              -                        -
46 users listed.

It works fine for users not managed by systemd-homed.

[mheon]

I believe systemd-homed uses NFS, no? If so, I think this is expected as NFS does not support subuid/subgid ranges now. @giuseppe Does that sound correct?

[IPlayZed]

I believe systemd-homed uses NFS, no? If so, I think this is expected as NFS does not support subuid/subgid ranges now. @giuseppe Does that sound correct?

What do you mean by NFS? The filesystem the user's home directory is mapped to is Btrfs, if that is the concern.

[mheon]

Ah, looks like that's not necessarily a requirement.

Googling about suggests that you could be seeing something related to systemd/systemd#21952 (or, alternatively, systemd-homed has not configured subuid/subgid ranges for the users in question at all?)

[rhatdan]

The problem is around the /etc/subuid and /etc/subgid fields not being populated.

[giuseppe]

you are allowed to have only one uid/gid with systemd-homed. All you can do is to set ignore_chown_errors and squash all images to use one single ID

[kir68k]

you are allowed to have only one uid/gid with systemd-homed. All you can do is to set ignore_chown_errors and squash all images to use one single ID

So is this simply unsolvable for now, due to the way both of these programs work?

[IPlayZed]

you are allowed to have only one uid/gid with systemd-homed. All you can do is to set ignore_chown_errors and squash all images to use one single ID

I am a bit confused with this situation, the Systemd issue @mheon presented is closed and I would assume that this issue is fixed, but it seems like not? I do not even know which software (Podman or Homed) should be considered as breaking?

I would assume that distributions will adopt Homed in the future as it really has it's upsides, does that mean that Podman would break for all of those users?

[rhatdan]

In order to setup a user namespace, users have to have entries in /etc/subuid and /etc/subgid, If systemd-homed is not populating these filed with entries then users can not create UIDs different then their default UID, which breaks all containers. If you think this is a bug then it needs to be reported to systemd, there is nothing that the podman team can do.

[IPlayZed]

I will report this to Systemd then referencing this issue then, thank you for the clarification.

[IPlayZed]

I created an issue in the Systemd repo for it btw.