apt-get failing on rootless Podman

[secondspass]

Hi there,

I'm on HPC system where we're testing out Rootless Podman (same place as this discussion: #11184), specifically seeing if we can provide users the ability to build images on their own on the shared login nodes. Due to admin limitations, it's not really possible to provide /etc/subuid and /etc/subgid mappings for all our users (we have hundreds with new users coming in all the time).

When you try to do an apt-get update or apt-get install on an ubuntu container with rootless Podman on a system without the subuid/subgid mappings, it fails like so (this is a test on a workstation identical to the HPC node):

(base) bash-4.4$  podman run --rm --runtime crun -it --entrypoint bash docker.io/library/ubuntu
root@be1d3f62fed7:/# apt-get update
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (22: Invalid argument)
E: seteuid 100 failed - seteuid (22: Invalid argument)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
Reading package lists... Done
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (22: Invalid argument)
W: chown to _apt:root of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (22: Invalid argument)
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (22: Invalid argument)
E: seteuid 100 failed - seteuid (22: Invalid argument)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
E: Method gave invalid 400 URI Failure message: Failed to setgroups - setgroups (1: Operation not permitted)
E: Method http has died unexpectedly!
E: Sub-process http returned an error code (112)

Running with flags like this allows apt-get update to finish but fails for apt-get install. (The following is from a workstation identical to the HPC node. I've verified running this on the HPC node behaves the same way).

[76a@raptor07 ~]$ podman run --rm --runtime crun  --user=<uid>:<gid> --uidmap 0:0:1 --env-host  --net=host --pid=host --ipc=host -it --entrypoint bash docker.io/library/ubuntu
40892@raptor07:/$ apt-get update
Get:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease [265 kB]
[…]
Get:18 http://ports.ubuntu.com/ubuntu-ports focal-security/main ppc64el Packages [528 kB]
Fetched 16.2 MB in 7s (2301 kB/s)
Reading package lists... Done
40892@raptor07:/$ apt-get -y install gcc-8
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  binutils binutils-common binutils-powerpc64le-linux-gnu cpp-8 gcc-8-base gcc-9-base libasan5 libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0
  libgcc-8-dev libgomp1 libisl22 libitm1 liblsan0 libmpc3 libmpfr6 libquadmath0 libtsan0 libubsan1 linux-libc-dev manpages manpages-dev
Suggested packages:
  binutils-doc gcc-8-locales gcc-8-doc glibc-doc man-browser
The following NEW packages will be installed:
  binutils binutils-common binutils-powerpc64le-linux-gnu cpp-8 gcc-8 gcc-8-base gcc-9-base libasan5 libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0
  libctf0 libgcc-8-dev libgomp1 libisl22 libitm1 liblsan0 libmpc3 libmpfr6 libquadmath0 libtsan0 libubsan1 linux-libc-dev manpages manpages-dev
0 upgraded, 29 newly installed, 0 to remove and 3 not upgraded.
Need to get 36.1 MB of archives.
After this operation, 144 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports focal/main ppc64el manpages all 5.05-1 [1314 kB]
[…]
Get:29 http://ports.ubuntu.com/ubuntu-ports focal/main ppc64el manpages-dev all 5.05-1 [2266 kB]
Fetched 36.1 MB in 1s (31.9 MB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
debconf: delaying package configuration, since apt-utils is not installed
dpkg: error: requested operation requires superuser privilege
E: Sub-process /usr/bin/dpkg returned an error code (2)
40892@raptor07:/$

It's the same errors during a Podman build as well when I try to do a RUN apt-get update or a RUN apt-get install ....

To recreate this, all you need is to try and do the above with rootless Podman after removing the /etc/sub[uid|gid] mappings for your user from your machine (you may also want to kill any existing Podman processes, and then log off and log in). I tested it on Podman v3.0.2-dev and v3.2.3.

Does anyone have any thoughts of how this could be worked around? Looking around the internet, I'd found one blog post talking about this from 2016: https://www.cyphar.com/blog/post/20160627-rootless-containers-with-runc but it seems like they didn't have a solution either.

[rhatdan]

In order to build an image with multiple UIDs you need to own multiple UIDs.
Some people have tried to hack around this, but even those cases will not work correctly if one of the processes does a setuid() or setgid() to a UID other then root.

[secondspass]

ah okay, that makes sense.

[rhatdan]

containers/storage just got the ability to use /etc/subuid and /etc/subgid from the nextwork. This will be working it's way though the system, You need shadow-utils-4.9 to make this work, and I am not sure whether FREEIPA or other ldap support is available yet.

[secondspass]

I think having ldap support would make it more feasible. I don't know enough how that would work to know if we would run out of subuid/subgid mappings given the number of users we have. It would be nice if we could figure out a way to dynamically add and remove those mappings during a user's job submission (a user has exclusive access to the allocated nodes during a job submission) and thus avoid the problem of running out of mappings.

[rhatdan]

Yes there are around 4 Billion UIDs available giving each user 64k allows for 64k users.

But if you can drop down to say 10000 UIDS/per user now you have 430000 users.

If you blocked ranges of UIDs from ever being used on network storage then you could allocate the UIDs per user logging onto a host.

[mickare]

Not related, but for anyone who finds this discussion in the future when searching for the error messages setgroups 65534 failed or setegid 65534 failed.

Fix for setgroups 65534 failed or setegid 65534 failed

Follow the Arch configuration guide for podman and set the subuid and subgid with a large enough ID space!
If the number is too low it will fail!!!

1. Set a large enough sub-space of ids (> 65536), e.g. replace username:

   $ usermod --add-subuids 100000-200000 --add-subgids 100000-200000 username

2. Check if the change has been applied to /etc/subuid and /etc/subgid.
3. Apply the change: podman system migrate

I had my troubles to find that information that my id-space was too small!

[Dreista]

Large enough subuid and subgid range may not be sufficient. In my case I need to specify an explicit size for the automatic user namespace (--userns=auto:size=65535).

The documentation says auto will estimate a size for the user namespace, but unfortunately the estimation is just off by 1 to make apt work 😑

podman inspect <container>

...
               "IDMappings": {
                    "UidMap": [
                         "0:100000:65534"
                    ],
                    "GidMap": [
                         "0:100000:65534"
                    ]
               },
...

cat /etc/passwd

...
_apt:x:42:65534::/nonexistent:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
...