Fully rootless operation with ignore_chown_errors = "true" and without newuidmap ?

[dinvlad]

Hi folks,

I'm trying to get Podman working in an environment where not only I don't have root privileges, but we're not permitted to install Podman (or any other executables or configuration files) globally or to make newuidmap available to users. In other words, we have to operate within the constraints of a regular user, without the ability to configure anything on the host that's outside of user $HOME - i.e. a sort of "fully rootless mode".

I should note that I'm ok with running all processes as my UID and having basically no isolation from inside the container - so a container runner like Podman is just used to conveniently run executables packaged in OCI/Docker images (together with their dynamic libraries independent of those on the host).

I was trying to get this working by having ignore_chown_errors = "true" in storage.conf. However, when I run Podman installed under ~/.local/podman/bin, it still fails for each and every command with:

Error: command required for rootless mode with multiple IDs: exec: "newuidmap": executable file not found in $PATH

I'm probably misunderstanding the purpose of ignore_chown_errors = "true" - is there any other way to make this work? I've also tried NEWUIDMAP=/bin/false podman ... and podman run --uidmap 0:0:1 ..., but in each case even podman info --debug fails with the same error.

Note that I do have podman version 4.5.1, along with

fuse-overlayfs: version 1.12
FUSE library version 3.14.1
using FUSE kernel interface version 7.31
fusermount3 version: 3.14.1

crun version 1.8.5
commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
rundir: /run/user/1001/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4

all installed under ~/.local/podman/bin. I also have

# ~/.config/containers/containers.conf 
# See https://github.com/containers/common/blob/master/pkg/config/containers.conf
[engine]
infra_image="k8s.gcr.io/pause:3.8"
# can be croupfs, systemd
cgroup_manager = "systemd"
# can be file, journald
events_logger="file"
exit_command_delay = 10
# can be runc, crun
runtime = "crun"
stop_timeout = 5
conmon_path=[ "/home/jupyter/.local/podman/lib/podman/conmon" ]
helper_binaries_dir = [ "/home/jupyter/.local/podman/lib/podman" ]
static_dir = "/home/jupyter/.local/podman/share/podman/libpod"
volume_path = "/home/jupyter/.local/podman/share/podman/volume"
[engine.runtimes]
crun = [ "/home/jupyter/.local/podman/bin/crun" ]
runc = [ "/home/jupyter/.local/podman/bin/runc" ]
[network]
cni_plugin_dirs = [ "/home/jupyter/.local/podman/lib/cni" ]

and

# grep -vP '^(#.*|)$' ~/.config/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/home/jupyter/.local/podman/run/containers/storage"
graphroot = "/home/jupyter/.local/podman/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
[storage.options.overlay]
ignore_chown_errors = "true"
mount_program = "/home/jupyter/.local/podman/bin/fuse-overlayfs"
mountopt = "nodev,fsync=0"
[storage.options.thinpool]

Fwiw, all of these were configured using

curl -s https://raw.githubusercontent.com/89luca89/distrobox/main/extras/install-podman | sh -s -- --prefix ~/.local

on a host with

Linux vm534567493 5.15.0-1041-azure #48~20.04.1-Ubuntu SMP Wed Jun 21 15:03:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Is it even possible to do achieve what I've described with Podman?

Thank you very much in advance for any tips.

[rhatdan]

We have never tried this without newuidmap and newgidmap installed.

If you add a newuidmap and newgidmap shell script to your PATH environment, you might be able to get it to work.

$ echo $PATH
/home/dwalsh/.cargo/bin:/home/dwalsh/go/bin:/home/dwalsh/bin:/usr/local/sbin:/usr/kerberos/bin:/usr/local/mozilla:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/sbin:/usr/sbin:/var/lib/snapd/snap/bin:/home/dwalsh/bin:/home/dwalsh/go/bin

If I setup /home/dwalsh/bin/newuidmap and /home/dwalsh/bin/newgidmap, it would probably work

[dinvlad]

Thanks for the suggestion! Unfortunately, when I try this, it attempts to use newuidmap/newgidmap and then fails:

$ podman info --debug
ERRO[0000] running `/home/jupyter/.local/podman/bin/newuidmap 1618881 0 1001 1 1 165536 65536`: newuidmap: write to uid_map failed: Operation not permitted 
Error: cannot set up namespace using "/home/jupyter/.local/podman/bin/newuidmap": should have setuid or have filecaps setuid: exit status

[rhatdan]

Did you make /home/jupyter/.local/podman/bin/newuidmap executable?

chmod +x /home/jupyter/.local/podman/bin/newuidmap

[rhatdan]

It should just return 0

[dinvlad]

Yes, of course:

$ /home/jupyter/.local/podman/bin/newuidmap
usage: newuidmap <pid> <uid> <loweruid> <count> [ <uid> <loweruid> <count> ] ...

$ podman info --debug
ERRO[0000] running `/home/jupyter/.local/podman/bin/newuidmap 1618881 0 1001 1 1 165536 65536`: newuidmap: write to uid_map failed: Operation not permitted 
Error: cannot set up namespace using "/home/jupyter/.local/podman/bin/newuidmap": should have setuid or have filecaps setuid: exit status

[rhatdan]

You put the actual binaries there. I just wanted to have an app that returned success. not the real app.

ln -s /bin/true newuidmap

For example

[dinvlad]

Interesting - now I see

$ podman info --debug
cannot setresgid: Invalid argument
Error: setting up the process

I tried to make an empty script in place of setresgid but the error is the same for that.

Looking at the code, could it be that it requires an additional permission to make this syscall?

[dinvlad]

I've also tried unset XDG_RUNTIME_DIR and export XDG_RUNTIME_DIR= but same error

Which seems to suggest the error is coming from a different (seemingly non-optional) branch of the code:

podman/pkg/rootless/rootless_linux.c

Lines 896 to 898 in 1f455cf

	if (syscall_setresgid (0, 0, 0) < 0)
	{
	fprintf (stderr, "cannot setresgid: %m\n");

And this is with --cgroup-manager set to either systemd or cgroupfs.

[eriksjolund]

I tried this on a Fedora 38 machine

1. Open a root shell

   sudo -i
   

2. Create the file /root/test.bash with this contents

   #!/bin/bash
   
   set -o errexit
   set -o nounset
   
   user=$1
   cmd=$2
   
   useradd --create-home --system $user
   
   echo "
   mkdir -p bin
   cd bin
   ln -s $cmd newuidmap
   ln -s $cmd newgidmap
   
   " '
   
   uid=0
   gid=0 
   export PATH=/home/$USER/bin:$PATH
   podman  --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs \
           --storage-opt=overlay.ignore_chown_errors=true \
           run --rm \
               --sysctl net.ipv4.ping_group_range="$uid $gid" \
               --user $uid:$gid \
               --uidmap $uid:0:1 \
               --gidmap $gid:0:1 \
               docker.io/library/alpine cat /etc/os-release
   
   ' | systemd-run --collect \
                   --machine=${user}@ \
                   --pipe \
                   --quiet \
                   --user \
                   --wait bash 
   

3. Run the script. Pass a non-existing username as the first argument (for example test1). The second argument is the replacement for newuidmap and newgidmap.

   [root@fedora ~]# bash test.bash test1 /bin/true
   ERRO[0000] cannot find UID/GID for user test1: no subuid ranges found for user "test1" in /etc/subuid - check rootless mode in man pages.
   WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user
   Trying to pull docker.io/library/alpine:latest...
   Getting image source signatures
   Copying blob 31e352740f53 done
   Copying config c1aabb73d2 done
   Writing manifest to image destination
   Storing signatures
   NAME="Alpine Linux"
   ID=alpine
   VERSION_ID=3.18.2
   PRETTY_NAME="Alpine Linux v3.18"
   HOME_URL="https://alpinelinux.org/"
   BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
   [root@fedora ~]# echo $?
   0
   [root@fedora ~]# 
   

Note, the two first lines in the output

ERRO[0000] cannot find UID/GID for user test1: no subuid ranges found for user "test1" in /etc/subuid - check rootless mode in man pages.
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user

Variation1 of the experiment

I tried replacing /bin/true with /bin/false

bash test.bash test2 /bin/false

It worked.

Variation2 of the experiment

I tried replacing

uid=0
gid=0

with

uid=$(id -u)
gid=$(id -g)

It worked.

Variation3 of the experiment

I tried

sudo mv /usr/bin/newuidmap /usr/bin/newuidmap.moved
sudo mv /usr/bin/newgidmap /usr/bin/newgidmap.moved

It didn't make a difference in the examples above.

Variation4 of the experiment

Remove the option --system from the useradd command.
Now the script fails the error message

cannot setresgid: Invalid argument
Error: setting up the process

[root@fedora ~]# bash test.bash test5 /bin/true
cannot setresgid: Invalid argument
Error: setting up the process
[root@fedora ~]#

The difference is that subuids and subgids are allocated to the user when --system is not given.

Variation5 of the experiment

[root@fedora ~]# cat /proc/sys/user/max_user_namespaces
29044
[root@fedora ~]# echo 0 > /proc/sys/user/max_user_namespaces
[root@fedora ~]# bash test.bash test6 /bin/true
cannot clone: No space left on device
user namespaces are not enabled in /proc/sys/user/max_user_namespaces
Error: cannot re-exec process
[root@fedora ~]# 

Previous discussion

• Add support for disabling the use of newuidmap and newgidmap #8929

[dinvlad]

Thanks for looking into it - do I take it that the main outcome of your testing is that truly rootless operation depends on the user being a system user?

[eriksjolund]

I just used the useradd option --system because I wanted to create a user without any subuids and subgids.
Another way to do it should be to create a normal user and then remove the subuids and subgids afterwards.

In Variation4 I also got the same error message as you

cannot setresgid: Invalid argument
Error: setting up the process

Did your user have subuids or subgids defined in /etc/subuid and /etc/subgid?

I deliberately created a script for doing the test in one step with a newly created user, just to be sure
that the Podman user namespace for that user has not yet been set up.

I wonder exactly what the requirements are for using podman in
the mode: Using rootless single mapping into the namespace

Maybe these two requirements?

• Value of /proc/sys/user/max_user_namespaces should be greater than zero
• The executable /usr/bin/fusermount3 with setuid permission

  $ ls -l /usr/bin/fusermount3
  -rwsr-xr-x. 1 root root 41072 Apr  3 02:00 /usr/bin/fusermount3
  

One thing that might good to know for all Debian and Ubuntu users:
It seems this command is required:

echo 1 > /proc/sys/kernel/unprivileged_userns_clone

I learned about unprivileged_userns_clone here
https://rootlesscontaine.rs/getting-started/common/sysctl/

On my Fedora computer there is no such file

$ ls /proc/sys/kernel/unprivileged_userns_clone
ls: cannot access '/proc/sys/kernel/unprivileged_userns_clone': No such file or directory

[dinvlad]

Did your user have subuids or subgids defined in /etc/subuid and /etc/subgid?

Yes - jupyter:165536:65536. I also have:

jupyter@vm534567493:~$ cat /proc/sys/user/max_user_namespaces
27689
jupyter@vm534567493:~$ ls -al $(which fusermount3)
-rwxr-xr-x 1 jupyter jupyter 79504 Jul 24 18:57 /home/jupyter/.local/podman/bin/fusermount3
jupyter@vm534567493:~$ cat /proc/sys/kernel/unprivileged_userns_clone
1

Does this make any of what I'm trying to achieve easier or more complicated?

FWIW, I also have another - even more stringent - environment where all user processes are already running inside a Docker container. I'm assuming it wouldn't be possible to run even rootless Podman in it (i.e. to have nested execution inside a regular container)?

[rhatdan]

@giuseppe PTAL

[rhatdan]

setresgid is a system call not an executable.

[dinvlad]

Right! That's why I mentioned this as well:

Looking at the code, could it be that it requires an additional permission to make this syscall?

[giuseppe]

we have a check in place where if you specify --uidmap we expect there are multiple IDs available.

Can you try without using --uidmap?

[dinvlad]

Sorry, what do you mean? I'm just using podman info --debug without any--uidmap

[giuseppe]

ok sorry I misread the code.

You need to drop the user from the /etc/subuid and /etc/subgid file.

If the user is assigned additional IDs there we prefer to raise the error instead of ignoring it.

[dinvlad]

Hmm, not sure I follow tbh - I'm just a regular user who has no control over what's in /etc/subuid and /etc/subgid - how can we work with that?

[eriksjolund]

Last week a workaround was presented here:
#8929 (comment)
(I haven't tried it out)