Why --userns=keep-id doesn't work?

[abitrolly]

✗ mkdir xxx
✗ podman run -it --rm --security-opt label=disable --userns=keep-id -v $PWD/xxx:/xxx gitpod
gitpod ~ $ ls -la /xxx
total 8
drwxr-xr-x.  2 anatoli anatoli 4096 Oct 22 04:32 .
dr-xr-xr-x. 20 root    root    4096 Oct 22 04:32 ..
gitpod ~ $ echo $UID
33333
gitpod ~ $ touch /xxx/aaa
touch: cannot touch '/xxx/aaa': Permission denied

Why it doesn't map to my $UID? These are values from my host.

✗ ps faux | grep "b\ash"
133332     75825  0.1  0.1  12392  9188 pts/0    Ss+  07:32   0:00      \_ bash
✗ echo $UID
1000

 podman version
Client:       Podman Engine
Version:      4.2.1
API Version:  4.2.1
Go Version:   go1.18.5
Built:        Wed Sep  7 22:58:19 2022
OS/Arch:      linux/amd64

[mheon]

THe ownership of . from the ls -la seems correct, though? Are you sure the EPERM isn't from something else (SELinux being a prominent cause)?

[mheon]

I can't grab the Gitpod image at the moment (Dockerhub unhelpfully giving me a 401), but using fedora:36 everything works as I expect.

[abitrolly]

I bet you need the image. Please try with from GitLab. It fails to touch /xxx/111 too.

podman run -it --rm --security-opt label=disable --userns=keep-id -v $PWD/xxx:/xxx registry.gitlab.com/gitlab-org/gitlab-development-kit/gitpod-workspace:stable

[rhatdan]

The process running within the container is running as gitpod, your user is not gitpod, so it is not allowed to write to the directory.

Use the :U option on the volume mount to fix the ownership of the directory.

[rhatdan]

$ podman run -it --rm --security-opt label=disable --userns=keep-id -v $PWD/xxx:/xxx:U registry.gitlab.com/gitlab-org/gitlab-development-kit/gitpod-workspace:stable
gitpod ~ $ touch /xxx/dan
gitpod ~ $ exit
$ ls -ld xxx
drwxr-xr-x. 2 133332 133332 17 Oct 24 10:19 xxx

[abitrolly]

:U doesn't work, because it modifies file system permissions on the host filesystem, and as a result I lose write access to this dir from the host.

[eriksjolund]

@abitrolly I think you need to use --userns=keep-id:uid=$uid,gid=$gid if you want to map the container user gitpod to your regular user on the host. The new options uid and gid are only available in the newly released Podman 4.3.
The container user gitpod has the UID 33333 and the GID 33333, so you would need to specify --userns=keep-id:uid=33333,gid=33333. You don't need to use the :U option.

If your Podman version is older than Podman 4.3, you could achieve the same thing by using --uidmap and --gidmap (see tip Podman run fails with "Error: unrecognized namespace mode keep-id:uid=1000,gid=1000 passed" in troubleshooting.md)

I tested it out on Fedora 36 with Podman 4.2.1

1. Run sudo useradd test

2. Run sudo machinectl shell test@

3. Run these commands

[test@linux ~]$ podman pull -q registry.gitlab.com/gitlab-org/gitlab-development-kit/gitpod-workspace:stable 
[test@linux ~]$ podman image inspect --format '{{.Config.User}} ' registry.gitlab.com/gitlab-org/gitlab-development-kit/gitpod-workspace:stable
gitpod 
[test@linux ~]$ podman run --rm registry.gitlab.com/gitlab-org/gitlab-development-kit/gitpod-workspace:stable /bin/grep gitpod /etc/passwd
gitpod:x:33333:33333::/home/gitpod:/bin/bash

4. Run podman system reset (I don't know if this step is needed. See the discussion below)

5. Run these commands

[test@linux ~]$ uid=33333
[test@linux ~]$ gid=33333
[test@linux ~]$ subuidSize=$(( $(podman info --format "{{ range .Host.IDMappings.UIDMap }}+{{.Size }}{{end }}" ) - 1 ))
[test@linux ~]$ echo $subuidSize 
65536
[test@linux ~]$ subgidSize=$(( $(podman info --format "{{ range .Host.IDMappings.GIDMap }}+{{.Size }}{{end }}" ) - 1 ))
[test@linux ~]$ echo $subgidSize
65536
[test@linux ~]$ mkdir xxx
[test@linux ~]$ ls -ld xxx
drwxr-xr-x. 2 test test 18 Oct 25 18:53 xxx
[test@linux ~]$ podman run \
    --user $uid:$gid \
    --uidmap $uid:0:1 \
    --uidmap 0:1:$uid \
    --uidmap $(($uid+1)):$(($uid+1)):$(($subuidSize-$uid)) \
    --gidmap $gid:0:1 \
    --gidmap 0:1:$gid \
    --gidmap $(($gid+1)):$(($gid+1)):$(($subgidSize-$gid)) \
    -it \
    --rm \
    -v $PWD/xxx:/xxx:Z \
        registry.gitlab.com/gitlab-org/gitlab-development-kit/gitpod-workspace:stable
gitpod ~ $ ls -ld /xxx
drwxr-xr-x. 2 gitpod gitpod 6 Oct 25 16:53 /xxx
gitpod ~ $ touch /xxx/file
gitpod ~ $ exit
exit
[test@linux ~]$ ls -ld xxx
drwxr-xr-x. 2 test test 18 Oct 25 18:54 xxx
[test@linux ~]$ ls -ld xxx/file
-rw-r--r--. 1 test test 0 Oct 25 18:54 xxx/file
[test@linux ~]$

First I tried the same thing without doing step 4 (i.e. running podman system reset) but then the podman run ... command in step 5 took forever. (I pressed Ctrl-C after a few minutes because I didn't want to wait for the command to finish).

This is the error message I got after pressing Ctrl-C

^CError: error creating container storage: creating an ID-mapped copy of layer "b954d3d642f12b6d10ac13b01a1985e03cbdce66dea86df312823dcd88e65b92": signal: interrupt

[abitrolly]

@eriksjolund I am going to try it. Thanks.

Is it possible to record the numeric UID into Dockerfile USER directive at the build time?

[abitrolly]

@eriksjolund seems to work fine. https://github.com/yakshaveinc/linux/blob/master/bin/runin-podman.sh

Although I have concerns about how --userns=keep-id works. Looks like it runs container with my UID? What security issues can arise from that? I can run sudo and access some services that default users can't.

[rhatdan]

In containers we map the users UID into the container by default. It is either mapped in as UID==0 or as UID==$UID (the users UID, with keep-id) If you do not want the users UID mapped at all, you can use nomap.

$ id -u
3267
$ podman run --rm fedora cat /proc/self/uid_map 
         0       3267          1
         1     100000      65536
$ podman run --userns=keep-id --rm fedora cat /proc/self/uid_map 
         0          1       3267
      3267          0          1
      3268       3268      62269
$ podman run --userns=nomap --rm fedora cat /proc/self/uid_map 
         0          1      65536

The mappings get a little confusing here, since they are remapping the default user namespace. A better way to show what is going on would be here:

$ podman run -d --rm fedora sleep 100
8988c2979ab3548753d7b2947a0261c9eaa31a7c651b0ba5a8411822c9617df3
kube (volume) $ podman top -l user huser 
USER        HUSER
root        3267
$ podman run -d --userns=keep-id --rm fedora sleep 100
08cf48b71a137974ca07a690a39f5e308849eb7ae12c2a16849f638939e4d22d
$ podman top -l user huser 
USER        HUSER
dwalsh      dwalsh
$ podman run -d --userns=nomap --rm fedora sleep 100
cc09a876879dc4604ca2656da303ff06e54088fd62794a4277d7b170586ca461
$ podman top -l user huser 
USER        HUSER
root        100000

[rhatdan]

Finally another example of the three mappings. I create a directory with a file in my home directory and then mount it into the container. If I look at the file in the three different modes, the file ownership changes depending on the mapping.

$ mkdir test
$ touch test/foobar
$ podman run -v ./test:/test:Z --rm fedora ls -l /test
total 0
-rw-r--r--. 1 root root 0 Dec 29 12:20 foobar
$ podman run --userns=keep-id -v ./test:/test:Z --rm fedora ls -l /test
total 0
-rw-r--r--. 1 dwalsh dwalsh 0 Dec 29 12:20 foobar
$ podman run --userns=nomap -v ./test:/test:Z --rm fedora ls -l /test
total 0
-rw-r--r--. 1 nobody nobody 0 Dec 29 12:20 foobar

[abitrolly]

@rhatdan hi. Have you made any progress with the article about UID mappings? I am particularly interested in security implications and attack vectors depending on various combinations of these.

[rhatdan]

BTW I will be converting this into a Blog.

[abitrolly]

Without diagrams all UID mapping articles are mostly useless to me. When authors write something like UID=$UID, it all makes sense to them, because they already know which UID is where, but for users who don't even understand what there is some UID inside and UID outside, it is often doesn't make any sense. Curse of knowledge.

Like when you say "map", do you really mean that there is some shared memory with UID that is mapped inside of container? Or it is just associative array somewhere in Linux kernel, which is called "map" in some languages (dict in Python) which is a lookup table to find out which host UID to use to run container process. There are too many questions and that's makes it hard.

As a good example I would say that this article helped greatly https://www.redhat.com/en/blog/understanding-root-inside-and-outside-container It is not a diagram, but a good comparison table.

[eriksjolund]

Or it is just associative array

I found some explanation in
the article Minimizing Privilege for Building HPC Containers (Reid Priedhorsky, R. Shane Canon, Timothy Randles, and Andrew J. Younge)
https://arxiv.org/pdf/2104.07508.pdf

2.1.1 UID and GID maps.

quote:
Because the map is one-to-one, there is no squashing of multiple IDs to a single ID in either direction.

quote:
These mappings need not be complete. The four possibilities for a given pair of IDs are