Different behavior of bind-mounting in non-initial mount namespace

[tarun360]

If I enter a mount namespace and then create a podman container, I see two podman processes instead of one:

gupttaru    4711  0.0  0.0 220236  8240 ?        S    13:52   0:00  |   \_ sshd: gupttaru@pts/2
gupttaru    4714  0.0  0.1 109648 13428 pts/2    Ss   13:52   0:00  |       \_ -bash
root        6504  0.0  0.1 144788 11152 pts/2    S    13:55   0:00  |           \_ sudo unshare -m bash
root        6505  0.1  0.0  18156  6820 pts/2    S    13:55   0:00  |               \_ bash
root        6611  0.0  0.1 144804 12884 pts/2    S    13:55   0:00  |                   \_ sudo -u gupttaru bash
gupttaru    6612  0.1  0.1 109600 13536 pts/2    S    13:55   0:00  |                       \_ bash
gupttaru    6730  0.1  0.3 1269544 35308 pts/2   Sl+  13:56   0:00  |                           \_ podman --root /tmp/containers run --rm -ti my-image /bin/bash
gupttaru    6738  0.6  0.4 1417008 40564 pts/2   Sl+  13:56   0:00  |                               \_ podman --root /tmp/containers run --rm -ti my-image /bin/bash
gupttaru    6750  0.0  0.0   5156  3024 pts/2    S    13:56   0:00  |                                   \_ slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/21363/netns/netns-fb16aa1a-bfa9-bae0-74f3-50b26ca1c422 tap0

In above:

• namespaces of pid 6612 and 6730 are exactly same
• namespaces of pid 6730 and 6738 differ for mnt and user namespaces
• namespaces of pid 6738 and 6750 differ for mnt namespace

In comparison, if I just create a podman container without entering any mount namespace, then I only see 1 podman process:

gupttaru    4711  0.0  0.0 220236  8240 ?        S    13:52   0:00  |   \_ sshd: gupttaru@pts/2
gupttaru    4714  0.0  0.1 109648 13428 pts/2    Ss   13:52   0:00  |       \_ -bash
gupttaru    6957  1.6  0.4 1490996 41276 pts/2   Sl+  14:03   0:00  |           \_ podman --root /tmp/containers run --rm -ti my-image /bin/bash
gupttaru    6970  0.0  0.0   5156  2512 pts/2    S    14:03   0:00  |               \_ slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/21363/netns/netns-0e4a4d57-4100-ab6c-53ab-3ff757b01905 tap0  

In above:

• namespaces of pid 4714 and 6957 differ for mnt and user namespaces
• namespaces of pid 6957 and 6970 differ for mnt namespace

Question: Why in 1st case there are apparently two podman processes?

---

In the 1st case it appears that bind-mounts are performed from user-namespace of pid 6738, which has different uid as compared to the user; this is creating problems for bind-mounting of some fuse fs apparently (The fuse kernel layer itself restricts file system access to the mounting user (fuse.txt)).

In 2nd case, it appears that bind-mounts are performed from user-namespace of pid 4714 (which has same uid as the user so it doesn't create any problems)

Example showing error while bind-mounting a fuse fs when podman container is created within a mount namespace:

$ sudo unshare -m bash
# sudo -u gupttaru bash
$ gcsfuse --debug_fuse --implicit-dirs bucket-name /u/gupttaru/gcsfuse
I0823 14:19:35.302571 2023/08/23 14:19:35.302529 Start gcsfuse/0.42.4 (Go version go1.20.4) for app "" using mount point: /u/gupttaru/gcsfuse
$ /u/gupttaru/gcsfuse/bucket-name/
bash: /u/gupttaru/gcsfuse/bucket-name/: Is a directory
$ podman --root  /tmp/containers run --rm -ti --volume /u/gupttaru/gcsfuse/bucket-name/:/u/gupttaru/gcsfuse/bucket-name/:ro my-image /bin/bash
Error: statfs /u/gupttaru/gcsfuse/bucket-name: no such file or directory

The above error doesn't occur when I am not within a mount namespace:

$ gcsfuse --debug_fuse --implicit-dirs bucket-name /u/gupttaru/gcsfuse
I0823 14:21:37.800611 2023/08/23 14:21:37.800565 Start gcsfuse/0.42.4 (Go version go1.20.4) for app "" using mount point: /u/gupttaru/gcsfuse
$ /u/gupttaru/gcsfuse/bucket-name/
-bash: /u/gupttaru/gcsfuse/bucket-name/: Is a directory
$ podman --root  /tmp/containers run --rm -ti --volume /u/gupttaru/gcsfuse/bucket-name/:/u/gupttaru/gcsfuse/bucket-name/:ro my-image /bin/bash
# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  u  usr  var

[tarun360]

Update:
In the 1st example above, if I run the gcsfuse command within podman unshare, then I am able to bind-mount the directory, although I am not sure why is it needed. So the following works

$ sudo unshare -m bash
# sudo -u gupttaru bash
$ podman unshare gcsfuse --debug_fuse --implicit-dirs bucket-name /u/gupttaru/gcsfuse
I0823 14:19:35.302571 2023/08/23 14:19:35.302529 Start gcsfuse/0.42.4 (Go version go1.20.4) for app "" using mount point: /u/gupttaru/gcsfuse
$ podman --root  /tmp/containers run --rm -ti --volume /u/gupttaru/gcsfuse/bucket-name/:/u/gupttaru/gcsfuse/bucket-name/:ro my-
[root@622a9586381d /]# ls /u/gupttaru/gcsfuse/bucket-name/
data

[Luap99]

The two processes are because of the reexec, the first podman command you run after the boot will always reexec to create+enter the podman userns+mounts. All following commands can just join that userns and do not need rexxec part.

Now if you involve mount namesapces then you very likely completely break the podman storage management unless you know exactly what you are doing. Thus running podman commands in different mount namespaces is definitely not recommend.
Likely because the new mounts the joining the existing namesapces is not possible and it falls back to reexec part.

Using podman unshare is a good way to do this as you are in the correct environment with that and it should not have any negative side effects, well other than other applications would not be able to see that mount point unless you execute them inside podman unshare.

[tarun360]

Using podman unshare is a good way to do this as you are in the correct environment with that and it should not have any negative side effects, well other than other applications would not be able to see that mount point unless you execute them inside podman unshare.

Thanks @Luap99, that answers my question, I'll then just mount it both with & without podman unshare so that it's visible to normal as well as podman processes.

---

I had few more related questions, it would be very helpful for me if you could answer these:

Likely because the new mounts the joining the existing namesapces is not possible and it falls back to reexec part.

• I didn't understand why there are two podman processes (reexec) when a podman command is launched from a non-initial mount namespace (even when there were podman commands executed before it which have already created the "rootless podman user & mount namespace") ?

the first podman command you run after the boot will always reexec to create+enter the podman userns+mounts.

• Could you please share me the link of code where this logic of reexec is written?

[tarun360]

Hi @Luap99 , could you please take a look whenever you get time. Thanks!

[Luap99]

Sorry for the late reply, I was on vacation the last weeks.

The code most of this is in https://github.com/containers/podman/tree/main/pkg/rootless

I didn't understand why there are two podman processes (reexec) when a podman command is launched from a non-initial mount namespace (even when there were podman commands executed before it which have already created the "rootless podman user & mount namespace") ?

I don't know, the kernel has some restrictions so I assume that would be one of them, you can run it with strace -f to what is happening with the unshare/setns system calls.