[Bug]: Cannot access root directory of containerized process through /proc/[pid]/root on rootless containers

[agorgl]

Issue Description

Normally, for every linux process we can access its view of the filesystem (including namespaces and the set of per-process mounts) through /proc pseudo-filesystem specifically through /proc/[pid]/root for a given pid.

I noticed that this applies to the containerized applications too, with the following exception:

When running a rootful podman container with e.g. sudo podman run --rm -it ubuntu,
I can find the PID of the shell inside the container from the host (using pgrep/htop or whatever) and I can peek into the container filesystem using sudo ls -la /proc/<PID>/root/

When running a rootless podman container with e.g. podman run --rm -it ubuntu (no sudo)
I can find the PID of the shell inside the container from the host and but I CANNOT peek into the container filesystem from relevant /proc/<PID>/root/ directory, with a Permission denied error.

Using either ls -la /proc/<PID>/root/ or sudo ls -la /proc/<PID>/root/, leads to:

ls: cannot access '/proc/<PID>/root/': Permission denied.

The /proc/<PID>/root symlink itself seems to be user owned and fully accessible by everyone because when executing
ls -la /proc/<PID>/root (without the trailing slash) it gives me:

lrwxrwxrwx 1 myuser myuser 0 Jan 11 14:17 /proc/<PID>/root -> /

I cannot understand why this happens. Tried various tests by creating namespaces (even nested) using the more primitive unshare command to create non-root mount namespaces with like unshare -Umrfp --mount-proc /bin/bash and in all my tests I could access the process' view of the filesystem through /proc fs.

Seems something special is happening in rootless podman containers that only someone with deep knowledge in podman's or runc's internals can explain.

As another note, I can still use nsenter to enter rootless container's the namespace and peek into the filesystem, although an explanation why access through /proc is not working is still preferred.

Steps to reproduce the issue

Steps to reproduce the issue

1. Run any rootless container
2. Find the pid of the process inside the container from the host
3. Try to access /proc/<pid>/root/ from the host`

Describe the results you received

Permission denied error.

Describe the results you expected

To be able to access the /proc/<pid>/root/ symlink/directory

podman info output

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.5-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: c9f7f19eb82d5b8151fc3ba7fbbccf03fdcd0325'
  cpuUtilization:
    idlePercent: 98.01
    systemPercent: 0.53
    userPercent: 1.46
  cpus: 16
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: laptop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.4-zen2-1-zen
  linkmode: dynamic
  logDriver: journald
  memFree: 8129323008
  memTotal: 14501855232
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.7.2-1
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/myuser/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/myuser/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 4h 52m 26.00s (Approximately 0.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/myuser/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/myuser/.local/share/containers/storage
  graphRootAllocated: 236677234688
  graphRootUsed: 44494618624
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/myuser/1000/containers
  volumePath: /home/myuser/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668983565
  BuiltTime: Mon Nov 21 00:32:45 2022
  GitCommit: 814b7b003cc630bf6ab188274706c383f9fb9915-dirty
  GoVersion: go1.19.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[mheon]

@giuseppe PTAL - I imagine it's related to the rootless user namespace?

[giuseppe]

am I aware only of FUSE causing this kind of issues.

From podman info it seems you are not using fuse-overlayfs though.

If I repeat your same test:

$ podman run --rm -ti ubuntu sh
/ # echo hello > foo

and from another shell:

$ cat /proc/1831929/root/foo 
hello

[agorgl]

I have both fuse-overlayfs package installed and it is in my PATH:

❯ pacman -Qi fuse-overlayfs
Name            : fuse-overlayfs
Version         : 1.10-1
Description     : FUSE implementation of overlayfs
Architecture    : x86_64
URL             : https://github.com/containers/fuse-overlayfs
Licenses        : GPL3
Groups          : None
Provides        : None
Depends On      : fuse3
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 108.03 KiB
Packager        : Morten Linderud <[email protected]>
Build Date      : Sun 04 Dec 2022 04:13:32 PM EET
Install Date    : Sun 04 Dec 2022 11:26:52 PM EET
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

❯ which fuse-overlayfs
/usr/bin/fuse-overlayfs

[agorgl]

Seems that uninstalling the fuse-overlayfs package fixes the problem, although at this point given the info on the arch wiki, I'm unsure if fuse-overlayfs is required,recommended or deprecated for rootless containers.

[giuseppe]

I think it is better to use the native overlay driver in the kernel than fuse-overlayfs.

fuse-overlayfs was needed when overlay was limited to the root user.

[agorgl]

How does podman pick what driver to use? I suppose it checks if fuse-overlayfs is available in PATH and if not it fallbacks to the in-kernel overlayfs driver?

[agorgl]

Another dimension:

I mainly started this bug / topic because I liked the way this worked and I was trying to replicate the functionality using more primitive tools. My goal was to create a script / program / tool that can create a mount namespace, mount an encrypted filesystem like gocryptfs inside it, and launch a process, constraining the visibility of the mount to that process only / disallowing any other user processes to see it.

Initially I implemented this by creating a mount namespace using unshare -cm that mounted/unlocked the encrypted filesystem before launching the process that should be able to see the decrypted fs. After that I found that my user in the host could see all the processes' view of their filesystem+private mounts through /proc and I was searching of a way to disable / bypass this, until I stumbled uppon the above behavior on rootless podman containers which gave me some hope that is possible.

I suppose this was only possible due to a bug in fuse-overlayfs. Is there any other way to constraint a mount namespace so that other user processes in the host may not be able to peek inside it (not without root permissions at least)?

I know that linux namespaces / containers where designed to isolate the containerized process from the host and not vise-versa.
Although I see interesting properties on the opposite especially when trying to isolate a process from other userland processes running on root / default namespace.

Any hint, direction would be really appreciated

[giuseppe]

fuse-overlayfs does it on purpose otherwise other users within the user namespace are not able to use the file system.

You may want to take a look at the FUSE options allow_other and allow_root

[agorgl]

Hmm, so running a rootless container with fuse-overlayfs package existing on the system leads to this root mount inside the container:

# mount
fuse-overlayfs on / type fuse.fuse-overlayfs (rw,noatime,user_id=0,group_id=0,default_permissions,allow_other)

and running a rootless container with fuse-overlayfs package missing from the system leads to this:

# mount
overlay on / type overlay (rw,relatime,lowerdir=/home/user/.local/share/containers/storage/overlay/l/OVNL5KCQBU3NV52QHNHEYRAMOW,upperdir=/home/user/.local/share/containers/storage/overlay/d03af8712c7753c7fc6c89d797071ddefd8ff5aa78a35318fcb07811db600ee5/diff,workdir=/home/user/.local/share/containers/storage/overlay/d03af8712c7753c7fc6c89d797071ddefd8ff5aa78a35318fcb07811db600ee5/work,index=off,metacopy=off,volatile,userxattr)

I'm not really sure if either default_permissions or allow_other option lead to this behavior because according to the manpage:

       default_permissions
              This option instructs the kernel to perform its own
              permission check instead of deferring all permission
              checking to the filesystem. The check by the kernel is
              done in addition to any permission checks by the
              filesystem, and both have to succeed for an operation to
              be allowed. The kernel performs a standard UNIX permission
              check (based on mode bits and ownership of the directory
              entry, and uid/gid of the client).

              This mount option is activated implicitly if the
              filesystem enables ACL support during the initial feature
              negotiation when opening the device fd. In this case, the
              kernel performs both ACL and standard unix permission
              checking.

              Filesystems that do not implement any permission checking
              should generally add this option internally.

       allow_other
              This option overrides the security measure restricting
              file access to the filesystem owner, so that all users
              (including root) can access the files.

[agorgl]

On another note, launching a namespaced shell using:

unshare -Umrfp --mount-proc bash

and mounting an encrypted directory inside this shell using

gocryptfs -allow_other encrypted/ decrypted/

(-allow_other flag in gocryptfs adds allow_other and default_permissions options in decrypted directory mount)
actually accomplishes what I was trying to do! The decrypted directory inside the namespaced shell is NOT accessible through
/proc/<pid>/root/path/to/decrypted/ even with root privileges!

Given the option descriptions above though I'm still not sure why this happens and if I can depend on this being a feature and not a bug.
I suppose I should take this question to fuse developers?

[giuseppe]

that is done on purpose I think because the IDs from the host cannot be mapped inside the user namespace. So the FUSE file system will see requests coming from the unknown ID for every ID not mapped inside the user namespace.

You may try forcing access to the FUSE file system with (echo Y > /sys/module/fuse/parameters/allow_sys_admin_access) but use at your own risk.

[lpgc]

thanks for this guys I have been trying to figure out why /proc/pid/root/... is not accessible!

although I am a little confused since 'allow_other' "should" ostensibly permit access via /proc/.../root but it does not, nor does allow_root, in fact setting mountopts in storage.conf does not appear to be applied subsequently to containers started.

this is an issue for tools in the host namespaces that attempt to introspect on a containers env... for example 'jcmd' on the Java platform cannot 'attach' to a containerized JVM because of this issue.

I was not able to successfully configure either 'allow_others' or 'allow_root' to determine if this resolves this issue, however in configuring podman to use the kernel overlay instead of overlayfs-fuse the access issue appears to be resolved.

[lpgc]

I have to add that I believe this to be a BUG - certainly from the POV of the OS and contracts it defines & implements.

I suspect that the issue with overlayfs-fuse is to do with uid mapping from the host -> container user ns, it is either that or an issue with mount ns mapping ...

obviously the "workaround" is to not use overlayfs-fuse and employ the kernel overlay driver which works!