Running rootless container inside systemd-nspawn

[carbolymer]

I'm trying to run a rootless container inside a systemd-nspawn container but I'm getting an eror:

[nixos@blueridge-nixos:~]$ podman run --rm -it bash
Error: crun: mount `proc` to `proc`: Operation not permitted: OCI permission denied

Where running it rootful (inside systemd-nspawn container) it works just fine:

[root@blueridge-nixos:~]# podman run --rm -it bash
bash-5.2# 
exit

I have found this tip for nesting containers:

--security-opt unmask=/proc/*

But I'm not sure how to unmask /proc for nspawn container:

[root@blueridge-nixos:~]# findmnt -R /proc
TARGET                              SOURCE                                                            FSTYPE OPTIONS
/proc                               proc                                                              proc   rw,nosuid,nodev,noexec,relatime
├─/proc/sys                         proc[/sys]                                                        proc   ro,nosuid,nodev,noexec,relatime
│ ├─/proc/sys/net                   proc[/sys/net]                                                    proc   rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/kernel/random/boot_id tmpfs[/.#proc-sys-kernel-random-boot-idcdd8ba1a97756623//deleted] tmpfs  ro,nosuid,nodev,noexec,size=3227928k,nr_inodes=819200,mode=755,uid=524288,gid=524288,inode64
├─/proc/sys/net                     proc[/sys/net]                                                    proc   rw,nosuid,nodev,noexec,relatime
├─/proc/acpi                        proc[/acpi]                                                       proc   ro,nosuid,nodev,noexec,relatime
├─/proc/asound                      proc[/asound]                                                     proc   ro,nosuid,nodev,noexec,relatime
├─/proc/bus                         proc[/bus]                                                        proc   ro,nosuid,nodev,noexec,relatime
├─/proc/fs                          proc[/fs]                                                         proc   ro,nosuid,nodev,noexec,relatime
├─/proc/irq                         proc[/irq]                                                        proc   ro,nosuid,nodev,noexec,relatime
├─/proc/scsi                        proc[/scsi]                                                       proc   ro,nosuid,nodev,noexec,relatime
├─/proc/sys/kernel/random/boot_id   tmpfs[/.#proc-sys-kernel-random-boot-idcdd8ba1a97756623//deleted] tmpfs  rw,nosuid,nodev,size=3227928k,nr_inodes=819200,mode=755,uid=524288,gid=524288,inode64
└─/proc/kmsg                        tmpfs[/.#proc-kmsg5993197c6f2ca582//deleted]                      tmpfs  rw,nosuid,nodev,size=3227928k,nr_inodes=819200,mode=755,uid=524288,gid=524288,inode64

[root@blueridge-nixos:~]# umount -f /proc/sys
umount: /proc/sys: must be superuser to unmount.

I've tried Bind=/proc in nspawn container configuration, but that just duplicates the amount of /proc mount.

podman system info output for nixos user

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /nix/store/afhlbaxjdw2jji60ljpi8qcy9clk1klf-conmon-2.1.8/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 95.28
    systemPercent: 3.05
    userPercent: 1.67
  cpus: 12
  databaseBackend: boltdb
  distribution:
    codename: tapir
    distribution: nixos
    version: "23.11"
  eventLogger: journald
  freeLocks: 2048
  hostname: blueridge-nixos
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.20.25.realtime1-1-rt-lts
  linkmode: dynamic
  logDriver: journald
  memFree: 353460224
  memTotal: 16526983168
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /nix/store/5krys5yppq1szas2zqxms4ja44880zc5-podman-4.7.2/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: Unknown
    path: /nix/store/5krys5yppq1szas2zqxms4ja44880zc5-podman-4.7.2/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: Unknown
    path: /nix/store/5i30ia7ilfnrn8fcf6g2sdnfci0j6w89-crun-1.12/bin/crun
    version: |-
      crun version 1.12
      commit: 1.12
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /nix/store/5krys5yppq1szas2zqxms4ja44880zc5-podman-4.7.2/libexec/podman/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 12475691008
  swapTotal: 17180913664
  uptime: 76h 52m 43.00s (Approximately 3.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/nixos/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/nixos/.local/share/containers/storage
  graphRootAllocated: 232340324352
  graphRootUsed: 152348033024
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/nixos/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 315532800
  BuiltTime: Tue Jan  1 01:00:00 1980
  GitCommit: ""
  GoVersion: go1.21.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

My nspawn configuration

$ cat /etc/systemd/nspawn/nixos.nspawn  
[Network]
VirtualEthernet=yes
Port=8022

[Files]
Bind=/dev/dri
Bind=/dev/fuse
Bind=/data/archive:/data/archive:idmap

[Exec]
SystemCallFilter=add_key keyctl bpf 

Additional devices in systemd unit configuration

$ cat /etc/systemd/system.control/[email protected]/50-DeviceAllow.conf 
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Service]
DeviceAllow=
DeviceAllow=/dev/fuse rwm
DeviceAllow=block-device-mapper rw
DeviceAllow=/dev/mapper/control rw
DeviceAllow=block-blkext rw
DeviceAllow=block-loop rw
DeviceAllow=/dev/loop-control rw
DeviceAllow=char-pts rw
DeviceAllow=/dev/net/tun rwm

ExecStart override

$ cat /etc/systemd/system/[email protected]/override.conf 
[Service]
ExecStart=
ExecStart=systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth --private-users=524288:7077888 --private-users-ownership=auto --settings=override --machine=%i
Environment="SYSTEMD_NSPAWN_UNIFIED_HIERARCHY=1"

Any tips would be appreciated.

[Luap99]

I think you can run add -v /proc:/proc tot he podman command to work around it. But this is not good if applications depend on a proper mounted /proc as this will mess things up.

[carbolymer]

Oh wow it works. Thanks!

$ podman run --rm -v /proc:/proc -it bash
bash-5.2# 

Would you mind explaining it a bit, what are implications of -v /proc:/proc, and what could be the risks there?

[Luap99]

By default we run in another pid namespace so your container process can not see all the processes on the host. In a pid namesapce the pid values are different so reading the values from /proc is not correct in this case as you are using /proc on the host.

Compare podman run alpine:latest ps aux, podman run -v /proc/:/proc/ alpine:latest ps aux and podman run -v /proc/:/proc/ --pid host alpine:latest ps aux, only the first and last one really give the correct view.

As far as problems go

$ podman run -v /proc/:/proc/  alpine:latest sh -c 'readlink /proc/self & echo $! && wait'
2
9151
$ podman run  alpine:latest sh -c 'readlink /proc/self & echo $! && wait'
2
2
$ podman run -v /proc/:/proc/ --pid host  alpine:latest sh -c 'readlink /proc/self & echo $! && wait'
9088
9088

Reading /proc/self returns the wrong pid so that can cause applications to do the wrong thing if they use files in /proc.