podman run does not work in Eclipse Che (OpenShift Dev Spaces)

[cgruver]

Issue Description

Eclipse Che version 7.63 (Upstream of OpenShift Dev Spaces)
OpenShift 4.12 (OKD - Community supported OpenShift)
Fedora CoreOS 37.20230218.3.0
Kernel: 6.1.11-200.fc37.x86_64

Podman in an Eclipse Che workspace can pull images, and build images. It fails to run containers.

podman run -it registry.access.redhat.com/ubi9/ubi-minimal

Output:

Trying to pull registry.access.redhat.com/ubi9/ubi-minimal:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob ba958a445f00 done  
Copying config 3135bd90aa done  
Writing manifest to image destination
Storing signatures
WARN[0007] failed to set net.ipv6.conf.default.accept_dad sysctl: open /proc/sys/net/ipv6/conf/default/accept_dad: read-only file system 
ERRO[0008] Unmounting partially created network namespace for container 59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81: failed to unmount NS: at /tmp/podman-run-1000700000/netns/netns-65f8f083-aa9b-feed-2dad-d6cb44ffb4aa: permission denied 
ERRO[0008] Preparing container 59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81: /usr/bin/slirp4netns failed: "open(\"/dev/net/tun\"): No such file or directory\nWARNING: Support for seccomp is experimental\nWARNING: Support for IPv6 is experimental\nchild failed(1)\nWARNING: Support for seccomp is experimental\nWARNING: Support for IPv6 is experimental\n" 
Error: error unmounting storage for container 59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81 after network create failure: error unmounting container 59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81 SHM mount /home/user/.local/share/containers/storage/vfs-containers/59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81/userdata/shm: permission denied

Steps to reproduce the issue

Steps to reproduce the issue

1. Enable the container-build scc in Eclipse Che
2. Create a workspace from: https://github.com/cgruver/che-vscode-demo
3. Open a terminal in VS Code
4. run: podman run -it registry.access.redhat.com/ubi9/ubi-minimal

Describe the results you received

Trying to pull registry.access.redhat.com/ubi9/ubi-minimal:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob ba958a445f00 done  
Copying config 3135bd90aa done  
Writing manifest to image destination
Storing signatures
WARN[0007] failed to set net.ipv6.conf.default.accept_dad sysctl: open /proc/sys/net/ipv6/conf/default/accept_dad: read-only file system 
ERRO[0008] Unmounting partially created network namespace for container 59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81: failed to unmount NS: at /tmp/podman-run-1000700000/netns/netns-65f8f083-aa9b-feed-2dad-d6cb44ffb4aa: permission denied 
ERRO[0008] Preparing container 59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81: /usr/bin/slirp4netns failed: "open(\"/dev/net/tun\"): No such file or directory\nWARNING: Support for seccomp is experimental\nWARNING: Support for IPv6 is experimental\nchild failed(1)\nWARNING: Support for seccomp is experimental\nWARNING: Support for IPv6 is experimental\n" 
Error: error unmounting storage for container 59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81 after network create failure: error unmounting container 59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81 SHM mount /home/user/.local/share/containers/storage/vfs-containers/59274435d86c3fccf94f825a9f1b00c1e434dff955711393ac022bff8bca8b81/userdata/shm: permission denied

Describe the results you expected

Shell into a running container

podman info output

host:
  arch: amd64
  buildahVersion: 1.27.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.4-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: 56561007b6a59ea175ee9a67384639721499e160'
  cpuUtilization:
    idlePercent: 93.9
    systemPercent: 1.55
    userPercent: 4.54
  cpus: 12
  distribution:
    distribution: '"rhel"'
    version: "9.1"
  eventLogger: file
  hostname: workspacea074194c43fb406c-749d889f7f-mrqh4
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 0
      size: 1
    - container_id: 1
      host_id: 20000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000700000
      size: 1
    - container_id: 1
      host_id: 20000
      size: 65536
  kernel: 6.1.11-200.fc37.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 18093432832
  memTotal: 63201439744
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.5-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1000700000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.el9_0.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 24h 52m 53.00s (Approximately 1.00 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 858388410368
  graphRootUsed: 41806852096
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1000700000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1675767401
  BuiltTime: Tue Feb  7 10:56:41 2023
  GitCommit: ""
  GoVersion: go1.18.9
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Eclipse Che 7.63
OKD 4.12
Fedora CoreOS 37.20230218.3.0
Kernel: 6.1.11-200.fc37.x86_64

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

I assume this is running podman inside a container, it is not going to work unless the outer container gives the necessary permission.
see https://www.redhat.com/sysadmin/podman-inside-container and https://www.redhat.com/sysadmin/podman-inside-kubernetes

[cgruver]

@Luap99

podman pull ... works and podman build ... works.

What is preventing run from working?

[cgruver]

Update:

cat << EOF > containers.conf                              
[containers]
netns="host"
ipcns="host"
default_sysctls = [] # Workaround
EOF

podman run -it registry.access.redhat.com/ubi9/ubi-minimal

Error: crun: mount `proc` to `/proc`: Permission denied: OCI permission denied

[cgruver]

@Luap99 Did you intend to close this issue?

[Luap99]

I converted it to a discussion now, I do not think this is a bug in podman rather your environment set-up which is causing problems.

[cgruver]

Thanks, I chatted with @umohnani8 about this on Slack.

Builds work fine, but we're using driver = "vfs" in storage.conf.

We'd like to get run working. But if we have to disable selinux on the compute nodes, that's a non-started as this is on OpenShift.

[cgruver]

I'm going to try the user namespace route and see if I can get that to work on OpenShift without having to do major surgery to it that will upset InfoSec... :-)

[rhatdan]

The mount of /proc is not an SELinux issue. The problem is you are trying to mount /proc with different options then the parent /proc, which is causing the issue. You need to mount the /proc from the host into the container.

@giuseppe knows the issue.

[giuseppe]

the kernel requires a proc mount to be fully visible before it allows mounting a new procfs. Inside your container the /proc mount has other mounts on the top, so we are not allowed to mount a fresh proc inside a nested container

[cgruver]

@giuseppe @rhatdan @umohnani8 I feel like I've almost got this working, but for the /proc mount.

I'm trying to take advantage of userns support in OpenShift 4.12

Here's where I'm at so far:

Create the fuse device plugin: Note: I forked that project and modified it to build from ubi-minimal: https://github.com/cgruver/fuse-device-plugin. wip branch

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fuse-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fuse-device-plugin-ds
  template:
    metadata:
      labels:
        name: fuse-device-plugin-ds
    spec:
      nodeSelector:
        node-role.kubernetes.io/worker: ""
      hostNetwork: true
      containers:
      - image: quay.io/cgruver0/che/fuse-device-plugin:v1.1
        name: fuse-device-plugin-ctr
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
          - name: device-plugin
            mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
      imagePullSecrets:
        - name: registry-secret

Enable userns for the containers user on the compute nodes:

cat << EOF | butane | oc apply -f -
variant: openshift
version: 4.12.0
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: subuid-subgid
storage:
  files:
  - path: /etc/subuid
    mode: 0644
    overwrite: true
    contents:
      inline: |
        core:100000:65536
        containers:200000:268435456
  - path: /etc/subgid
    mode: 0644
    overwrite: true
    contents:
      inline: |
        core:100000:65536
        containers:200000:268435456
EOF

When I create a pod, I get the error:

Error: crun: mount `proc` to `/proc`: Permission denied: OCI permission denied

Because I don't know what to do about /proc.

Can you offer any guidance?

Cheers