Nested podman containers: failed to mount /sys (operation not permitted) #10958

npes-95 · 2021-07-16T10:06:29Z

npes-95
Jul 16, 2021

Hi,

I want to containerise a TeamCity buildagent which runs CI jobs for a Yocto-based project I'm working on. The project's build system uses podman (which effectively wraps Yocto), which works fine when running on a host machine, but runs into permission issues when running within a container. The outer (i.e. buildagent) container is setup as described in https://www.redhat.com/sysadmin/podman-inside-container, started with systemd.

The inner container (i.e. the build system container, run under a non-root user) is unable to mount /sys due to "OCI permission denied" when run (see error message). Building the container image works in this scenario. I haven't seen this issue mentioned elsewhere online, so I'm hoping I could get some insight into it here :) Please let me know if there's any additional information I can provide.

Error message:
Error: mount /systo/sys: Operation not permitted: OCI permission denied

Outer container run options:

--privileged

Inner container run options:

--userns keep-id
--group-add keep-groups
--privileged
--device /some/device
-v /some/build/volume:Z
-w /workdir

Podman info inside buildagent container:

host:
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: file
  hostname: ef2d6d912c10
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 6666
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 6666
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 5.12.14-300.fc34.x86_64
  linkmode: dynamic
  memFree: 12464877568
  memTotal: 33506308096
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/6666/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 25413279744
  swapTotal: 25413279744
  uptime: 2h 9m 58.45s (Approximately 0.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/buildagent/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.4
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.4
        using FUSE kernel interface version 7.31
  graphRoot: /home/buildagent/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 20
  runRoot: /run/user/6666/containers
  volumePath: /home/buildagent/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.2
  Built: 1624664959
  BuiltTime: Fri Jun 25 23:49:19 2021
  GitCommit: ""
  GoVersion: go1.16.4
  OsArch: linux/amd64
  Version: 3.2.2

Podman info on host:

host:
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 131072
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 131072
  kernel: 5.12.14-300.fc34.x86_64
  linkmode: dynamic
  memFree: 12421197824
  memTotal: 33506308096
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 25413279744
  swapTotal: 25413279744
  uptime: 2h 11m 51.16s (Approximately 0.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/nick/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/nick/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 133
  runRoot: /run/user/1000/containers
  volumePath: /home/nick/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.2
  Built: 1624664959
  BuiltTime: Sat Jun 26 01:49:19 2021
  GitCommit: ""
  GoVersion: go1.16.4
  OsArch: linux/amd64
  Version: 3.2.2

rhatdan · 2021-07-16T11:44:47Z

rhatdan
Jul 16, 2021
Maintainer

Do you have a reproducer? Command line arguments you are using. These are very difficult to diagnose without the Containerfile or commandline you are using.

3 replies

npes-95 Jul 16, 2021
Author

Here's the Containerfile for the outer container:

FROM fedora:34

# SYSTEM CONFIG

# install packages required for native building and testing
COPY packages.txt /tmp/
RUN dnf install -y $(cat /tmp/packages.txt)

# unmask logind (so there will be a user session launched at login)
RUN systemctl unmask systemd-logind

# setup for running podman in podman
RUN dnf reinstall -y shadow-utils

VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers

ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf

RUN chmod 644 /etc/containers/containers.conf; sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock; touch /var/lib/shared/vfs-images/images.lock; touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

# clean package manager cache
RUN dnf clean all


# USER CONFIG

# create buildagent user & enable lingering (loginctl enable-linger will not work as systemd isn't started)
RUN groupadd -f -g 6666 buildagent && useradd -l -m -u 6666 -g 6666 -G buildagent,audio,pulse,pulse-access buildagent
RUN mkdir -p /var/lib/systemd/linger && touch /var/lib/systemd/linger/buildagent

# copy TeamCity agent to image, unzip and give access to buildagent
ADD --chown=6666:6666 <internal-server>/buildAgent.zip /opt/buildagent/
RUN unzip /opt/buildagent/buildAgent.zip -d /opt/buildagent/ && rm -f /opt/buildagent/buildAgent.zip
RUN chown -R 6666:6666 /opt/buildagent

# buildagent runner properties (specifies TC server address, runner name, etc...)
COPY --chown=6666:6666 buildAgent.properties /opt/buildagent/conf/

# ssh keys are required to connect to the TC server
COPY --chown=6666:6666 .ssh/* /home/buildagent/.ssh/

# buildagent runner is launched as a user service
COPY --chown=6666:6666 agent.service /home/buildagent/.config/systemd/user/

# custom pulseaudio config (see file for explanation)
COPY --chown=6666:6666 .pulse/* /home/buildagent/.pulse/

# user container config
ADD --chown=6666:6666 https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf /home/buildagent/.config/containers/containers.conf
RUN echo buildagent:10000:65536 > /etc/subuid; \
    echo buildagent:10000:65536 > /etc/subgid;


# switch to buildagent to enable user services (they will be launched when buildagent logs in)
USER buildagent
RUN systemctl --user enable pulseaudio && systemctl --user enable agent

# RUNTIME

# start systemd as PID 1
USER root
CMD [ "/sbin/init" ]

Inner container file:

FROM ubuntu:16.04

ENV LANG en_US.UTF-8

ARG USER=bob-the-builder
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update -y \
    && apt-get install -y \
               bc build-essential chrpath cmake cpio curl device-tree-compiler diffstat gawk git gnupg g++ iptables \
               iputils-ping libbz2-dev libncurses5-dev libssl-dev libusb-1.0-0-dev libzip-dev \
               locales nano pigz pkg-config python python3 sudo texinfo tmux \
               u-boot-tools vim wget

RUN echo "LC_ALL=\"en_US.UTF-8\"" >> /etc/default/locale
RUN echo "LC_CTYPE=\"en_US.UTF-8\"" >> /etc/default/locale
RUN echo "LANGUAGE=\"en_US.UTF-8\"" >> /etc/default/locale
RUN locale-gen en_US.UTF-8

RUN curl https://storage.googleapis.com/git-repo-downloads/repo-1 > /bin/repo
RUN chmod a+x /bin/repo

RUN mkdir /mfgtools
RUN git clone --branch uuu_1.4.43 https://github.com/NXPmicro/mfgtools.git /mfgtools
RUN mkdir -p /mfgtools/build
WORKDIR /mfgtools/build
RUN cmake ..
RUN make
RUN make install
RUN rm -r /mfgtools

RUN echo "$USER   ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

npes-95 Jul 16, 2021
Author

The inner image is built using podman build --build-arg USER=$(whoami), and run as a non-root user in the outer image:

podman run --rm --userns keep-id \
--group-add keep-groups \
--device /dev/bus/usb \
--privileged \
-v ${sourceDir}:/sourceDir:Z  \
-v ${scriptDir}:/scripts:Z \
-v ${workdir}:/workdir:Z \
-w workdir \
inner-image echo hello

(note: the mounted volumes are specific to the build system and are available in the user's home directory, they can be replaced by an arbitrary directory).

The outer image is built using podman build and run with podman run --privileged as a non root user on the host.

npes-95 Jul 16, 2021
Author

Debug output here:

INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run --rm --userns keep-id --group-add keep-groups --device /dev/bus/usb --privileged -v ../test:/test:Z --log-level debug a8b97141c3c1 echo hello)
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/etc/containers/containers.conf"
DEBU[0000] Merged system config "/home/buildagent/.config/containers/containers.conf"
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/buildagent/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/buildagent/.local/share/containers/storage
DEBU[0000] Using run root /run/user/6666/containers
DEBU[0000] Using static dir /home/buildagent/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/6666/libpod/tmp
DEBU[0000] Using volume path /home/buildagent/.local/share/containers/storage/volumes
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=overlayfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 25
DEBU[0000] Pulling image a8b97141c3c1 (policy: missing)
DEBU[0000] Looking up image "a8b97141c3c1" in local containers storage
DEBU[0000] Trying "a8b97141c3c1" ...
DEBU[0000] parsed reference into "[overlay@/home/buildagent/.local/share/containers/storage+/run/user/6666/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] Found image "a8b97141c3c1" as "a8b97141c3c1" in local containers storage
DEBU[0000] Looking up image "a8b97141c3c1" in local containers storage
DEBU[0000] Trying "a8b97141c3c1" ...
DEBU[0000] parsed reference into "[overlay@/home/buildagent/.local/share/containers/storage+/run/user/6666/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] Found image "a8b97141c3c1" as "a8b97141c3c1" in local containers storage
DEBU[0000] User mount ../test:/test options [Z]
DEBU[0000] Looking up image "a8b97141c3c1" in local containers storage
DEBU[0000] Trying "a8b97141c3c1" ...
DEBU[0000] parsed reference into "[overlay@/home/buildagent/.local/share/containers/storage+/run/user/6666/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] Found image "a8b97141c3c1" as "a8b97141c3c1" in local containers storage
DEBU[0000] Inspecting image a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4
DEBU[0000] exporting opaque data as blob "sha256:a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] exporting opaque data as blob "sha256:a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] exporting opaque data as blob "sha256:a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] parsed reference into "[overlay@/home/buildagent/.local/share/containers/storage+/run/user/6666/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@f6b6783a9fa5c4261869e33503641d15064a3920cb5377d03b96db3d0d6e2a93"
DEBU[0000] exporting opaque data as blob "sha256:f6b6783a9fa5c4261869e33503641d15064a3920cb5377d03b96db3d0d6e2a93"
DEBU[0000] Looking up image "a8b97141c3c1" in local containers storage
DEBU[0000] Trying "a8b97141c3c1" ...
DEBU[0000] parsed reference into "[overlay@/home/buildagent/.local/share/containers/storage+/run/user/6666/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] Found image "a8b97141c3c1" as "a8b97141c3c1" in local containers storage
DEBU[0000] Inspecting image a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4
DEBU[0000] exporting opaque data as blob "sha256:a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] exporting opaque data as blob "sha256:a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] exporting opaque data as blob "sha256:a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] parsed reference into "[overlay@/home/buildagent/.local/share/containers/storage+/run/user/6666/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@f6b6783a9fa5c4261869e33503641d15064a3920cb5377d03b96db3d0d6e2a93"
DEBU[0000] exporting opaque data as blob "sha256:f6b6783a9fa5c4261869e33503641d15064a3920cb5377d03b96db3d0d6e2a93"
DEBU[0000] Inspecting image a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4
DEBU[0000] User mount /proc:/proc options []
DEBU[0000] using systemd mode: false
DEBU[0000] CDI HasDevice Error: Could not find device "/dev/bus/usb"
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
INFO[0000] Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host
DEBU[0000] Adding mount /dev/autofs
DEBU[0000] Adding mount /dev/bus/usb/001/001
DEBU[0000] Adding mount /dev/bus/usb/001/002
DEBU[0000] Adding mount /dev/bus/usb/001/003
DEBU[0000] Adding mount /dev/bus/usb/001/005
DEBU[0000] Adding mount /dev/bus/usb/001/006
DEBU[0000] Adding mount /dev/bus/usb/002/001
DEBU[0000] Adding mount /dev/dri/renderD128
DEBU[0000] Adding mount /dev/full
DEBU[0000] Adding mount /dev/fuse
DEBU[0000] Adding mount /dev/kmsg
DEBU[0000] Adding mount /dev/kvm
DEBU[0000] Adding mount /dev/net/tun
DEBU[0000] Adding mount /dev/null
DEBU[0000] Adding mount /dev/random
DEBU[0000] Adding mount /dev/rfkill
DEBU[0000] Adding mount /dev/urandom
DEBU[0000] Adding mount /dev/vboxdrvu
DEBU[0000] Adding mount /dev/vhost-net
DEBU[0000] Adding mount /dev/vhost-vsock
DEBU[0000] Adding mount /dev/wmi/dell-smbios
DEBU[0000] Adding mount /dev/zero
DEBU[0000] Adding mount /dev
DEBU[0000] Adding mount /dev/pts
DEBU[0000] Adding mount /sys
DEBU[0000] Adding mount /dev/mqueue
DEBU[0000] Allocated lock 0 for container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43
DEBU[0000] parsed reference into "[overlay@/home/buildagent/.local/share/containers/storage+/run/user/6666/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] exporting opaque data as blob "sha256:a8b97141c3c18b911c16764827badc0b137c2c42103dc0d58eeadffe3ffecaf4"
DEBU[0000] created container "80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43"
DEBU[0000] container "80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43" has work directory "/home/buildagent/.local/share/containers/storage/overlay-containers/80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43/userdata"
DEBU[0000] container "80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43" has run directory "/run/user/6666/containers/overlay-containers/80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43/userdata"
DEBU[0000] Not attaching to stdin
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=overlayfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] overlay: mount_data=,lowerdir=/home/buildagent/.local/share/containers/storage/overlay/l/YXJRF6IES2HFMZYNMXRFH5VUQH:/home/buildagent/.local/share/containers/storage/overlay/l/TWFFG3XMBDHVBXKEGD7UVOJP63:/home/buildagent/.local/share/containers/storage/overlay/l/B4M723FAOZLLNWSIU7O5QQQPRL:/home/buildagent/.local/share/containers/storage/overlay/l/XYS6NBNJD5NUWO5ANDNGGRO52D:/home/buildagent/.local/share/containers/storage/overlay/l/OTTUS5FXWF5AJD5ZBAUWTXFHHU:/home/buildagent/.local/share/containers/storage/overlay/l/NZFETYBJLXDCSMPEO3XA53KC4X:/home/buildagent/.local/share/containers/storage/overlay/l/Q5PTBWMQKPIRLIB4NYEWNLZLQ2:/home/buildagent/.local/share/containers/storage/overlay/l/RJTGP5MRBF4WL72A6FV56XJWY2:/home/buildagent/.local/share/containers/storage/overlay/l/ZESCWWJE2GG3WZBLG47OXAWQJP:/home/buildagent/.local/share/containers/storage/overlay/l/Z7DZJ5QIO73725X5NIKTQYUHG7:/home/buildagent/.local/share/containers/storage/overlay/l/PR3IQXCBY4KQ526NX6RJO73VLF:/home/buildagent/.local/share/containers/storage/overlay/l/EIC7GFITPWIOQX5RHR7KC23KUS:/home/buildagent/.local/share/containers/storage/overlay/l/ZUZHEEQT5XEBP5OUQZ7HF3YY26:/home/buildagent/.local/share/containers/storage/overlay/l/HMESWV3OURNKPMXHAR6S2QI7BM:/home/buildagent/.local/share/containers/storage/overlay/l/3HITTYUL62RK7ZM22QZRS4S3XL:/home/buildagent/.local/share/containers/storage/overlay/l/4HZHATIXGMK7JP43WUAQY2Q36T:/home/buildagent/.local/share/containers/storage/overlay/l/TEF5TBTCAJPL4UMJQPQJCGIF5I:/home/buildagent/.local/share/containers/storage/overlay/l/IAUKPOZNEGEDX6B76SHVM46F2M:/home/buildagent/.local/share/containers/storage/overlay/l/WLDXEH32GWCC3HTE6SIR7A5J6T,upperdir=/home/buildagent/.local/share/containers/storage/overlay/4ccb4c4e2bfe621e7abfe6a6f087d538205bdd44e6bad1d2887b07e8d12d1447/diff,workdir=/home/buildagent/.local/share/containers/storage/overlay/4ccb4c4e2bfe621e7abfe6a6f087d538205bdd44e6bad1d2887b07e8d12d1447/work,volatile
DEBU[0000] mounted container "80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43" at "/home/buildagent/.local/share/containers/storage/overlay/4ccb4c4e2bfe621e7abfe6a6f087d538205bdd44e6bad1d2887b07e8d12d1447/merged"
DEBU[0000] Created root filesystem for container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43 at /home/buildagent/.local/share/containers/storage/overlay/4ccb4c4e2bfe621e7abfe6a6f087d538205bdd44e6bad1d2887b07e8d12d1447/merged
DEBU[0000] Workdir "/mfgtools/build" resolved to host path "/home/buildagent/.local/share/containers/storage/overlay/4ccb4c4e2bfe621e7abfe6a6f087d538205bdd44e6bad1d2887b07e8d12d1447/merged/mfgtools/build"
DEBU[0000] network configuration does not support host.containers.internal address
DEBU[0000] Modifying container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43 /etc/passwd
DEBU[0000] Modifying container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43 /etc/group
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] set root propagation to "rslave"
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Created OCI spec for container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43 at /home/buildagent/.local/share/containers/storage/overlay-containers/80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] Running with no CGroups
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43 -u 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43 -r /usr/bin/crun -b /home/buildagent/.local/share/containers/storage/overlay-containers/80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43/userdata -p /run/user/6666/containers/overlay-containers/80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43/userdata/pidfile -n zealous_chaum --exit-dir /run/user/6666/libpod/tmp/exits --full-attach -l k8s-file:/home/buildagent/.local/share/containers/storage/overlay-containers/80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43/userdata/ctr.log --log-level debug --syslog --runtime-arg --cgroup-manager --runtime-arg disabled --conmon-pidfile /run/user/6666/containers/overlay-containers/80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/buildagent/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/6666/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/6666/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1
DEBU[0000] Cleaning up container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] unmounted container "80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43"
DEBU[0000] Removing container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43
DEBU[0000] Removing all exec sessions for container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43
DEBU[0000] Cleaning up container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43 storage is already unmounted, skipping...
DEBU[0000] Container 80500839d1cfe525f336935ddaad91f89c000ad510bf9ed922c0dec9be6fff43 storage is already unmounted, skipping...
DEBU[0000] ExitCode msg: "mount `/sys` to `/sys`: operation not permitted: oci permission denied"
Error: mount `/sys` to `/sys`: Operation not permitted: OCI permission denied

rhatdan · 2021-07-16T17:36:36Z

rhatdan
Jul 16, 2021
Maintainer

Strange could you try --security-opt umask=all on the outer podman?

1 reply

npes-95 Jul 16, 2021
Author

No luck I'm afraid, the issue persists.

rhatdan · 2021-07-16T18:10:01Z

rhatdan
Jul 16, 2021
Maintainer

The outer podman is running as non root? Does it have enough UIDs to run the inner in a separate user namespace?

6 replies

rhatdan Jul 16, 2021
Maintainer

I have an idea that the --userns=keep-id is causing issues.

npes-95 Jul 16, 2021
Author

Removing --userns didn't solve the issue, but removing the --privileged and --device flags worked! I think for the buildagent use case this is acceptable as I only included those flags to allow access to devices on the host (e.g. for flashing hardware, which wouldn't be applicable for CI builds). The issue now is that the build system can't access /proc/<pid>/stat for "process I/O statistics" - would this be solved simply by mounting /proc?

npes-95 Jul 21, 2021
Author

@rhatdan I've been experimenting a bit more, and I've found that the inner container is mounting the outer container's /proc, which then leads to issues accessing /proc/<pid>/stat since the inner container's PIDs are run in their own namespace. Interestingly, when you run the inner container directly on the host, this does not happen, the container PIDs are correctly mapped to /proc. Is this expected behaviour?

rhatdan Jul 21, 2021
Maintainer

Yes why are you mount -v /proc:/proc?

npes-95 Jul 21, 2021
Author

I haven't got /proc mounted in this scenario, apologies for being unclear. What I mean is that calling ls /proc from the inner container returns an identical result to calling ls /proc from the outer container. In contrast running the same command on the host and on the inner container (not nested) will return different results.

Nested podman containers: failed to mount /sys (operation not permitted) #10958

Uh oh!

npes-95 Jul 16, 2021

Replies: 3 comments · 10 replies

Uh oh!

rhatdan Jul 16, 2021 Maintainer

Uh oh!

npes-95 Jul 16, 2021 Author

Uh oh!

Uh oh!

npes-95 Jul 16, 2021 Author

Uh oh!

npes-95 Jul 16, 2021 Author

Uh oh!

rhatdan Jul 16, 2021 Maintainer

Uh oh!

npes-95 Jul 16, 2021 Author

Uh oh!

rhatdan Jul 16, 2021 Maintainer

Uh oh!

rhatdan Jul 16, 2021 Maintainer

Uh oh!

npes-95 Jul 16, 2021 Author

Uh oh!

npes-95 Jul 21, 2021 Author

Uh oh!

rhatdan Jul 21, 2021 Maintainer

Uh oh!

Uh oh!

npes-95 Jul 21, 2021 Author

npes-95
Jul 16, 2021

Replies: 3 comments 10 replies

rhatdan
Jul 16, 2021
Maintainer

npes-95 Jul 16, 2021
Author

npes-95 Jul 16, 2021
Author

npes-95 Jul 16, 2021
Author

rhatdan
Jul 16, 2021
Maintainer

npes-95 Jul 16, 2021
Author

rhatdan
Jul 16, 2021
Maintainer

rhatdan Jul 16, 2021
Maintainer

npes-95 Jul 16, 2021
Author

npes-95 Jul 21, 2021
Author

rhatdan Jul 21, 2021
Maintainer

npes-95 Jul 21, 2021
Author