mount tmpfs to /dev: Permission denied

[dpkshetty]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. My gitlab-ci.yml (code snip which is failing...)

    - echo "Building container on ppc64le"
    - buildah --storage-driver=vfs build -t $APP .
    - echo "Build complete."

    - echo "Listing buildah images"
    - buildah --storage-driver=vfs images

    - echo "Push image to ${IMAGE_REGISTRY}:${TAG}"
    - buildah --storage-driver=vfs push --creds $quay_user:$quay_passwd $APP ${IMAGE_REGISTRY}:${TAG}-ppc64le

3. As you can see, I am using podman image to create a Dockerfile for my app. Link to Dockerfile
   https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/blob/main/Dockerfile

4. podman build fails as below...

$ podman --storage-driver=vfs build -t $APP .
time="2022-10-28T07:49:58Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
STEP 1/7: FROM registry.access.redhat.com/ubi8/python-38
Trying to pull registry.access.redhat.com/ubi8/python-38:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:d0d2e9042a028b50d899ac4c9a760d0a2305d515d0ad127b377a8e437dde0623
Copying blob sha256:d172799e5c5b9a62cce0219edd286f524a25155a018d7eda982c0d0706ed9fa9
Copying blob sha256:32af3091e80b9aa028a918579576c3a34d03a59746461d16990a9077ddb8541b
Copying blob sha256:9fb9feb1f76f717c6ae977c6d19b7aa7970602cbda1aea0072e9a669f0c94ad5
Copying config sha256:5e3e4648bb5c196e28e438f5ee535b7ffb36b0394dfab8ea1159d3ef5d3[26](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3241428683#L26)95a
Writing manifest to image destination
Storing signatures
STEP 2/7: USER 0
--> 74803e52454
STEP 3/7: ADD app-src /tmp/src
--> [35](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3241428683#L35)[38](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3241428683#L38)03cc6da
STEP 4/7: RUN chown -R 1001:0 /tmp/src
error running container: error from /usr/bin/crun creating container for [/bin/sh -c chown -R 1001:0 /tmp/src]: mount `tmpfs` to `/dev`: Permission denied
: exit status 1
Error: error building at STEP "RUN chown -R 1001:0 /tmp/src": error while running runtime: exit status 1
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

Describe the results you received:

podman build fail

Describe the results you expected:

podman build should succeed

Additional information you deem important (e.g. issue happens only occasionally):

Please note that in the above gitlab-ci.yml, if I replace podman with buildah (image and CLI), it works!!

Output of podman version:

(paste your output here)

Output of podman info:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

podman container is running in openshift as part of gitlab CI pipeline

[rhatdan]

Have you read https://www.redhat.com/sysadmin/podman-inside-kubernetes?

[dpkshetty]

I took a glance at that article
I am not an expert so a bit difficult to digest what exactly to try out... for some issue the article asks to change/edit host config files which can't be done in k8s/openshift context. Can u point me to the exact solution I should be using ?

[dpkshetty]

Does it help, if I say.. that the exact same buildah steps work successfully on OCP on ppc64le (IBM Power) cluster, but fails with the above error on OCP on x86 cluster! Strange!

[rhatdan]

Are you running both with the same security profile.?

@umohnani8 PTAL

[dpkshetty]

I am just executing my Gitlab pipeline and it should be running the runner pods with the same security profile. I am using the arch specific gitlab runners , installed via the respective operators. I have created custom SA and associated with anyuid perms on both and it works like a charm on ppc64le but fails on x86 cluster.

[dpkshetty]

@rhatdan my bad.. let me stand corrected... I was trying both buildah and podman methods - hence confusion
My apologies... above i gave buildah steps (from my gitlab CI pipeline) but error msg was from podman

Here is what is happening...

• buildah method works on ppc64le (per the steps listed at the top of this discussion) but fails on x86
• podman fails on both ppc64le and x86 clusters - as shown in error msg above
• Bottomline.. I am unable to build successfully for both ppc64le and x86 clusters as neither buildah works entirely nor podman

[rhatdan]

I moved this to a discussion, since this is not a bug in Podman, but a setup issue in OpenShift.

[rhatdan]

I would need to see the audit.log on both boxes to see if there are SELinux issues.

[dpkshetty]

@rhatdan is there a way to connect on Slack so that I can show you the issue. I have switched to using podman now. I don't use buildah at all in my CI pipeline. There is a different issue I am facing now... which a week back magically disappeared when i used --isolation chroot but it re-appears now! Having a slack discussion would be better, once root caused I can dump all the relevant info here for the sake of community good. Also I don't own these openshift env. I provision them via a internal portal of ours so might need some help (hence slack discussion) to get the relevant files like audit.log. Let me know.

FYI: I am using the same CI pipeline which runs on x86_64 and ppc64le arch. clusters.

Below is the failure (only seen on x86_64) while on ppc64le it works fine

Building container on x86
$ podman --storage-driver=vfs build --isolation chroot -t $APP -f ./Dockerfile --no-cache
time="2022-11-25T08:57:40Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
Error: cannot write uid_map: write /proc/25/uid_map: operation not permitted

Building container on ppc64le
$ podman --storage-driver=vfs build --isolation chroot -t $APP -f ./Dockerfile --no-cache
STEP 1/7: FROM registry.access.redhat.com/ubi7/python-38
Trying to pull registry.access.redhat.com/ubi7/python-38:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:3e12824404c8f9ece1102b36177afcc00b6a175bee6feb71746d4925b0c0b5f4
Copying blob sha256:58910fdfeb79e1c95a750d913adf6d0f9d224b2517a1544ec4f9112a2758668e
Copying blob sha256:7b827a621e5acd17204d927aa4306a362d2e7ee4f875eb84973e4e062de2184b
Copying blob sha256:019f02db9974ecbb2caa2a3f0[29](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L29)4fc145add8f86f97f86a[30](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L30)fe542006f9db79a
Copying config sha256:9edfd1fa2ba506edea3570ce51fed30964ff66457d9e1de07d1443a357886f4a
Writing manifest to image destination
Storing signatures
STEP 2/7: USER 0
--> 62973fda67a
STEP 3/7: ADD app-src /tmp/src
--> 666c5dccdbd
STEP 4/7: RUN chown -R 1001:0 /tmp/src
--> 4d7077bd9ff
STEP 5/7: USER 1001
--> d309dca4[31](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L31)c
STEP 6/7: RUN /usr/libexec/s2i/assemble
---> Installing application source ...
---> Installing dependencies ...
Collecting flask
  Downloading Flask-2.2.2-py3-none-any.whl (101 kB)
Collecting Jinja2>=3.0
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting itsdangerous>=2.0
  Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting click>=8.0
  Downloading click-8.1.3-py3-none-any.whl (96 kB)
Collecting importlib-metadata>=3.6.0
  Downloading importlib_metadata-5.1.0-py3-none-any.whl (21 kB)
Collecting Werkzeug>=2.2.2
  Downloading Werkzeug-2.2.2-py3-none-any.whl (2[32](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L32) kB)
Collecting zipp>=0.5
  Downloading zipp-3.10.0-py3-none-any.whl (6.2 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.1.tar.gz (18 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Using legacy 'setup.py install' for MarkupSafe, since package 'wheel' is not installed.
Installing collected packages: zipp, MarkupSafe, Werkzeug, Jinja2, itsdangerous, importlib-metadata, click, flask
    Running setup.py install for MarkupSafe: started
    Running setup.py install for MarkupSafe: finished with status 'done'
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.2 click-8.1.3 flask-2.2.2 importlib-metadata-5.1.0 itsdangerous-2.1.2 zipp-3.10.0
WARNING: You are using pip version 21.3.1; however, version 22.3.1 is available.
You should consider upgrading via the '/opt/app-root/bin/python3.8 -m pip install --upgrade pip' command.
--> 57b8570b5d6
STEP 7/7: CMD /usr/libexec/s2i/run
COMMIT dpk-pyflask
--> b8abe0c3d60
Successfully tagged localhost/dpk-pyflask:latest
b8abe0c3d6065a01ccaf[35](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L35)7fef29621b135e7589fe73b87[51](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L51)293aa7[63](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L63)bf7[67](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L67)af
$ echo "Build complete."
Build complete

[dpkshetty]

I have referred to podman troubleshooting guide and tried to disable SELinux, but that didn't help.

$ echo "Building container on x86"
Building container on x86
$ podman --storage-driver=vfs build --isolation chroot --security-opt label=disable -t $APP -f ./Dockerfile --no-cache
time="2022-11-25T09:10:10Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
Error: cannot write uid_map: write /proc/27/uid_map: operation not permitted
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

Again, this works fine on ppc64le cluster

[dpkshetty]

@rhatdan I have referred to the podman in k8s but I don't see this exact problem referenced there. I still went ahead and tried --userns option. x86 it still fails the same way and on ppc64le i think that option (syscall?) isn't supported looks like (might need another bug for that ?)

Building container on x86
$ podman --storage-driver=vfs build --isolation chroot --security-opt label=disable --userns=keep-id -t $APP -f ./Dockerfile --no-cache
time="2022-11-[25](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378790103#L25)T09:19:09Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
Error: cannot write uid_map: write /proc/24/uid_map: operation not permitted
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

Building container on ppc64le
$ podman --storage-driver=vfs build --isolation chroot --security-opt label=disable --userns=keep-id -t $APP -f ./Dockerfile --no-cache
Error: checking user namespace: stat keep-id: no such file or directory
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

[rhatdan]

Don't use --userns=keep-id, I am not sure this is supported for podman build, only podman run.

[giuseppe]

Error: cannot write uid_map: write /proc/24/uid_map: operation not permitted

this seems like either CAP_SETUID/CAP_SETGID are not available in the container, or the newuidmap/newgidmap do not have file capabilities.

[dpkshetty]

Any suggestions on what to do next ?
The exact same gitab CI yaml works fine for a ppc64le cluster I have. But fails with the above error on a x86_64 cluster.

[rhatdan]

getcap /usr/bin/newuidmap /usr/bin/newgidmap
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep

[dpkshetty]

@rhatdan @giuseppe thanks for your support. I tried to put some debug prints as suggested above and here are the dumps for x86 and ppc64le case. My CI yaml file is here - x86 fails, ppc64le works fine.

x86 - fails
https://gist.github.com/dpkshetty/050fde23b679d123d5cf5ae6afe177a1

ppc64le - succeeds
https://gist.github.com/dpkshetty/f59c85fea6e5d69dc82e1b8b464aa114