Cannot run rootless podman inside rootless podman using ubi9:minimal

[Jokester761]

I am trying to follow the instructions of https://developers.redhat.com/articles/2024/10/01/build-container-images-openshift-using-podman-gitlab-runner#set_up_and_configure_gitlab_runner to run a rootless podman as an unprivileged user but I can't seem to get it working. I don't understand the underlying technologies to see what I could be doing wrong. Is this something that should work? Thank you in advance.

$ cat Containerfile
FROM registry.access.redhat.com/ubi9/ubi-minimal

ARG USER_HOME_DIR="/home/user"

ENV HOME=${USER_HOME_DIR}
ENV BUILDAH_ISOLATION=chroot

RUN microdnf --disableplugin=subscription-manager install -y openssl git tar which shadow-utils bash zsh wget jq podman buildah skopeo; \
    microdnf update -y ; \
    microdnf clean all ; \
    mkdir -p ${USER_HOME_DIR} ; \
    chgrp -R 0 /home ; \
    #
    # Setup for root-less podman
    #
    setcap cap_setuid+ep /usr/bin/newuidmap ; \
    setcap cap_setgid+ep /usr/bin/newgidmap ; \
    touch /etc/subgid /etc/subuid ; \
    chmod -R g=u /etc/passwd /etc/group /etc/subuid /etc/subgid /home
WORKDIR ${HOME}
COPY entrypoint.sh .
RUN chmod +x entrypoint.sh
ENTRYPOINT [ "./entrypoint.sh" ]

$ cat entrypoint.sh
#!/usr/bin/env bash

if [ ! -d "${HOME}" ]
then
  mkdir -p "${HOME}"
fi

mkdir -p ${HOME}/.config/containers
# Below line is not needed for ocp 4.15+ clusters as fuse-overlayfs will be made available
(echo '[storage]';echo 'driver = "vfs"') > ${HOME}/.config/containers/storage.conf

if ! whoami &> /dev/null
then
  if [ -w /etc/passwd ]
  then
    echo "${USER_NAME:-user}:x:$(id -u):0:${USER_NAME:-user} user:${HOME}:/bin/bash" >> /etc/passwd
    echo "${USER_NAME:-user}:x:$(id -u):" >> /etc/group
  fi
fi
USER=$(whoami)
START_ID=$(( $(id -u)+1 ))
echo "${USER}:${START_ID}:32768" > /etc/subuid
echo "${USER}:${START_ID}:32768" > /etc/subgid

exec "$@"

Building the image

$ podman build -t rht:rootless

Testing the image:

Works as root user inside the container:

$ podman run --rm -it --cap-add=setuid,setgid,chown,sys_admin rht:rootless /bin/bash
bash-5.1# podman version
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.
Client:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.22.7 (Red Hat 1.22.7-2.el9_5)
Built:        Fri Nov 15 08:49:31 2024
OS/Arch:      linux/amd64
bash-5.1# cat /etc/sub{u,g}id ; cat /etc/passwd | grep user ; cat /etc/group | grep user ; getcap /usr/bin/newuidmap ; getcap /usr/bin/newgidmap ; podman info | grep -i driver
root:1:32768
root:1:32768
users:x:100:
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.
  logDriver: k8s-file
  graphDriverName: overlay
bash-5.1#
exit

Fails as unprivileged user inside the container:

$ podman run --rm -it -u 1234 --cap-add=setuid,setgid,chown,sys_admin rht:rootless /bin/bash
bash-5.1$ podman version
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.
ERRO[0000] running `/usr/bin/newuidmap 28 0 1234 1 1 1235 32768 32769 1235 32768`: newuidmap: write to uid_map failed: Invalid argument
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1
bash-5.1$ cat /etc/sub{u,g}id ; cat /etc/passwd | grep user ; cat /etc/group | grep user ; getcap /usr/bin/newuidmap ; getcap /usr/bin/newgidmap ; podman info | grep -i driver
1234:1235:32768
1234:1235:32768
1234:*:1234:0:container user:/home/user:/bin/sh
users:x:100:
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.
ERRO[0000] running `/usr/bin/newuidmap 20 0 1234 1 1 1235 32768 32769 1235 32768`: newuidmap: write to uid_map failed: Invalid argument
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

Fails as unprivileged user inside the container with --privileged flag

$ podman run --rm -it -u 1234 --privileged --cap-add=setuid,setgid,chown,sys_admin,mknod --device=/dev/fuse --security-opt label=disable --se
curity-opt unmask=ALL rht:rootless /bin/bash
bash-5.1$ id
uid=1234(1234) gid=0(root) groups=0(root)
bash-5.1$ podman version
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV
1_WARNING` to hide this warning.
ERRO[0000] running `/usr/bin/newuidmap 21 0 1234 1 1 1235 32768 32769 1235 32768`: newuidmap: write to uid_map failed: Invalid argument
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1
bash-5.1$ cat /etc/sub{u,g}id ; cat /etc/passwd | grep user ; cat /etc/group | grep user ; getcap /usr/bin/newuidmap ; getcap /usr/bin/newgidmap ; podman info | grep -i driver
1234:1235:32768
1234:1235:32768
1234:*:1234:0:container user:/home/user:/bin/sh
users:x:100:
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV
1_WARNING` to hide this warning.
ERRO[0000] running `/usr/bin/newuidmap 39 0 1234 1 1 1235 32768 32769 1235 32768`: newuidmap: write to uid_map failed: Invalid argument
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

Fails as unprivileged user when run by root

# podman run --rm -it -u 1234 --cap-add=setuid,setgid,chown,sys_admin rht:rootless /bin/bash
bash-5.1$ podman version
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV
1_WARNING` to hide this warning.
ERRO[0000] running `/usr/bin/newuidmap 20 0 1234 1 1 1235 32768 32769 1235 32768`: newuidmap: write to uid_map failed: Invalid argument
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1
bash-5.1$ cat /etc/sub{u,g}id ; cat /etc/passwd | grep user ; cat /etc/group | grep user ; getcap /usr/bin/newuidmap ; getcap /usr/bin/newgidmap ; podman info | grep -i driver
1234:1235:32768
1234:1235:32768
1234:*:1234:0:container user:/home/user:/bin/sh
users:x:100:
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV
1_WARNING` to hide this warning.
ERRO[0000] running `/usr/bin/newuidmap 38 0 1234 1 1 1235 32768 32769 1235 32768`: newuidmap: write to uid_map failed: Invalid argument
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

h3. References
https://developers.redhat.com/articles/2024/10/01/build-container-images-openshift-using-podman-gitlab-runner#set_up_and_configure_gitlab_runner
https://www.redhat.com/en/blog/podman-inside-container

[Jokester761]

I can see that podman run --security-opt label=disable --user podman quay.io/podman/stable podman version works though. Maybe I can learn something from that container image.

[Jokester761]

Unfortunately I couldn't glean anything comparing https://github.com/containers/image_build/blob/main/podman/Containerfile with my containerfile.

[rhatdan]

You are trying to run a container within a container using 1234:1235:2147483646

Does the rooless user have access to this many UIDs?

[Jokester761]

Thank you for the quick response, my rootless user didn't have access to that many UIDs. I reduced the number to 32768 because I have 65535 according to getsubids. The results were the same though.

$ getsubids $USER
0: jokester761 362144 65536

[rhatdan]

Does it work in rootful mode?

[rhatdan]

Also did you try with a --privileged outer container?

[Jokester761]

Does it work in rootful mode?

It does not work if I run the pod as root in the host and an unprivileged user in the container.

Also did you try with a --privileged outer container?

It does not work if I add --privileged to the podman run command.