Getting permission denied error when running podman 3.1.1 with root user without privilege set to true in k8

[SahilChaudhary25]

Error: mount /home/podman/.local/share/containers/storage/overlay:/home/podman/.local/share/containers/storage/overlay, flags: 0x1000: permission denied

I am getting above error while running podman info command on podman 3.1.1 rootless in k8

uname -a
Linux test-deployment-d78b66b6d-vd2x8 5.4.0-1036-azure #38~18.04.1-Ubuntu SMP Wed Jan 6 18:26:30 UTC 2021 x86_64 Linux

[mheon]

@rhatdan Podman-in-container issue - you want to take it?

[rhatdan]

What is the actual command you are executing. We have been doing a lot of testing on this, and can successfully run podman rootless within a container, Running rootfull podman within a container requires --privileged still. (Or --cap-add SYS_ADMIN) But we are working to fix this.

[SahilChaudhary25]

@rhatdan Thanks for reply, I tried with simple podman info command. Rootfull is working fine, but when i am trying with rootless without privileged but setting SYS_ADMIN, i am getting above error

[SahilChaudhary25]

@rhatdan @mheon Please find podman related information
podmanDebugging.txt

[rhatdan]

@giuseppe Looks like the same problem on podman 3.1.1.?

[giuseppe]

could you check if #10079 works any better?

Can you please share the full command used to create the parent container?

[SahilChaudhary25]

@giuseppe Thanks for reply. I have used following command
kubectl apply -f deployment.yaml
podman-deployment.txt

Docker file

`FROM adoptopenjdk/openjdk11:alpine-slim

RUN (apk add cni-plugins --update-cache --repository http://dl-3.alpinelinux.org/alpine/edge/community/ --allow-untrusted)
RUN (apk add libslirp --update-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/community/ --allow-untrusted)
RUN (apk add podman --update-cache --repository http://dl-3.alpinelinux.org/alpine/edge/community/ --allow-untrusted)

VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers

ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf
ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf /home/podman/.config/containers/containers.conf

chmod containers.conf and adjust storage.conf to enable Fuse storage.

RUN chmod 644 /etc/containers/containers.conf; sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage./a "/var/lib/shared",' -e 's|^mountopt[[:space:]]=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock; touch /var/lib/shared/vfs-images/images.lock; touch /var/lib/shared/vfs-layers/layers.lock

adduser podman && echo 'podman:165536:65536' >> /etc/subuid && echo 'podman:165536:65536' >> /etc/subgid

ENV _CONTAINERS_USERNS_CONFIGURED=""`

[mohitseth-nitrr]

@rhatdan @giuseppe Can you please confirm if it is podman bug or configuration issue ?

[rhatdan]

It will be in podman 3.2 release. Second release candidate should go out next week.

[SahilChaudhary25]

@rhatdan Any tentative date for release into alpine ?

[rhatdan]

We don't release into alpine, so I would figure whoever packages for alpine, will do so after we declare release. I would figure this will happen before end of month.

[SahilChaudhary25]

@rhatdan It was planned to release podman 3.2 by end of May. Is there any change in plan ?

[mheon]

We are expecting a release today or tomorrow - awaiting final confirmation from several teams that depend on us that the new release is fully working.

[mohitseth-nitrr]

@mheon @rhatdan @giuseppe I see that podman v3.2.0 is released 2 days back which could potentially resolve this issue. Any timeline on when we can have it on alpine repository https://pkgs.alpinelinux.org/packages?name=podman

[SahilChaudhary25]

@mheon @rhatdan hello, After 3.2.0 podman release, I am not even able to pull image and getting below error when run as root but privilege flag is false. Can you please let me know if i am missing some configuration or this will work in this way. It would be better to make decision for team.

ERROR:

cc: @mohitseth-nitrr

[SahilChaudhary25]

podman ps --log-level=debug

INFO[0000] podman filtering at log level debug
DEBU[0000] Called ps.PersistentPreRunE(podman ps --log-level=debug)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/etc/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: imagestore=/var/lib/shared
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] overlay: skip_mount_home=true
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 25
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
DEBU[0000] Called ps.PersistentPostRunE(podman ps --log-level=debug)

[SahilChaudhary25]

Storage.conf

storage.txt

Container.conf

[containers]
netns="host"
userns="host"
ipcns="host"
utsns="host"
cgroupns="host"
cgroups="disabled"
log_driver = "k8s-file"
[engine]
cgroup_manager = "cgroupfs"
events_logger="file"
runtime="crun"

[mheon]

@rhatdan PTAL

[SahilChaudhary25]

If it looks config issue then can you please provide sample config, NOTE: This worked fine root & privilege's true