Podman unable to run on Azure Kubernetes pods

[SusheelReddy00]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Briefly describe the problem you are having in a few paragraphs.
I am intending to run podman inside a kubernetes pod whose base image shall be debian/ubuntu
I have created a Dockerfile thus with base image as Ubuntu/Debian/mcr.microsoft.com/dotnet/sdk:6
Running the same in my local machine and creating a container from the Dockerfile with podman installed, lets me create a container and run inside the container which runs on WSL ubuntu. The same when run on Azure VM or Azure Kubernetes ndoes gives me the below error.

root@podman:/# podman run  -itd nginx
ERRO[0000] [graphdriver] prior storage driver aufs failed: mount /var/lib/containers/storage/aufs:/var/lib/containers/storage/aufs, f
lags: 0x1000: permission denied
Error: mount /var/lib/containers/storage/aufs:/var/lib/containers/storage/aufs, flags: 0x1000: permission denied

Steps to reproduce the issue:

1. Create a VM in Azure with OS as ubuntu

2. Build an image using below Dockerfile and create a container using this image. Try to run a new container inside the container.

3. PS: The commented lines in the Dockerfile was required when I ran in WSL. Kept it commented for reference. I am sure this has something to do with the storage driver based on some other issues I observed. I tried to specify drivers like Overlay, Overlayfs which do not exists in the underlying VM, but need guidance to how overcome.

FROM debian:11 as build
# Install the Credential Provider to configure the access
RUN apt-get update -y && apt-get install wget -y && apt-get install curl -y
RUN apt-get update && apt-get install ca-certificates -y &&  update-ca-certificates
RUN apt-get update -qq
RUN apt-get -qq -y install podman

#RUN mkdir -p /dev/net
#RUN mknod /dev/net/tun c 10 200
#RUN chmod 600 /dev/net/tun
#RUN sed -i '66d;67d;68d' /usr/share/containers/containers.conf

Describe the results you received:

root@podman:/# podman run  -itd nginx
ERRO[0000] [graphdriver] prior storage driver aufs failed: mount /var/lib/containers/storage/aufs:/var/lib/containers/storage/aufs, f
lags: 0x1000: permission denied
Error: mount /var/lib/containers/storage/aufs:/var/lib/containers/storage/aufs, flags: 0x1000: permission denied

Describe the results you expected:
Container is created and runs fine inside kubernetes pod that runs on Azure VM's

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:
Unable to obtain, however should be latest. Gives me same above error running version

(paste your output here)

Output of podman info:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes
Additional environment details (AWS, VirtualBox, physical, etc.):

[Luap99]

Without podman info it is hard to help you.
Looks like you use the aufs graph driver, I am not sure what the status with that one is but most people are using overalyfs so I recommend you try that one instead.

[SusheelReddy00]

@Luap99 I was able to overcome that issue by setting the security context in the pod to run in privileged mode. However like you mentioned, I cannot run with any storage driver as I get the below error.

root@pod:/# podman run --events-backend=file --cap-add=all --storage-driver=overlayfs nginx
Error: database storage graph driver "" does not match our storage graph driver "overlayfs": database configuration mismatch

Even if I run with aufs driver, I get the below error.

root@pod:/# podman run --events-backend=file --cap-add=all nginx
WARN[0000] Couldn't run auplink before unmount /var/lib/containers/storage/aufs/mnt/e23da07e064860950c4a514de81189e58689e867c35333daa33a3d1b5b
4d2735: exit status 22
Error: error mounting storage for container 593632dcc209966591989832dcc099b18ff58b66ff55586b2308fcf84cdcbb0d: error creating aufs mount to /va
r/lib/containers/storage/aufs/mnt/e23da07e064860950c4a514de81189e58689e867c35333daa33a3d1b5b4d2735: invalid argument

Posting the podman info that you requested below

root@pod:/# podman info
host:
  arch: amd64
  buildahVersion: 1.19.6
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    distribution: debian
    version: "11"
  eventLogger: journald
  hostname: pod
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.0-1074-azure
  linkmode: dynamic
  memFree: 685842432
  memTotal: 16786665472
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 2061h 14m 47.89s (Approximately 85.88 days)
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 0
    stopped: 11
  graphDriverName: aufs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: overlayfs
    Dirperm1 Supported: "false"
    Dirs: "17"
    Root Dir: /var/lib/containers/storage/aufs
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.15
  OsArch: linux/amd64
  Version: 3.0.1

Please have a look into it

[SusheelReddy00]

The logs in debug mode is attached for your reference

root@826afae58f5f:/# podman run --log-level=debug --events-backend=file docker.io/nginx
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level=debug --events-backend=file docker.io/nginx)
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.33.4 Annotations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[nproc=4194304:4194304] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:bridge NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{ImageBuildFormat:oci CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/libpod/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand: InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/var/lib/containers/storage/libpod StopTimeout:10 TmpDir:/run/libpod VolumePath:/var/lib/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/etc/cni/net.d/}}
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
INFO[0000] [graphdriver] using prior storage driver: aufs
DEBU[0000] Initializing event backend file
INFO[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] using runtime "/usr/bin/crun"
INFO[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument
INFO[0000] Found CNI network podman (type=ptp) at /etc/cni/net.d/87-podman-ptp.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 7
DEBU[0000] parsed reference into "[aufs@/var/lib/containers/storage+/run/containers/storage]docker.io/library/nginx:latest"
DEBU[0000] parsed reference into "[aufs@/var/lib/containers/storage+/run/containers/storage]docker.io/library/nginx:latest"
DEBU[0000] parsed reference into "[aufs@/var/lib/containers/storage+/run/containers/storage]@88736fe827391462a4db99252117f136b2b25d1d31719006326a437bb40cb12d"
DEBU[0000] exporting opaque data as blob "sha256:88736fe827391462a4db99252117f136b2b25d1d31719006326a437bb40cb12d"
DEBU[0000] parsed reference into "[aufs@/var/lib/containers/storage+/run/containers/storage]docker.io/library/nginx:latest"
DEBU[0000] parsed reference into "[aufs@/var/lib/containers/storage+/run/containers/storage]@88736fe827391462a4db99252117f136b2b25d1d31719006326a437bb40cb12d"
DEBU[0000] exporting opaque data as blob "sha256:88736fe827391462a4db99252117f136b2b25d1d31719006326a437bb40cb12d"
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading default seccomp profile
DEBU[0000] Allocated lock 3 for container ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149
DEBU[0000] parsed reference into "[aufs@/var/lib/containers/storage+/run/containers/storage]@88736fe827391462a4db99252117f136b2b25d1d31719006326a437bb40cb12d"
DEBU[0000] exporting opaque data as blob "sha256:88736fe827391462a4db99252117f136b2b25d1d31719006326a437bb40cb12d"
DEBU[0000] created container "ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149"
DEBU[0000] container "ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149" has work directory "/var/lib/containers/storage/aufs-containers/ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149/userdata"

DEBU[0000] container "ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149" has run directory "/run/containers/storage/aufs-containers/ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149/userdata"
DEBU[0000] Not attaching to stdin
DEBU[0000] Made network namespace at /run/netns/cni-3d8adaa9-f762-7411-f201-78bd93481f50 for container ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149
WARN[0000] Couldn't run auplink before unmount /var/lib/containers/storage/aufs/mnt/7db88c66c1e7927070f7c08813127ef05dad4d8fee600cd56b2bfa5e38c800d6: exit status 22
DEBU[0000] failed to mount container "ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149": error creating aufs mount to /var/lib/containers/storage/aufs/mnt/7db88c66c1e7927070f7c08813127ef05dad4d8fee600cd56b2bfa5e38c800d6: invalid argument
INFO[0000] Got pod network &{Name:interesting_bose Namespace:interesting_bose ID:ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149 NetNS:/run/netns/cni-3d8adaa9-f762-7411-f201-78bd93481f50 Networks:[{Name:podman Ifname:eth0}] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth: IpRanges:[]}] Aliases:map[]}
INFO[0000] About to add CNI network podman (type=ptp)
DEBU[0000] [0] CNI result: &{0.4.0 [{Name:veth4990b760 Mac:ee:4f:db:77:8b:ca Sandbox:} {Name:eth0 Mac:b6:da:1b:c5:1b:39 Sandbox:/run/netns/cni-3d8adaa9-f762-7411-f201-78bd93481f50}] [{Version:4 Interface:0xc0003581d8 Address:{IP:172.16.16.6 Mask:ffffff00} Gateway:172.16.16.1}] [{Dst:{IP:0.0.0.0 Mask:00000000} GW:}] {[] [] []}}
DEBU[0000] Tearing down network namespace at /run/netns/cni-3d8adaa9-f762-7411-f201-78bd93481f50 for container ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149
INFO[0000] Got pod network &{Name:interesting_bose Namespace:interesting_bose ID:ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149 NetNS:/run/netns/cni-3d8adaa9-f762-7411-f201-78bd93481f50 Networks:[{Name:podman Ifname:}] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth: IpRanges:[]}] Aliases:map[]}
INFO[0000] About to del CNI network podman (type=ptp)
DEBU[0000] Cleaning up container ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149 storage is already unmounted, skipping...
DEBU[0000] ExitCode msg: "error mounting storage for container ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149: error creating aufs mount to /var/lib/containers/storage/aufs/mnt/7db88c66c1e7927070f7c08813127ef05dad4d8fee600cd56b2bfa5e38c800d6: invalid argument"
Error: error mounting storage for container ccc718aef92879557d9a1cb0cc33fe2a673aef4c04180dc1c57bf979c1aa3149: error creating aufs mount to /var/lib/containers/storage/aufs/mnt/7db88c66c1e7927070f7c08813127ef05dad4d8fee600cd56b2bfa5e38c800d6: invalid argument

[rhatdan]

After changing the storage driver, execute podman system reset