podman-in-kubernetes kernel overlay

[ooraini]

I'm following the article:
https://www.redhat.com/sysadmin/podman-inside-kubernetes

With the recent release of RHEL 8.6 and supporting kernel overlay, I was wondering if I can get away with rootful podman in kubernetes without fuse.

Unfortunately, I was not able to get it to work. I can see the overlay driver

lsmod | grep overlay
overlay               139264  15

But podman reports no support, and I can't run any container.

store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.7
  Built: 1650546866
  BuiltTime: Thu Apr 21 13:14:26 2022
  GitCommit: ""
  GoVersion: go1.16.15
  OsArch: linux/amd64
  Version: 3.4.7

Is there a configuration that I'm missing? or is this a limitation of kernel overlay?

Thanks,

[ooraini]

I have managed to get podman-in-podman to work. I modified the Dockerfile for podman:stable

# stable/Dockerfile
#
# Build a Podman container image from the latest
# stable version of Podman on the Fedoras Updates System.
# https://bodhi.fedoraproject.org/updates/?search=podman
# This image can be used to create a secured container
# that runs safely with privileges within the container.
#
FROM docker.io/almalinux/8-base:8.6

# Don't include container-selinux and remove
# directories used by yum that are just taking
# up space.
RUN dnf -y update; rpm --restore shadow-utils 2>/dev/null; \
yum -y install podman crun --exclude container-selinux; \
rm -rf /var/cache /var/log/dnf* /var/log/yum.*

RUN useradd podman; \
echo -e "podman:1:999\npodman:1001:64535" > /etc/subuid; \
echo -e "podman:1:999\npodman:1001:64535" > /etc/subgid;

ADD https://raw.githubusercontent.com/containers/podman/main/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf
ADD https://raw.githubusercontent.com/containers/podman/main/contrib/podmanimage/stable/podman-containers.conf /home/podman/.config/containers/containers.conf

RUN mkdir -p /home/podman/.local/share/containers; chown podman:podman -R /home/podman

# Note VOLUME options must always happen after the chown call above
# RUN commands can not modify existing volumes
VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers

# chmod containers.conf and adjust storage.conf to enable Fuse storage.
RUN chmod 644 /etc/containers/containers.conf; sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock; touch /var/lib/shared/vfs-images/images.lock; touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

Started the container with bash:

odman run -it --cap-add=sys_admin,mknod mypodman bash

And commented the following lines in storage.conf:

#mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
#mountopt = "nodev,fsync=0"

podman run hello-world works.

podman info:

store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1652193619
  BuiltTime: Tue May 10 14:40:19 2022
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.2

podman run logs:

[root@ce1e0860af9e /]# podman run --log-level=debug hello-world
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level=debug hello-world) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/etc/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /run/containers/storage       
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/libpod                    
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: imagestore=/var/lib/shared          
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is not being used 
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: failed to set opaque flag on middle layer: lsetxattr /var/lib/containers/storage/overlay/opaque-bug-check1656618213/l2/d: operation not permitted 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 97             
DEBU[0000] Pulling image hello-world (policy: missing)  
DEBU[0000] Looking up image "hello-world" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "hello-world" ...                     
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf" 
DEBU[0000] Trying "docker.io/library/hello-world:latest" ... 
DEBU[0000] Trying "localhost/hello-world:latest" ...    
DEBU[0000] Trying "registry.fedoraproject.org/hello-world:latest" ... 
DEBU[0000] Trying "registry.access.redhat.com/hello-world:latest" ... 
DEBU[0000] Trying "registry.centos.org/hello-world:latest" ... 
DEBU[0000] Trying "docker.io/library/hello-world:latest" ... 
DEBU[0000] Trying "docker.io/library/hello-world:latest" ... 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate docker.io/library/hello-world:latest for hello-world 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared]docker.io/library/hello-world:latest" 
DEBU[0000] Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) 
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/hello-world:latest...
DEBU[0000] Copying source image //hello-world:latest to destination image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared]docker.io/library/hello-world:latest 
DEBU[0000] Trying to access "docker.io/library/hello-world:latest" 
DEBU[0000] No credentials matching docker.io/library/hello-world found in /run/containers/0/auth.json 
DEBU[0000] No credentials matching docker.io/library/hello-world found in /root/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/hello-world found in /root/.docker/config.json 
DEBU[0000] No credentials matching docker.io/library/hello-world found in /root/.dockercfg 
DEBU[0000] No credentials for docker.io/library/hello-world found 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for docker.io/library/hello-world:latest, using built-in default file:///var/lib/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0001] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0001] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=registry.docker.io 
DEBU[0005] GET https://registry-1.docker.io/v2/library/hello-world/manifests/latest 
DEBU[0006] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0006] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0006] Source is a manifest list; copying (only) instance sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4 for current system 
DEBU[0006] GET https://registry-1.docker.io/v2/library/hello-world/manifests/sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4 
DEBU[0007] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0007] IsRunningImageAllowed for image docker:docker.io/library/hello-world:latest 
DEBU[0007]  Using default policy section                
DEBU[0007]  Requirement 0: allowed                      
DEBU[0007] Overall: allowed                             
DEBU[0007] Downloading /v2/library/hello-world/blobs/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412 
DEBU[0007] GET https://registry-1.docker.io/v2/library/hello-world/blobs/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412 
Getting image source signatures
DEBU[0008] Reading /var/lib/containers/sigstore/library/hello-world@sha256=f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4/signature-1 
DEBU[0008] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0008] ... will first try using the original manifest unmodified 
DEBU[0008] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0008] Downloading /v2/library/hello-world/blobs/sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54 
DEBU[0008] GET https://registry-1.docker.io/v2/library/hello-world/blobs/sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54 
DEBU[0009] Detected compression format gzip             
DEBU[0009] Using original blob without modification     
Copying blob 2db29710123e done  
Copying blob 2db29710123e done  
DEBU[0009] No compression detected                      
DEBU[0009] Using original blob without modification     
Copying config feb5d9fea6 done  
Writing manifest to image destination
Storing signatures
DEBU[0009] setting image creation date to 2021-09-23 23:47:57.442225064 +0000 UTC 
DEBU[0009] created new image ID "feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0009] saved image metadata "{\"signatures-sizes\":{\"sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4\":[]}}" 
DEBU[0009] added name "docker.io/library/hello-world:latest" to image "feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0009] Pulled candidate docker.io/library/hello-world:latest successfully 
DEBU[0009] Looking up image "feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" in local containers storage 
DEBU[0009] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0009] Trying "feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" ... 
DEBU[0009] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared]@feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0009] Found image "feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" as "feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" in local containers storage 
DEBU[0009] Found image "feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" as "feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared]@feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412) 
DEBU[0009] Looking up image "hello-world" in local containers storage 
DEBU[0009] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0009] Trying "hello-world" ...                     
DEBU[0009] Trying "docker.io/library/hello-world:latest" ... 
DEBU[0009] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared]@feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0009] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage 
DEBU[0009] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared]@feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412) 
DEBU[0009] Inspecting image feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412 
DEBU[0009] exporting opaque data as blob "sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0009] exporting opaque data as blob "sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0009] exporting opaque data as blob "sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0009] Inspecting image feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412 
DEBU[0009] Inspecting image feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412 
DEBU[0009] Inspecting image feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412 
DEBU[0009] using systemd mode: false                    
DEBU[0009] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
INFO[0009] Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host 
DEBU[0009] Allocated lock 0 for container cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca 
DEBU[0009] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared]@feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0009] exporting opaque data as blob "sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412" 
DEBU[0009] Created container "cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca" 
DEBU[0009] Container "cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca" has work directory "/var/lib/containers/storage/overlay-containers/cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca/userdata" 
DEBU[0009] Container "cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca" has run directory "/run/containers/storage/overlay-containers/cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca/userdata" 
DEBU[0009] Not attaching to stdin                       
DEBU[0009] [graphdriver] trying provided driver "overlay" 
DEBU[0009] overlay: imagestore=/var/lib/shared          
DEBU[0009] Cached value indicated that overlay is supported 
DEBU[0009] Cached value indicated that metacopy is not being used 
DEBU[0009] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0009] overlay: mount_data=,lowerdir=/var/lib/containers/storage/overlay/l/PQ7LEAG65JEHAVDYNU5AB3XQ42,upperdir=/var/lib/containers/storage/overlay/886953fcfa49e5a6c5ac53eff4115ffe0c21cea8f1fa0e7c8fab4b95dd6eacae/diff,workdir=/var/lib/containers/storage/overlay/886953fcfa49e5a6c5ac53eff4115ffe0c21cea8f1fa0e7c8fab4b95dd6eacae/work 
DEBU[0009] Mounted container "cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca" at "/var/lib/containers/storage/overlay/886953fcfa49e5a6c5ac53eff4115ffe0c21cea8f1fa0e7c8fab4b95dd6eacae/merged" 
DEBU[0009] Created root filesystem for container cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca at /var/lib/containers/storage/overlay/886953fcfa49e5a6c5ac53eff4115ffe0c21cea8f1fa0e7c8fab4b95dd6eacae/merged 
DEBU[0009] Network configuration does not support host.containers.internal address 
DEBU[0009] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0009] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0009] Workdir "/" resolved to host path "/var/lib/containers/storage/overlay/886953fcfa49e5a6c5ac53eff4115ffe0c21cea8f1fa0e7c8fab4b95dd6eacae/merged" 
DEBU[0009] Created OCI spec for container cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca at /var/lib/containers/storage/overlay-containers/cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca/userdata/config.json 
DEBU[0009] /usr/bin/conmon messages will be logged to syslog 
DEBU[0009] Running with no Cgroups                      
DEBU[0009] running conmon: /usr/bin/conmon               args="[--api-version 1 -c cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca -u cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca/userdata -p /run/containers/storage/overlay-containers/cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca/userdata/pidfile -n boring_bose --exit-dir /run/libpod/exits --full-attach -l k8s-file:/var/lib/containers/storage/overlay-containers/cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca/userdata/ctr.log --log-level debug --syslog --runtime-arg --cgroup-manager --runtime-arg disabled --conmon-pidfile /run/containers/storage/overlay-containers/cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.imagestore=/var/lib/shared --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0009] Received: 183                                
INFO[0009] Got Conmon PID as 180                        
DEBU[0009] Created container cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca in OCI runtime 
DEBU[0009] Attaching to container cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca 
DEBU[0009] Starting container cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca with command [/hello] 
DEBU[0009] Started container cfb804f99b262e165b3e9364b5063fcff2e22e2fe56be9114ebed6fe3d4d66ca 

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

INFO[0009] Received shutdown.Stop(), terminating!        PID=119
DEBU[0009] Enabling signal proxying                     
DEBU[0009] Called run.PersistentPostRunE(podman run --log-level=debug hello-world) 

[ooraini]

Or delete the file /etc/containers/storage.conf.

[ooraini]

For Kubernetes:

apiVersion: v1
kind: Pod
metadata:
 name: no-priv-rootful
spec:
 containers:
   - name: no-priv-rootful
     image: docker.io/omaraloraini/podman:latest
     args:
       - sleep
       - "1000000"
     volumeMounts:
       - name: no-overlay
         mountPath: /var/lib/containers/storage
     securityContext:
       seLinuxOptions:
         type: spc_t
       capabilities:
         add:
           - "SYS_ADMIN"
           - "MKNOD"
           - "SYS_CHROOT"
           - "SETFCAP"
 volumes:
    - name: no-overlay
      emptyDir: {}

I was getting some SELinux denials, so I useed the spc_t label. The podman-in-podman article mentions this:

Disable SELinux: The --security-opt label=disable option tells the host's Podman to disable SElinux separation for the container. SELinux does not allow containerized processes to mount all of the file systems required to run inside a container.

[ooraini]

I have managed to get userspace networking in rootful podman inside Kubernetes to work.

containers.conf

[root@no-priv-rootful /]# cat /etc/containers/containers.conf 
[containers]
default_sysctls=[]
netns="slirp4netns"
userns="host"
ipcns="host"
utsns="host"
cgroupns="host"
cgroups="disabled"
log_driver = "k8s-file"
[engine]
cgroup_manager = "cgroupfs"
events_logger="file"
runtime="crun"

podman.yaml:

apiVersion: v1
kind: Pod
metadata:
 name: no-priv-rootful
spec:
 containers:
   - name: no-priv-rootful
     image: docker.io/omaraloraini/podman:latest
     args:
       - sleep
       - "1000000"
     volumeMounts:
       - name: no-overlay
         mountPath: /var/lib/containers/storage
       - mountPath: /dev/net/tun
         name: dev-tun
     securityContext:
       seLinuxOptions:
         type: spc_t
       capabilities:
         add:
           - "SYS_ADMIN"
           - "MKNOD"
           - "SYS_CHROOT"
           - "SETFCAP"
           - "NET_ADMIN"
 volumes:
    - name: no-overlay
      emptyDir: {}
    - hostPath:
        path: /dev/net/tun
        type: CharDevice
      name: dev-tun

I had to add NET_ADMIN cap to create a tap devices, without it I got ioctl TUNSETIFF tun: Operation not permitted.

The motivation for this was running testcontainers in Gitlab CI and using the podman service as a replacement for DIND.

I'm getting the warning:
WARN[0000] failed to set net.ipv6.conf.default.accept_dad sysctl: open /proc/sys/net/ipv6/conf/default/accept_dad: read-only file system
But everything seems to work.