Possible bug? Rootless with fuse-overlayfs in rootful container creates unlabeled_t files it can't delete

[meln5674]

I'm running into a strange issue when running rootless podman nested in another (rootful) container regarding selinux, but there's so many pieces in play, I'm not certain if its a bug in podman or something else, or even how to figure that out. Any advice as to where to start looking would be appreciated.

Symptoms

When running podman in an affected configuration, building an image results in errors like this:

Error: deleting build container "50385867eb32bddd1806fdb85fdb83e87161815a1e0ac1fb24619e96f122f296": removing temp dir /home/podman/.local/share/containers/storage/overlay/tempdirs/temp-dir-1392570856 failed: unlinkat /home/podman/.local/share/containers/storage/overlay/tempdirs/temp-dir-1392570856/1-826347f89aa72b0db7ee837ae27536dd77aec11688b6eddc9a9bb728e3db2728/work/work/#24ce: permission denied: building at STEP "RUN apk add nginx": exit status 1

When this occurs, denials similar to the following appear in the selinux audit logs:

type=AVC msg=audit(1756924807.537:1248): avc:  denied  { getattr } for  pid=68964 comm="ls" path="/home/podman/.local/share/containers/storage/overlay/volatile-check3705286593/work/work/#366" dev="dm-0" ino=20908648 scontext=system_u:system_r:container_t:s0:c245,c642 tcontext=system_u:object_r:unlabeled_t:s0 tclass=chr_file permissive=0
type=AVC msg=audit(1756924814.307:1249): avc:  denied  { unlink } for  pid=68538 comm="exe" name="#368" dev="dm-0" ino=3641316 scontext=system_u:system_r:container_t:s0:c245,c642 tcontext=system_u:object_r:unlabeled_t:s0 tclass=chr_file permissive=0

My experience with selinux is admittedly limited, but to my understanding "unlabeled_t" either means "this was made on a non-selinux kernel" or "selinux has failed in some capacity". To my best understanding, podman is somehow managing to create a file without a valid selinux label (should this even be possible?) and then fails to delete it.

Tested Configurations

This issue can be reproduced 100% of the time on a fresh oracle linux 8.10 VM (kernel version 5.15) running rke2 v1.32 (containerd v1.7.27), but works 100% of the time on a fedora 42 VM running the same rke2. However, it works 100% of the time on the same OL8 VM running rke1 v1.8.6 (docker v28.1.1) . I think rules out an issue purely in kernel version, (outer) container runtime, or distro-provided selinux policy set.

Podman v5.6.0 and v5.4.0 were tested and produce the same results as one-another, so its not a regression.

Reproduction Steps

To demonstrate, run the following in the cluster in question.

kubectl create -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  restartPolicy: Never
  containers:
  - name: test
    image: quay.io/containers/podman:v5.6.0
    securityContext:
      runAsUser: 1000
      capabilities:
        add:
        - CAP_SETUID
        - CAP_SETGID
    volumeMounts:
    - mountPath: /home/podman
      name: home
    - mountPath: /dev/fuse
      name: fuse
    # command: [sleep, infinity]
    command: [/bin/bash, -xeuc]
    args:
    - |-
      podman build -f - . <<'EOD'
        FROM docker.io/library/alpine:3
        RUN apk add nginx
      EOD
  volumes:
  - name: home
    emptyDir: {}
  - name: fuse
    hostPath:
      type: CharDevice
      path: /dev/fuse

EOF

kubectl logs -f test

To install RKE1

curl -fL https://download.docker.com/linux/static/stable/x86_64/docker-28.1.1.tgz | tar xz --strip-components=1 -C /usr/bin/

systemd-run --unit=docker /usr/bin/dockerd

curl -fL -o /usr/bin/rke https://github.com/rancher/rke/releases/download/v1.8.6/rke_linux-amd64

chmod +x /usr/bin/rke

cat > cluster.yml <<'EOF'
kubernetes_version: v1.30.14-rancher1-1
nodes:
    - address: !!ip_address_goes_here!!
      user: root
      role:
        - controlplane
        - etcd
        - worker
EOF

rke up

To install RKE2

curl -sfL https://get.rke2.io | sh -

# not enabled on fedora
systemctl stop firewalld || true
systemctl start rke2-server

command cp -vrf /var/lib/rancher/rke2/bin/kubectl /usr/bin/kubectl
command cp -vrf /var/lib/rancher/rke2/bin/ctr /usr/bin/ctr

mkdir -vp /home/podman/.kube

command cp -vrf /etc/rancher/rke2/rke2.yaml /home/podman/.kube/config

chown -vR 1000:1000 /home/podman/.kube

[nalind]

If you're running all of this on top of xfs, then it could be https://lore.kernel.org/linux-xfs/[email protected]/, which was merged upstream in 5.20.

[meln5674]

Thank you, that does appear to be related to this. This is on xfs, and mounting an ext4 device over /var makes the issue go away when running under containerd, though that does leave the question of how it ever worked in the first place under docker.