Test (podman) fails frequently in 'podman build and remove basic alpine with TMPDIR as relative'

[kolyshkin]

2026-02-07T21:36:24.3731523Z ------------------------------
2026-02-07T21:36:24.3731914Z Podman build
2026-02-07T21:36:24.3732366Z /root/go/src/github.com/containers/podman/test/e2e/build_test.go:21
2026-02-07T21:36:24.3733352Z   podman build and remove basic alpine with TMPDIR as relative
2026-02-07T21:36:24.3734071Z   /root/go/src/github.com/containers/podman/test/e2e/build_test.go:75
2026-02-07T21:36:24.3735179Z   > Enter [BeforeEach] TOP-LEVEL - /root/go/src/github.com/containers/podman/test/e2e/common_test.go:121 @ 02/07/26 21:36:24.372
2026-02-07T21:36:24.3736487Z   < Exit [BeforeEach] TOP-LEVEL - /root/go/src/github.com/containers/podman/test/e2e/common_test.go:121 @ 02/07/26 21:36:24.373 (0s)
2026-02-07T21:36:24.3737723Z   > Enter [It] podman build and remove basic alpine with TMPDIR as relative - /root/go/src/github.com/containers/podman/test/e2e/build_test.go:75 @ 02/07/26 21:36:24.373
2026-02-07T21:36:24.3738516Z   Restoring quay.io/libpod/alpine:latest...
2026-02-07T21:36:24.3740556Z   Running: /root/go/src/github.com/containers/podman/bin/podman --root /var/tmp/podman-e2e-1509464956/subtest-3236157071/p/root --runroot /var/tmp/podman-e2e-1509464956/subtest-3236157071/p/runroot --runtime /usr/bin/crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --cgroup-manager cgroupfs --tmpdir /var/tmp/podman-e2e-1509464956/subtest-3236157071/p --events-backend none --storage-driver overlay load -q -i /var/tmp/quay.io-libpod-alpine-latest.tar
2026-02-07T21:36:24.5060542Z   Loaded image: quay.io/libpod/alpine:latest
2026-02-07T21:36:24.5179999Z   Running: /root/go/src/github.com/containers/podman/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1509464956/imagecachedir --root /var/tmp/podman-e2e-1509464956/subtest-3236157071/p/root --runroot /var/tmp/podman-e2e-1509464956/subtest-3236157071/p/runroot --runtime /usr/bin/crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --cgroup-manager cgroupfs --tmpdir /var/tmp/podman-e2e-1509464956/subtest-3236157071/p --events-backend file --storage-driver overlay build --pull-never build/basicrun
2026-02-07T21:36:24.5459959Z   Error: mounting an overlay over build context directory: creating overlay scaffolding for build context directory: mount overlay:/root/go/src/github.com/containers/podman/test/e2e/buildah-context-3578011129/overlay/40134068/merge, data: lowerdir=/root/go/src/github.com/containers/podman/test/e2e/build/basicrun,upperdir=/root/go/src/github.com/containers/podman/test/e2e/buildah-context-3578011129/overlay/40134068/upper,workdir=/root/go/src/github.com/containers/podman/test/e2e/buildah-context-3578011129/overlay/40134068/work: invalid argument
2026-02-07T21:36:24.5502966Z   [FAILED] Command failed with exit status 125. See above for error message.
2026-02-07T21:36:24.5504067Z   In [It] at: /root/go/src/github.com/containers/podman/test/e2e/build_test.go:88 @ 02/07/26 21:36:24.55
2026-02-07T21:36:24.5505045Z 
2026-02-07T21:36:24.5505191Z   Full Stack Trace
2026-02-07T21:36:24.5505631Z     github.com/containers/podman/test/e2e.init.func5.3()
2026-02-07T21:36:24.5506360Z     	/root/go/src/github.com/containers/podman/test/e2e/build_test.go:88 +0x23d
2026-02-07T21:36:24.5507651Z   < Exit [It] podman build and remove basic alpine with TMPDIR as relative - /root/go/src/github.com/containers/podman/test/e2e/build_test.go:75 @ 02/07/26 21:36:24.55 (177ms)
2026-02-07T21:36:24.5509193Z   > Enter [AfterEach] TOP-LEVEL - /root/go/src/github.com/containers/podman/test/e2e/common_test.go:134 @ 02/07/26 21:36:24.55

[kolyshkin]

@flouthoc I see you're the author of the test that fails frequently in this repo; can you PTAL?

UPDATE podman PR is actually containers/podman#20367

[kolyshkin]

The error is:

2026-02-07T21:36:24.5459959Z Error: mounting an overlay over build context directory: creating overlay scaffolding for build context directory: mount overlay:/root/go/src/github.com/containers/podman/test/e2e/buildah-context-3578011129/overlay/40134068/merge, data: lowerdir=/root/go/src/github.com/containers/podman/test/e2e/build/basicrun,upperdir=/root/go/src/github.com/containers/podman/test/e2e/buildah-context-3578011129/overlay/40134068/upper,workdir=/root/go/src/github.com/containers/podman/test/e2e/buildah-context-3578011129/overlay/40134068/work: invalid argument

This is overlayfs mount returning EINVAL. Only the kernel log can tell us what's wrong.

[kolyshkin]

OK so it doesn't work because overlayfs on top of overlayfs doesn't work. What I still can't understand is how this test was ever passing.

[kolyshkin]

Indeed (from #2004):

2026-02-08T23:10:04.2435591Z [ 1145.949283] overlay: filesystem on /root/go/src/github.com/containers/podman/test/e2e/buildah-context-2195722604/overlay/1913652838/upper not supported as upperdir

Usually upperdir is either under /var/lib/containers or /var/tmp, both are bind-mounted from the host here:

crun/.github/workflows/test.yaml

Line 133 in 0bc3050

sudo docker run --cgroupns=host --privileged --rm -v /var/tmp:/var/tmp:rw -v /var/lib/containers:/var/lib/containers:rw -v /sys/fs/cgroup:/sys/fs/cgroup:rw,rslave -v ${PWD}:/crun crun-podman

and therefore are not on overlayfs.

We need to exclude the test.