Podman extremely slow on raspberry pi after OS upgrade

[endlisnis]

I was running podman in Debian 11 for many months and got decent performance out of it. It would take a few seconds to start a container; and a few minutes to build a new image.

When I upgraded to Debian 12, podman became about 50x slower. Just starting a container would take about 3 minutes and building a 30-step image would take hours.

I even tried a full OS reinstall using Debian 12 to avoid upgrade issues, and I saw the same result.

I'm running podman version 4.3.1 currently. Before the upgrade, I was running v3.something.

I tried running the same build script and starting the same containers using docker on the same raspberry pi; and it's performance is similar to podman in Debian 11.

Any ideas what's going on?

I'm running a raspberry pi 400 (4G) on an up-to-date Debian 12.

rolf@raspberrypi ~
$ time sudo docker run -it --rm fedora /bin/echo


real    0m1.341s
user    0m0.005s
sys     0m0.014s

rolf@raspberrypi ~
$ time podman run -it --rm fedora /bin/echo


real    0m17.962s
user    0m0.744s
sys     0m3.423s

(I ran these several times to get all the caches populated. I only posted the last runs).

Even just "exec" into an existing container takes dozens of seconds.

For extra details, after my OS upgrade, I could start and exec into an image/container with no obvious slowdown. It was only newly created images that suffered from this slowdown.

[vrothberg]

Can you share the output of podman info?

[endlisnis]

podmaninfo.txt

[rhatdan]

When I hear slowness, I usually think vfs storage driver versus overlay.

[endlisnis]

Do you mean "graphDriverName: vfs"? Which one is faster/better?

[rhatdan]

Overlay is much faster.

[rhatdan]

Edit storage.conf and do a podman system reset

[rhatdan]

Also make sure the arch for the Rasberry PI and the images you are running testing match. Don't run x86 images on an arm arch.

[endlisnis]

If overlay is much faster, why is vfs the default?

[endlisnis]

Changing it to "overlay" made it even faster than docker:

$ time podman run -it --rm fedora /bin/echo


real    0m0.648s
user    0m0.163s
sys     0m0.105s

[rhatdan]

What OS are you installing? On Fedora the default is overlay.

VFS is supported everwhere, overlay not so much, but we set it in storage.conf for distributions that we control.