Podman + Quadlet on Pi Zero 2 W

[torntrousers]

Hello! I'd really like to be able to use Podman and Quadlet on a Raspberry Pi Zero 2 W, can you help me get it working?

Suggestion from here is to start this discussion.

Any pointers?

Is the only way to build from source, and if so any tips on doing that? I've started following the instructions here and get to "Make sure that the Linux kernel supports user namespaces...If not, please update the kernel" and it looks like it does not (on the Pi Zero which is running Debian bullseye), and I'm not sure how to update the kernel.

Can i do this build not on the Pi Zero, on my x86 windows machine in WSL perhaps?

[afbjorklund]

On debian, you need to enable kernel.unprivileged_userns_clone to enable user namespaces. Maybe some other tweaks, too...

Maybe not with the newer kernel though. Rootless containers seem to be working without it, at least with nerdctl

Linux raspberrypi 6.1.0-rpi7-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux

But you can build on your other machine, using GOOS=linux GOARCH=arm64 cross-compile. Probably with CGO_ENABLED=0 (static)

---

EDIT: Kernel settings were only needed for Debian 10, and before.

Things have changed a bit, since Raspberry Pi 2 and HypriotOS :-)

https://blog.hypriot.com/getting-started-with-docker-on-your-arm-device/

A similarly preconfigured SD card for Raspberry Pi 3+ would be nice.

[torntrousers]

apt install podman give version 3.0.1 though which is pretty old and I want Quadlet so need at least 4.4, right?

[afbjorklund]

Right, with bullseye (oldstable) you get 3.0.1. Probably should have podman.io link to https://packages.debian.org/podman

[afbjorklund]

Could be worse, with CentOS XP you get 1.6.4

[afbjorklund]

I think I will just have people get another SD card

[afbjorklund]

But according to https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi it is not supported yet (Zero 2)

Official support for the Zero 2W is under review.

WiFi on the Raspberry Pi Zero 2W and Raspberry Pi 400 is currently unsupported.

I guess you are supposed to be using something immutable and closer to Fedora CoreOS, like Fedora IoT ?

[siretart]

Please consider running debian testing, see https://wiki.debian.org/DebianTesting for details.
There are possible compromises that are unacceptable to you, like rate of updates, the associated bandwith costs, etc. If that is the case, consider installing the packages from debian/testing on your bookworm systems using Apt Pinning: https://wiki.debian.org/AptConfiguration?action=show&redirect=AptPreferences

[afbjorklund]

Most users are looking for a custom repo, with only the updated packages (for podman, and dependencies)...

Or something different, like OBS with Nix packages (like cri-o) or perhaps a "podman-full" tarball (like nerdctl)?

[siretart]

Most users are looking for a custom repo.

Maintaining such a repo is a huge effort. I've been approached on several occasions to provide updated packages in bookworm-backports, but I don't currently find the time for that.

At this point, your best options are:

• use debian/testing
• use packages from debian/testing on earlier versions of debian (or ubuntu, I guess)
• compile yourself

[afbjorklund]

• use fedora

[torntrousers]

Ok so trying in Windows WSL running Ubuntu, the prereqs installed ok, and "zgrep CONFIG_USER_NS /proc/config.gz" shows "CONFIG_USER_NS=y" but "sudo sysctl kernel.unprivileged_userns_clone=1" gives "No such file or directory" and "sudo echo 'kernel.unprivileged_userns_clone=1' > /etc/sysctl.d/userns.conf" gives "/etc/sysctl.d/userns.conf: Permission denied"

[torntrousers]

oh, with dpkg, ...

sudo dpkg -i podman_4.9.0+ds1-2_arm64.deb
(Reading database ... 54518 files and directories currently installed.)
Preparing to unpack podman_4.9.0+ds1-2_arm64.deb ...
Unpacking podman (4.9.0+ds1-2) over (3.0.1+dfsg1-3+deb11u4) ...
dpkg: dependency problems prevent configuration of podman:
 podman depends on libc6 (>= 2.34); however:
  Version of libc6:arm64 on system is 2.31-13+rpt2+rpi1+deb11u7.
 podman depends on libsqlite3-0 (>= 3.36.0); however:
  Version of libsqlite3-0:arm64 on system is 3.34.1-3.
 podman depends on libsubid4 (>= 1:4.11.1); however:
  Package libsubid4 is not installed.

dpkg: error processing package podman (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.9.4-2) ...
Errors were encountered while processing:
 podman

[torntrousers]

sudo apt install libsubid4
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package libsubid4 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'libsubid4' has no installation candidate

[torntrousers]

trying the 4.8.3 release -

sudo dpkg -i podman-docker_4.8.3+ds1-2_arm64.deb
Selecting previously unselected package podman-docker.
dpkg: regarding podman-docker_4.8.3+ds1-2_arm64.deb containing podman-docker:
 podman-docker conflicts with docker.io
  docker.io (version 20.10.5+dfsg1-1+deb11u2) is present and installed.

dpkg: error processing archive podman-docker_4.8.3+ds1-2_arm64.deb (--install):
 conflicting packages - not installing podman-docker
Errors were encountered while processing:
 podman-docker_4.8.3+ds1-2_arm64.deb

[torntrousers]

woot!

ant@pizero2:~ $ podman --version
podman version 4.9.0

[siretart]

I hope you understand now why I suggested to look into "apt pinning", or if that is too much hassle, just run debian/testing in the first place.

there are quite a number of dependencies and package relationships to keep track of (and thus, to backport, which is what you were originally asking for)

[torntrousers]

woot!

ant@pizero2:~ $ podman --version
podman version 4.9.0

Steps were:

1. to get Debian 12 instead off 11 which seems to be the default:

• in Pi Imager tool, in the Device Type select "No Filtering" (otherwise Bookworm doesn't show as a choice)
• For Choose OS select Raspberry Pi OS 64bit Debian Bookworm

2. Boot up the Pi and then:

• cat /etc/os-release should show Debian 12
• sudo apt-get update && sudo apt-get upgrade
• wget https://ftp.debian.org/debian/pool/main/libp/libpod/podman_4.9.0+ds1-2_arm64.deb
• sudo dpkg -i podman_4.9.0+ds1-2_arm64.deb (which fails complaining about libsubid4 missing
• sudo apt install libsubid4 (which fails complaining about a bunch of things)
• sudo apt --fix-broken install
• sudo apt-get install uidmap
• sudo apt-get install slirp4netns
• sudo apt-get install crun

(if it hangs reading changelogs update the swap size: https://forums.raspberrypi.com/viewtopic.php?t=360234#p2160951)

and now it should all work:

podman --version
podman version 4.9.0

podman run --rm docker.io/library/hello-world

Thanks for all the help @afbjorklund and @siretart !

[afbjorklund]

We still need a repository, to handle the updates (4.9.1, etc) and the missing dependencies.

[afbjorklund]

Note that you can run podman on the default 32-bit OS (armv7l) too, if you want to / have to.

[torntrousers]

also needs uidmap and slirp4netns. have updated the steps in #21396 (comment)

[torntrousers]

Anyway, this is all working and i have a running Quadlet but one final question - I see in the log there is a warning. I do systemctl --user status pi-blink.service and in the output it shows:
Feb 23 05:55:05 pizeroups conmon[749]: conmon 523eb0be7e54b7e0bfcb <nwarn>: Failed to add inotify watch for /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/app.slice/pi-blink.service/libpod-payload-523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2/memory.events

What does that mean and is it anything to worry about?

Here is the full output:

● pi-blink.service
     Loaded: loaded (/home/ant/.config/containers/systemd/pi-blink.container; generated)
     Active: active (running) since Fri 2024-02-23 05:55:05 GMT; 40s ago
   Main PID: 749 (conmon)
      Tasks: 3 (limit: 176)
        CPU: 1.305s
     CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/pi-blink.service
             ├─libpod-payload-523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2
             │ └─754 python3 blink.py
             └─runtime
               ├─734 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -r 3 -e 4 --netns-type=path /run/user/1000/netns/netns-b7600762-499e-3d8a-90ba-c07eadd7e314 tap0
               └─749 /usr/bin/conmon --api-version 1 -c 523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2 -u 523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2 -r /usr/bin/crun -b /home/ant/.local/share/containers/storage/overlay-containers/523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2/userdata -p /run/user/1000/containers/overlay-containers/523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2/userdata/pidfile -n systemd-pi-blink --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1000/containers/overlay-containers/523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2/userdata/oci-log --conmon-pidfile /run/user/1000/containers/overlay-containers/523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/ant/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/ant/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2

Feb 23 05:55:03 pizeroups systemd[575]: Starting pi-blink.service...
Feb 23 05:55:04 pizeroups podman[651]: 2024-02-23 05:55:04.645906662 +0000 GMT m=+0.799316250 system refresh
Feb 23 05:55:04 pizeroups podman[651]: 2024-02-23 05:55:04.700712495 +0000 GMT m=+0.854122969 image pull 67d06bd996ddc11bf7253f1a583fcacec3d950ad695ff600f628f1d40b445d47 docker.io/quarklinkdev/pi-blink:latest
Feb 23 05:55:04 pizeroups podman[651]: 2024-02-23 05:55:04.878003589 +0000 GMT m=+1.031413125 container create 523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2 (image=docker.io/quarklinkdev/pi-blink:latest, name=systemd-pi-blink, PODMAN_SYSTEMD_UNIT=pi-blink.service)
Feb 23 05:55:05 pizeroups conmon[749]: conmon 523eb0be7e54b7e0bfcb <nwarn>: Failed to add inotify watch for /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/app.slice/pi-blink.service/libpod-payload-523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2/memory.events
Feb 23 05:55:05 pizeroups podman[651]: 2024-02-23 05:55:05.30912661 +0000 GMT m=+1.462536198 container init 523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2 (image=docker.io/quarklinkdev/pi-blink:latest, name=systemd-pi-blink, PODMAN_SYSTEMD_UNIT=pi-blink.service)
Feb 23 05:55:05 pizeroups podman[651]: 2024-02-23 05:55:05.333983641 +0000 GMT m=+1.487393177 container start 523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2 (image=docker.io/quarklinkdev/pi-blink:latest, name=systemd-pi-blink, PODMAN_SYSTEMD_UNIT=pi-blink.service)
Feb 23 05:55:05 pizeroups systemd[575]: Started pi-blink.service.
Feb 23 05:55:05 pizeroups pi-blink[651]: 523eb0be7e54b7e0bfcb58d16a74bc8f35f0c0cd58b2be09e763ce662e9609b2
Feb 23 05:55:06 pizeroups systemd-pi-blink[749]: Blinking: Colour.RGB, speed=1.0, and pi5=False