k3 fails to start using podman rootless

[cmoulliard]

Issue

The k3 kubernetes cluster fails to start using podman rootless as kubelet reports the following error

Error: failed to run Kubelet: failed to create kubelet: open /dev/kmsg: operation not permitted
time="2025-02-27T13:43:30Z" level=error msg="kubelet exited: failed to run Kubelet: failed to create kubelet: open /dev/kmsg: operation not permitted"

This issue is related to what we fixed for kind as this feature gate is missing

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  KubeletInUserNamespace: true

[cmoulliard]

We could set such a featureGate parameter using this flag

❯ podman run --privileged -p 6443:6443 rancher/k3s:latest server --kubelet-arg feature-gates=KubeletInUserNamespace=true

time="2025-02-27T14:07:44Z" level=info msg="Starting k3s v1.32.2+k3s1 (381620ef)"
...

but then we will got another error

E0227 14:07:46.717932      49 node_container_manager_linux.go:62] "Failed to create cgroup" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state" cgroupName=["kubepods"]
E0227 14:07:46.718024      49 kubelet.go:1649] "Failed to start ContainerManager" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state"

If now, we use the flag --rootless, then we got these error which can be fixed within the linux VM easily

❯ podman run --privileged -p 6443:6443 rancher/k3s:latest server --rootless
time="2025-02-27T14:12:01Z" level=fatal msg="expected sysctl value \"net.ipv4.ip_forward\" to be \"1\", got \"0\"; try adding \"net.ipv4.ip_forward=1\" to /etc/sysctl.conf and running `sudo sysctl --system`"

Nevertheless if we ran the command again then

❯ podman run --privileged -p 6443:6443 rancher/k3s:latest server --rootless
time="2025-02-27T14:14:44Z" level=warning msg="Running RootlessKit as the root user is unsupported."
time="2025-02-27T14:14:44Z" level=warning msg="The host root filesystem is mounted as \"\". Setting child propagation to \"\" is not supported."
time="2025-02-27T14:14:44Z" level=fatal msg="failed to setup UID/GID map: failed to compute uid/gid map: open /etc/subuid: no such file or directory"