Podman driver fails with multiple errors (kubelet, br_netfilter) #22630
Description
What Happened?
TLDR: don't use Podman. Install Docker to make it work out of box. Podman and Docker can coexist without issues.
For those interested in non-traditional sex practices, read AI-generated summary of the actions and errors:
Description
Tried running minikube with Podman (rootless) + containerd. Startup fails during kubelet health checks. After hours of debugging, the root causes boil down to missing host/kernel features that are not realistically fixable in a rootless setup (cgroups cpuset, br_netfilter, sysctls, etc.).
Switching to the Docker driver on the same machine works instantly, without any kernel or systemd tweaking.
The error output is also misleading:
- kubeadm suggests enabling kubelet.service (not applicable for container-based drivers)
- swap warnings distract from the actual blockers
- failures happen late instead of failing fast with a clear message
Commands I tried:
sudo mkdir -p /etc/systemd/system/user@.service.d
printf "[Service]\nDelegate=yes\n" | sudo tee /etc/systemd/system/user@.service.d/delegate.conf
sudo systemctl daemon-reload
sudo systemctl restart user@$(id -u).service
# these outputted that cpuset was only present in `/sys/fs/cgroup/cgroup.controllers`
stat -fc %T /sys/fs/cgroup/
cat /sys/fs/cgroup/cgroup.controllers
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.subtree_control
# ...
sudo mkdir -p /etc/systemd/system/user@.service.d
printf "[Service]\nDelegate=yes\n" | sudo tee /etc/systemd/system/user@.service.d/delegate.conf
sudo systemctl daemon-reload
sudo systemctl restart user@$(id -u).service
# ...
# load now
sudo modprobe br_netfilter
# make persistent across reboots
echo br_netfilter | sudo tee /etc/modules-load.d/k8s.conf
# required sysctls
sudo tee /etc/sysctl.d/99-kubernetes-cri.conf >/dev/null <<'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
Below are log snippets with errors.
Attach the log file
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is not healthy after 4m0.000603802s
Unfortunately, an error has occurred, likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
stderr:
[WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap. Please refer to https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory, or disable swap on the node
[WARNING SystemVerification]: missing optional cgroups: hugetlb io
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/6.8.0-94-generic\n", err: exit status 1
[WARNING SystemVerification]: missing required cgroups: cpuset
[WARNING Service-kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error: error execution phase wait-control-plane: failed while waiting for the kubelet to start: The HTTP call equal to 'curl -sSL http://127.0.0.1:10248/healthz' returned error: Get "http://127.0.0.1:10248/healthz": context deadline exceeded
tderr:
[WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap. Please refer to https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory, or disable swap on the node
[WARNING SystemVerification]: missing optional cgroups: hugetlb io
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/6.8.0-94-generic\n", err: exit status 1
[WARNING SystemVerification]: missing required cgroups: cpuset
[WARNING Service-kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error: error execution phase wait-control-plane: failed while waiting for the kubelet to start: The HTTP call equal to 'curl -sSL http://127.0.0.1:10248/healthz' returned error: Get "http://127.0.0.1:10248/healthz": context deadline exceeded
To see the stack trace of this error execute with --v=5 or higher
I0205 17:53:40.744953 87713 ssh_runner.go:194] Run: sudo /bin/bash -c "env PATH="/var/lib/minikube/binaries/v1.35.0:$PATH" kubeadm reset --cri-socket /run/containerd/containerd.sock --force"
I0205 17:53:41.131718 87713 ssh_runner.go:194] Run: sudo systemctl is-active --quiet service kubelet
I0205 17:53:41.141832 87713 kubeadm.go:214] ignoring SystemVerification for kubeadm because of podman driver
I0205 17:53:41.141877 87713 ssh_runner.go:194] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0205 17:53:41.147369 87713 kubeadm.go:155] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0205 17:53:41.147379 87713 kubeadm.go:157] found existing configuration files:
I0205 17:53:41.147417 87713 ssh_runner.go:194] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/admin.conf
I0205 17:53:41.154348 87713 kubeadm.go:163] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/admin.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/admin.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/admin.conf: No such file or directory
I0205 17:53:41.154435 87713 ssh_runner.go:194] Run: sudo rm -f /etc/kubernetes/admin.conf
I0205 17:53:41.160560 87713 ssh_runner.go:194] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/kubelet.conf
I0205 17:53:41.167294 87713 kubeadm.go:163] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/kubelet.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/kubelet.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/kubelet.conf: No such file or directory
I0205 17:53:41.167353 87713 ssh_runner.go:194] Run: sudo rm -f /etc/kubernetes/kubelet.conf
I0205 17:53:41.173825 87713 ssh_runner.go:194] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/controller-manager.conf
I0205 17:53:41.179751 87713 kubeadm.go:163] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/controller-manager.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/controller-manager.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/controller-manager.conf: No such file or directory
I0205 17:53:41.179801 87713 ssh_runner.go:194] Run: sudo rm -f /etc/kubernetes/controller-manager.conf
I0205 17:53:41.186150 87713 ssh_runner.go:194] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/scheduler.conf
I0205 17:53:41.191673 87713 kubeadm.go:163] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/scheduler.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/scheduler.conf: No such file or directory
I0205 17:53:41.191720 87713 ssh_runner.go:194] Run: sudo rm -f /etc/kubernetes/scheduler.conf
I0205 17:53:41.197049 87713 ssh_runner.go:285] Start: sudo /bin/bash -c "env PATH="/var/lib/minikube/binaries/v1.35.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0205 17:53:41.231610 87713 kubeadm.go:318] [WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap. Please refer to https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory, or disable swap on the node
I0205 17:53:41.254114 87713 kubeadm.go:318] [WARNING SystemVerification]: missing optional cgroups: hugetlb io
I0205 17:53:41.254266 87713 kubeadm.go:318] [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/6.8.0-94-generic\n", err: exit status 1
I0205 17:53:41.254320 87713 kubeadm.go:318] [WARNING SystemVerification]: missing required cgroups: cpuset
I0205 17:53:41.286057 87713 kubeadm.go:318] [WARNING Service-kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0205 18:09:51.970836 124073 cli_runner.go:164] Run: podman network inspect minikube
W0205 18:09:51.986628 124073 cli_runner.go:211] podman network inspect minikube returned with exit code 125
I0205 18:09:51.986649 124073 network_create.go:288] error running [podman network inspect minikube]: podman network inspect minikube: exit status 125
stdout:
[]
stderr:
Error: network minikube: unable to find network with name or ID minikube: network not found
I0205
I0205 18:09:58.410080 124073 crio.go:165] couldn't verify netfilter by "sudo sysctl net.bridge.bridge-nf-call-iptables" which might be okay. error: sudo sysctl net.bridge.bridge-nf-call-iptables: Process exited with status 1
modprobe: ERROR: could not insert 'br_netfilter': Operation not permitted
Operating System
Ubuntu
Driver
Podman