CoreDNS restart loop / FailedCreatePodSandBox due to flannel subnet.env race condition

[SamYWJ]

Problem Summary

CoreDNS pods intermittently restart and fail to create pod sandboxes due to missing Flannel subnet.env file:

failed to setup network for sandbox: plugin type="flannel" failed (add):
failed to load flannel 'subnet.env' file:
open /run/flannel/subnet.env: no such file or directory

The first time setting up was normal but until i reboot the server the restart count always +1. Upon further checking i have found out that core dns has a FailedCreatePodSandBox due to flannel subnet.env race condition

" Warning FailedCreatePodSandBox 4m3s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "86d70ca0ff96f046000ccc39d3d59ae893ce7bbce48f7b1a516c83f3d31fdfcf": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node.
"

root@samywj:/home/k8s# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-snv4l 1/1 Running 5 (17m ago) 70m
kube-system coredns-7d764666f9-d62dz 1/1 Running 5 (17m ago) 72m
kube-system coredns-7d764666f9-f8n44 1/1 Running 5 (17m ago) 72m
kube-system etcd-samywj 1/1 Running 5 (17m ago) 72m
kube-system kube-apiserver-samywj 1/1 Running 5 (17m ago) 72m
kube-system kube-controller-manager-samywj 1/1 Running 5 (17m ago) 72m
kube-system kube-proxy-9vpl5 1/1 Running 5 (17m ago) 72m
kube-system kube-scheduler-samywj 1/1 Running 5 (17m ago) 72m

My goal is to keep the restart at 0 at all times. I have no idea how to solve this issue. Would be great if anyone has stumble upon this bug/problem

Ubuntu 24.04
Client Version: v1.35.4
Kustomize Version: v5.7.1
Server Version: v1.35.4

FLANNEL CNI (NEWEST VERSION)

[rbrtbnfgl]

Hi Could you get Flannel logs too?

[SamYWJ]

@rbrtbnfgl

Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
I0428 07:58:49.423812 1 main.go:226] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ipMasqRandomFullyDisable:false ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true blackholeRoute:false netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0428 07:58:49.426861 1 client_config.go:667] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0428 07:58:49.500954 1 kube.go:139] Waiting 10m0s for node controller to sync
I0428 07:58:49.501332 1 kube.go:537] Starting kube subnet manager
I0428 07:58:49.529542 1 kube.go:558] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.244.0.0/24]
I0428 07:58:50.515426 1 kube.go:163] Node controller sync successful
I0428 07:58:50.515537 1 main.go:252] Created subnet manager: Kubernetes Subnet Manager - gitlab
I0428 07:58:50.515547 1 main.go:255] Installing signal handlers
I0428 07:58:50.527876 1 main.go:534] Found network config - Backend type: vxlan
I0428 07:58:50.551714 1 kube.go:737] List of node(gitlab) annotations: map[string]string{"flannel.alpha.coreos.com/backend-data":"{"VNI":1,"VtepMAC":"7e:bb:7f:7b:b3:90"}", "flannel.alpha.coreos.com/backend-type":"vxlan", "flannel.alpha.coreos.com/kube-subnet-manager":"true", "flannel.alpha.coreos.com/public-ip":"10.3.1.152", "node.alpha.kubernetes.io/ttl":"0", "volumes.kubernetes.io/controller-managed-attach-detach":"true"}
I0428 07:58:50.553523 1 match.go:211] Determining IP address of default interface
I0428 07:58:50.557045 1 match.go:269] Using interface with name enp0s3 and address 10.3.1.152
I0428 07:58:50.558234 1 match.go:291] Defaulting external address to interface address (10.3.1.152)
I0428 07:58:50.560706 1 vxlan.go:128] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I0428 07:58:50.580787 1 kube.go:704] List of node(gitlab) annotations: map[string]string{"flannel.alpha.coreos.com/backend-data":"{"VNI":1,"VtepMAC":"7e:bb:7f:7b:b3:90"}", "flannel.alpha.coreos.com/backend-type":"vxlan", "flannel.alpha.coreos.com/kube-subnet-manager":"true", "flannel.alpha.coreos.com/public-ip":"10.3.1.152", "node.alpha.kubernetes.io/ttl":"0", "volumes.kubernetes.io/controller-managed-attach-detach":"true"}
I0428 07:58:50.581815 1 vxlan.go:199] Interface flannel.1 mac address set to: 7e:bb:7f:7b:b3:90
I0428 07:58:50.697494 1 main.go:389] Cleaning-up unused traffic manager rules
I0428 07:58:50.698398 1 nftables.go:280] Cleaning-up nftables rules...
I0428 07:58:51.022158 1 iptables.go:50] Starting flannel in iptables mode...
W0428 07:58:51.027935 1 main.go:588] no subnet found for key: FLANNEL_NETWORK in file: /run/flannel/subnet.env
W0428 07:58:51.027953 1 main.go:588] no subnet found for key: FLANNEL_SUBNET in file: /run/flannel/subnet.env
W0428 07:58:51.027963 1 main.go:623] no subnet found for key: FLANNEL_IPV6_NETWORK in file: /run/flannel/subnet.env
W0428 07:58:51.027971 1 main.go:623] no subnet found for key: FLANNEL_IPV6_SUBNET in file: /run/flannel/subnet.env
I0428 07:58:51.027980 1 iptables.go:101] Current network or subnet (10.244.0.0/16, 10.244.0.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I0428 07:58:51.239744 1 iptables.go:111] Setting up masking rules
I0428 07:58:51.254784 1 iptables.go:212] Changing default FORWARD chain policy to ACCEPT
I0428 07:58:51.259216 1 main.go:478] Wrote subnet file to /run/flannel/subnet.env
I0428 07:58:51.259665 1 main.go:482] Running backend.
I0428 07:58:51.261325 1 vxlan_network.go:68] watching for new subnet leases
I0428 07:58:51.262075 1 vxlan_network.go:115] starting vxlan device watcher
I0428 07:58:51.336596 1 main.go:503] Waiting for all goroutines to exit
I0428 07:58:51.387617 1 iptables.go:358] bootstrap done
I0428 07:58:51.417346 1 iptables.go:358] bootstrap done

[rbrtbnfgl]

So the file is created later. Did coreDNS start fine later?

[SamYWJ]

yeap it did. but im curious why it behave like this. why it wouldnt wait for the cni to initialize fully then then only start the pods. and my goal was making sure that the restarts stays at 0 for personal purposes

[rbrtbnfgl]

that's not directly related to flannel that's more an issue with kubernetes scheduling