Fix Cilium lxc* network interface filtering crash

[artem-zinnatullin]

We just faced a critical bug which crashes hetzner-k3s create --config on existing k3s cluster with Cilium CNI:

  …
    echo "Waiting for private network interface in subnet $SUBNET... (Attempt $i/$MAX_ATTEMPTS)" 2>&1 | tee -a /var/log/hetzner-k3s.log
    sleep $DELAY
  do
debug1: pledge: fork
Device "lxc6b9e41974beb@if1025" does not exist.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 7464, received 4292 bytes, in 0.6 seconds
Bytes per second: sent 11682.2, received 6717.6
debug1: Exit status 1 (IO::Error)
  from /home/runner/work/hetzner-k3s/hetzner-k3s/src/util/ssh.cr:121:7 in 'run'
  from /home/runner/work/hetzner-k3s/hetzner-k3s/src/util/ssh.cr:81:3 in '->'
  from /usr/lib/crystal/core/fiber.cr:170:11 in 'run'
  from ???

Cilium docs state that it creates network interface for each pod with lxcXXXXXX naming schema:

The pod’s network namespace contains a default route which points to the node’s router IP via the veth pair which is named eth0 inside of the pod and lxcXXXXXX in the host namespace.
https://docs.cilium.io/en/stable/network/concepts/routing/#architecture

And we do have tons of these on each node:

ip link show | grep lxc
1026: lxc6b9e41974beb@if1025: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
91: lxc2ea2f0368cbe@if90: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
95: lxcfe7fafe80413@if94: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
99: lxc5f5f942b8937@if98: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
101: lxc0cde58a77769@if100: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
103: lxcb7e446c960e4@if102: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
105: lxc06bac5834002@if104: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
107: lxc098c18cde290@if106: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
109: lxc8bc8349de570@if108: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
111: lxc55921d5b14bb@if110: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
668: lxc188c516a08e3@if667: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
670: lxc_health@if669: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
224: lxcc22128de63df@if223: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
226: lxcbca769de0c00@if225: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000

(We don't run anything via LXC, only k3s with Cilium CNI).

This PR adds lxc prefix to the grep filter in master & worker install scripts to safely ignore Cilium lxc* interfaces from picking them up as private network interface on the node instead of crashing the deploy.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[artem-zinnatullin]

Forgot to attach, it just happens that a pod lxc interface is printed before the actual private network enp7s0 interface:

ssh xxx
$ ip -o link show | awk -F': ' '/mtu (1450|1280)/ {print $2}' | grep -Ev 'cilium|br|flannel|docker|veth'
lxc6b9e41974beb@if1025
enp7s0
lxc2ea2f0368cbe@if90
lxcfe7fafe80413@if94
lxc5f5f942b8937@if98
lxc0cde58a77769@if100
lxcb7e446c960e4@if102
lxc06bac5834002@if104
lxc098c18cde290@if106
lxc8bc8349de570@if108
lxc55921d5b14bb@if110
lxc188c516a08e3@if667
lxc_health@if669
lxcc22128de63df@if223
lxcbca769de0c00@if225

With the fix we only get:

ip -o link show | awk -F': ' '/mtu (1450|1280)/ {print $2}' | grep -Ev 'cilium|lxc|br|flannel|docker|veth'
enp7s0

[vitobotta]

Thanks! I am surprised it wasn't reported before. I've merged the PR for now, will make a release perhaps in the weekend :)

[artem-zinnatullin]

Me too! I actually had to delete the pod that had this interface, found via:

cilium-dbg endpoint list -o json | jq '.[] | select(.status.networking."interface-name" == "lxc6b9e41974beb") | .status."external-identifiers"."k8s-pod-name"'

^ By running this in cilium pod on the bad node.

The pod got reinstantiated by ReplicaSet and got a different lxc interface name that sorted below enp7s0 and deploy passed haha, but obviously lxc* interfaces should be filtered out :)

@vitobotta can you please take a look at Grow Partition ClusterAutoscaler fix PR too? #699

[vitobotta]

I haven't had a chance to test that one yet. I'll see if I have time in the weekend.