Incorrect MAC Address observed for flannel interface across kubernetes cluster nodes

[amolmishra23]

Incorrect MAC Address observed for flannel interface across cluster nodes.

Expected Behavior

While we perform ARP resolution for the IP of flannel, from all nodes of cluster, it is supposed to show the correct MAC address. Thats not happening in this case, we observe different values for MAC address for same flannel, across all the nodes.

Current Behavior

$ "arp -an | grep 192.168.143.192"
// Result of the above command from all the nodes
=========== 10.14.7.42 ===========
? (192.168.143.192) at 32:48:e7:bb:74:d8 [ether] PERM on flannel.1
=========== 10.14.7.44 ===========
? (192.168.143.192) at 52:83:c1:6b:df:08 [ether] PERM on flannel.1
=========== 10.14.7.55 ===========
? (192.168.143.192) at 32:48:e7:bb:74:d8 [ether] PERM on flannel.1
=========== 10.14.7.56 ===========
? (192.168.143.192) at 52:83:c1:6b:df:08 [ether] PERM on flannel.1
=========== 10.14.7.62 ===========
? (192.168.143.192) at 32:48:e7:bb:74:d8 [ether] PERM on flannel.1
=========== 10.14.7.63 ===========
? (192.168.143.192) at 2a:36:32:7b:41:32 [ether] PERM on flannel.1
=========== 10.14.7.64 ===========
? (192.168.143.192) at 32:48:e7:bb:74:d8 [ether] PERM on flannel.1
=========== 10.14.7.43 ===========
Non-zero exit status: 1

$ ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 192.168.143.192  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::2836:32ff:fe7b:4132  prefixlen 64  scopeid 0x20<link>
        ether 2a:36:32:7b:41:32  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 5 overruns 0  carrier 0  collisions 0

The values in etcd are observed to be correct, FYI

$ etcdctl --prefix /flannel/network get
/flannel/network/config
{
      "EnableIPv4": true,
      "Network": "192.168.128.0/18",
      "SubnetLen": 26,
      "Backend": {
          "Type": "vxlan",
          "DirectRouting": false
      }
}
/flannel/network/subnets/192.168.134.192-26
{"PublicIP":"10.14.7.44","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":1,"VtepMAC":"b6:f1:e3:69:06:ad"}}
/flannel/network/subnets/192.168.137.64-26
{"PublicIP":"10.14.7.55","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":1,"VtepMAC":"66:14:5a:2b:ae:b0"}}
/flannel/network/subnets/192.168.140.192-26
{"PublicIP":"10.14.7.62","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":1,"VtepMAC":"f2:17:19:89:06:eb"}}
/flannel/network/subnets/192.168.143.192-26
{"PublicIP":"10.14.7.43","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":1,"VtepMAC":"2a:36:32:7b:41:32"}}
/flannel/network/subnets/192.168.144.128-26
{"PublicIP":"10.14.7.42","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":1,"VtepMAC":"46:6e:38:8a:7e:c6"}}
/flannel/network/subnets/192.168.146.192-26
{"PublicIP":"10.14.7.64","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":1,"VtepMAC":"1a:01:1e:7f:fa:1d"}}
/flannel/network/subnets/192.168.148.128-26
{"PublicIP":"10.14.7.56","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":1,"VtepMAC":"02:26:18:53:4f:a8"}}
/flannel/network/subnets/192.168.152.128-26
{"PublicIP":"10.14.7.63","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":1,"VtepMAC":"06:e9:34:03:4e:f4"}}

Possible Solution

Periodically resync the ARP entries.

Steps to Reproduce (for bugs)

Nothing special was done to reproduce this, but it was observed regularly in our environment.

Context

This is regularly causing communication issues between the pods in our environment.
As part of preliminary investigation, as we tried to check for flannel logs, in journalctl, observed the following

Jul 27 05:23:15 system-test-03-cc30210035-node-2 flanneld[27730]: E0727 05:23:15.200382   27730 iptables.go:307] Failed to bootstrap IPTables: failed to apply partial iptables-restore unable to run iptables-restore (, ): exit status 4
Jul 27 05:23:15 system-test-03-cc30210035-node-2 flanneld[27730]: I0727 05:23:15.245615   27730 iptables.go:421] Some iptables rules are missing; deleting and recreating rules
Jul 27 05:23:15 system-test-03-cc30210035-node-2 flanneld[27730]: E0727 05:23:15.276349   27730 iptables.go:307] Failed to bootstrap IPTables: failed to apply partial iptables-restore unable to run iptables-restore (, ): exit status 4
Jul 27 05:23:15 system-test-03-cc30210035-node-2 flanneld[27730]: E0727 05:23:15.313107   27730 iptables.go:320] Failed to ensure iptables rules: error setting up rules: failed to apply partial iptables-restore unable to run iptables-re
store (, ): exit status 4
Jul 27 05:23:15 system-test-03-cc30210035-node-2 flanneld[27730]: I0727 05:23:15.320792   27730 iptables.go:421] Some iptables rules are missing; deleting and recreating rules
Jul 27 05:23:15 system-test-03-cc30210035-node-2 flanneld[27730]: I0727 05:23:15.491607   27730 iptables.go:283] bootstrap done 

The firewall rules, for which it was failing, we tried to manually execute iptables-restore for the same. Then also ran into the issue:

$ cat test1.txt
*filter
-D FORWARD -m comment --comment "flanneld forward" -j FLANNEL-FWD
-A FORWARD -m comment --comment "flanneld forward" -j FLANNEL-FWD
-A FLANNEL-FWD -s 192.168.192.0/18 -m comment --comment "flanneld forward" -j ACCEPT
-A FLANNEL-FWD -d 192.168.192.0/18 -m comment --comment "flanneld forward" -j ACCEPT
COMMIT

$ sudo iptables-restore < test1.txt
iptables-restore v1.4.21: Couldn't load target `FLANNEL-FWD':No such file or directoryError occurred at line: 2
Try `iptables-restore -h' or 'iptables-restore --help' for more information. 

Your Environment

• Flannel version: 0.22.0
• Backend used (e.g. vxlan or udp): VXLAN
• Etcd version: 3.5.8
• Kubernetes version (if used): 1.27.3
• Operating System and version: CentOS Linux release 7.9.2009 (Core)
• Link to your project (optional):

[rbrtbnfgl]

Your restore file is wrong. You need to create FLANNEL-FWD chain before and using the restore as you did will delete also the other rules. Why do you need the MAC address? If you check the rules with iptables-save are the forward rules missing?

[amolmishra23]

MAC address are being configured on flannel interface by flanneld and they are getting out of sync from flanneld config in etcd. etcd info is correct.

FLANNEL-FWD we couldnt find anywhere in the flannel documentation, as a requirement that we are supposed to configure? https://github.com/flannel-io/flannel/blob/master/Documentation/kubernetes.md

Also as for the output of iptables-save, it doesnt have FLANNEL-FWD

$ iptables-save | grep FLANNEL-FWD
$

[rbrtbnfgl]

The MAC should be generated by linux when the interface is created. Your issue is unrelated to the ARP I think that maybe the grep is suppressing some lines. You shouldn't configure anything on the FLANNEL-FWD flannel code should do it automatically.
For the ARP I checked your output that is per node. The MAC address in networking is used at L2 and flannel divides the different nodes on a L3 connection using the bridge as gateway so it's right that different nodes has different MAC address to reach the same IP (it's the MAC of their default gateway).

[rbrtbnfgl]

How are you running flannel?
Are there any other process that are using iptables status 4 means that iptables is currently locked and it's unavailable.

[amolmishra23]

To elaborate our issue, over a period of time, the MAC addresses go out of sync. I just took the output from our running cluster. You can observe the mac address of flannel.2 is fa:4e:0e:a8:e6:30, but its only correct on some of the cluster nodes. Not all of them.

$ ifconfig flannel.2
flannel.2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 192.168.209.64  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::f84e:eff:fea8:e630  prefixlen 64  scopeid 0x20<link>
        ether fa:4e:0e:a8:e6:30  txqueuelen 0  (Ethernet)
        RX packets 4108  bytes 347980 (339.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 130  bytes 7800 (7.6 KiB)
        TX errors 0  dropped 5 overruns 0  carrier 0  collisions 0

$ allssh.sh "arp -an | grep 192.168.209.64"
=========== 10.14.19.0 ===========
? (192.168.209.64) at 4a:7d:ba:6c:17:b8 [ether] PERM on flannel.2
=========== 10.14.19.1 ===========
? (192.168.209.64) at fa:4e:0e:a8:e6:30 [ether] PERM on flannel.2
=========== 10.14.19.231 ===========
? (192.168.209.64) at 4a:7d:ba:6c:17:b8 [ether] PERM on flannel.2
=========== 10.14.19.232 ===========
? (192.168.209.64) at fa:4e:0e:a8:e6:30 [ether] PERM on flannel.2
=========== 10.14.19.233 ===========
? (192.168.209.64) at 4a:7d:ba:6c:17:b8 [ether] PERM on flannel.2
=========== 10.14.19.230 ===========
Non-zero exit status: 1

All nodes were expected to have mac address as fe80::f84e:eff:fea8:e630 for the IP: 192.168.209.64 in their arp table. But you see how random it is across the cluster.

Also verified that value in etcd is correct:

$ etcdctl --prefix /flannel2/network get
/flannel2/network/config
{
  "Network": "192.168.192.0/18",
  "SubnetLen": 26,
  "Backend": {
    "VNI": 2,
    "Port": 8473,
    "Type": "vxlan",
    "DirectRouting": false
  }
}
/flannel2/network/subnets/192.168.192.64-26
{"PublicIP":"10.14.19.231","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":2,"VtepMAC":"5e:68:d0:6c:37:99"}}
/flannel2/network/subnets/192.168.201.64-26
{"PublicIP":"10.14.19.0","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":2,"VtepMAC":"12:5d:da:44:9f:8c"}}
/flannel2/network/subnets/192.168.202.0-26
{"PublicIP":"10.14.19.233","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":2,"VtepMAC":"76:e0:88:13:d1:85"}}
/flannel2/network/subnets/192.168.207.192-26
{"PublicIP":"10.14.19.232","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":2,"VtepMAC":"ea:50:31:f5:75:6f"}}
/flannel2/network/subnets/192.168.209.64-26
{"PublicIP":"10.14.19.230","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":2,"VtepMAC":"fa:4e:0e:a8:e6:30"}}
/flannel2/network/subnets/192.168.211.192-26
{"PublicIP":"10.14.19.1","PublicIPv6":null,"BackendType":"vxlan","BackendData":{"VNI":2,"VtepMAC":"de:9e:ef:eb:1d:27"}}

---

Interim fix we figured is, to restart flannel across all the nodes.

$ allssh.sh "sudo systemctl restart flanneld2"
=========== 10.14.19.0 ===========
=========== 10.14.19.1 ===========
=========== 10.14.19.231 ===========
=========== 10.14.19.232 ===========
=========== 10.14.19.233 ===========
=========== 10.14.19.230 ===========

$ ifconfig flannel.2
flannel.2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 192.168.209.64  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::f84e:eff:fea8:e630  prefixlen 64  scopeid 0x20<link>
        ether fa:4e:0e:a8:e6:30  txqueuelen 0  (Ethernet)
        RX packets 4145  bytes 351134 (342.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 140  bytes 8572 (8.3 KiB)
        TX errors 0  dropped 5 overruns 0  carrier 0  collisions 0

$ allssh.sh "arp -an | grep 192.168.209.64"
=========== 10.14.19.0 ===========
? (192.168.209.64) at fa:4e:0e:a8:e6:30 [ether] PERM on flannel.2
=========== 10.14.19.1 ===========
? (192.168.209.64) at fa:4e:0e:a8:e6:30 [ether] PERM on flannel.2
=========== 10.14.19.231 ===========
? (192.168.209.64) at fa:4e:0e:a8:e6:30 [ether] PERM on flannel.2
=========== 10.14.19.232 ===========
? (192.168.209.64) at fa:4e:0e:a8:e6:30 [ether] PERM on flannel.2
=========== 10.14.19.233 ===========
? (192.168.209.64) at fa:4e:0e:a8:e6:30 [ether] PERM on flannel.2
=========== 10.14.19.230 ===========
Non-zero exit status: 1

After restart everything seems fine. All have correct mac address.
This is how we expect them to be in the first place.

---

iptables is also used by kube-proxy and firewalld processes also, in our environment.

However, we highly think firewall is unrealed to this issue. We were using the version 0.19.0, there as well, we observed the iptables error. But never faced incorrect arp entries and network connectivity issues.

But we are facing this since we upgraded to 0.22.0. This breaks our kubernetes cluster networking (intermittently).

[rbrtbnfgl]

Yes but flannel shouldn't working with the ARP table this is linux doing.

[amolmishra23]

But why is Linux doing it only in 0.22.0?
Why weren't we hitting this issue in 0.19.0?

Haven't understood the doc completely, but does look like flannel has something to do with the ARP entries:

flannel/pkg/backend/vxlan/vxlan.go

Line 19 in c17e715

// Some design notes and history:

flannel/pkg/backend/vxlan/vxlan_network.go

Line 186 in c17e715

return nw.dev.AddARP(neighbor{IP: sn.IP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)})

[amolmishra23]

Also the fact as I mentioned in my previous comment, that flannel restart fixes it all.

Implies that flannel does take some action, which fixes the arp entries.

Same action, flannel probably needs to take more frequently during its run-time (in absence of which we are hitting the issue).

[rbrtbnfgl]

Those values are added on the bootstrap of flannel and shouldn't change that's why I mentioned linux because somehow they are modified.

[amolmishra23]

The issue is that it not getting updated in flannel.
Theoretically, MAC can change when network or node restarts. Arp entry will need to be re-synced when that happens

[rbrtbnfgl]

So you are restarting the nodes and when a node restarted you can't access it? This seems strange because your issue should be more frequent also for other users.

[rbrtbnfgl]

I am trying to setup a new cluster with multiple nodes and trying to reproduce your issue

[amolmishra23]

We aren't even restarting.
But just saying, MAC address can change for multiple reasons, and proper re-sync mechanism should be in place.
Flannel just cannot run with bootstrap assumption. Isn't it?

[rbrtbnfgl]

Right now when a new nodes is added to the cluster flannel will call that part of the code that you mentioned where it adds the MAC address that it get from the etcd entry. MAC address of an interface shouldn't change if you somehow didn't force that change. I am doing some tests to check if there are strange cases where it applies.

[rbrtbnfgl]

I did some test also trying to restarts some nodes on the cluster. In case a node is restarted and the MAC for that specific node changed the ARP entry is updated on every node. What are the specific action that you are doing to have this issue? Are you starting k8s with kubeadm and deploying flannel with the manifest modified with your CIDR? After every nodes are up you encountered this issue with the MAC right?