Persist second IP on suse microOS

[sebastianlutter]

I use a floating IP (manually managed) assigned to some hetzner node. Then configure the IP to the interface with:

ip addr add {FLOATING_IP}/32 dev eth0

The interface shows two IPs, I can ssh into the node using the floating IP, ingress is also working on that node. But sometimes (after some minutes) the eth0 device is loosing the second IP. To avoid this I created this eth0 config /etc/sysconfig/network/ifcfg-eth0 (inspired from #611):

BOOTPROTO='static'
STARTMODE='auto'
IPADDR=12.34.56.78/32
IPADDR_2=1.2.3.4/32

It has no effect. Since SUSE MicroOS switched from wickedd to NetworkManager I guess we need to configure NetworkManager instead.

This file does not seem to be responsible for networking. When I change some IP and do a systemctl restart network the values are not used. Since network manager is managing the interfaces on current SUSE MicroOS I searched for its configuration and found:

# Generated by cloud-init. Changes will be lost.

[connection]
id=cloud-init eth0
uuid=1dd9a779-d327-56e1-8454-c65e2556c12c
type=ethernet

[user]
org.freedesktop.NetworkManager.origin=cloud-init

[ethernet]
mac-address=96:00:03:54:1E:92

[ipv4]
method=auto
may-fail=false

[ipv6]
method=manual
may-fail=false
address1=2a01:4f8:c013:63c8::1/64
gateway=fe80::1
dns=2a01:4ff:ff00::add:2;3f01:4ff:ff00::add:1;

How to properly configure a second IP for the eth0 interface on a hetzner node on SUSE MicroOS so it is used by NetworkManager and survies restarts?

If it is true that /etc/sysconfig/network/ifcfg-eth0 has no effect then is this the proper way to use that file here:

terraform-hcloud-kube-hetzner/agents.tf

Line 174 in 7281c7e

"echo \"BOOTPROTO='static'\nSTARTMODE='auto'\nIPADDR=${hcloud_floating_ip.agents[each.key].ip_address}/32\nIPADDR_1=${module.agents[each.key].ipv4_address}/32\" > /etc/sysconfig/network/ifcfg-eth0",

Can someone shed some light please?

[mysticaltech]

@sebastianlutter I believed we discussed the subject already recently. We do support floating-ips on agent-nodepools. However, if you want to see how this is done, have a look at our cloud-init templates.

[sebastianlutter]

@mysticaltech We discussed how existing floating IPs can be used on an agent node here #857

This discussion is about the way the floating IP is persisted on Suse MicroOS in the existing floating_ip implementation from @JurgenTreeum. The cloud-init configs used for agents (modules/host/templates/cloudinit.yaml.tpl does not define any connection settings, so it uses DHCP. In case floating_ip is set for an agent node then this code in agents.tf is executed after cloud-init to add the floating_ip using ip addr (works well) and a config /etc/sysconfig/network/ifcfg-eth0 to make the changes persistent

My point is that the file /etc/sysconfig/network/ifcfg-eth0 is not used, and thus the existing implementation of floating_ip on agent nodes does not survive an OS/node restart. I verified this by starting a agent node on the hetzner cloud, apply the steps in the issue description and then restarted the node manually. Result was that despite of having /etc/sysconfig/network/ifcfg-eth0 in place NetworkManager uses dhcp for interface eth0 (resulting in having only the primary IP assigned after restart).

That means that the current way of persisting the floating_ip on the node does not work, and I want to discuss how to fix that

[sebastianlutter]

An approach that works is using NetworkManager dispatcher scripts. I created /etc/NetworkManager/dispatcher.d/90-add-floating-ip.sh:

#!/bin/bash
if [ "$1" == "eth0" -a "$2" == "up" ]; then
  ip addr add 111.222.0.33 dev eth0
fi

Then chmod a+x /etc/NetworkManager/dispatcher.d/90-add-floating-ip.sh, reboot the node and everything is fine:

# ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 96:00:02:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 128.140.XX.XX/32 scope global dynamic noprefixroute eth0
       valid_lft 86078sec preferred_lft 86078sec
    inet 111.222.0.33/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2a01:4f8:c012:b027::1/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::9400:2ff:fe53:de5a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

I'll use this in my implementation of having an existing floating_ip_id (#857) and I'm pretty sure the current implementation in agent.tf is not a persistent solution and will be lost on node restart.

[sebastianlutter]

I created a bug report here: #866

Closing the discussion since I answered my own question. Discussion about the floating_ip and how to get it persistent in the current implementation can be done in the bug report

[mysticaltech]

@sebastianlutter I believe you, thank you for digging deep on that. Please do submit a PR with your fix if you can! It would be very much appreciated! 🙏