Couldn't create an Arp client: no IPv4 address available for interface

[munin2k]

Hello,

we use vip-manager under Ubuntu 24.04 LTS. The DCS used is etcd. We used the .deb package with version 2.5.0. The installation went smoothly. The service starts, but there are problems assigning the IPv4 address:

Jul 25 07:51:30 scm-pgsql-01 systemd[1]: Started vip-manager.service - Manages Virtual IP for Patroni.
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:30 Using config from file: /etc/default/vip-manager.yml
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:30 This is the config that will be used:
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         config : /etc/default/vip-manager.yml
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         dcs-endpoints : [https://x.x.x.x:2379 https://x.x.x.x:2379 https://x.x.x.x:2379 https://x.x.x.x:2379]
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         dcs-type : etcd
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         etcd-ca-file : /etc/ssl/certs/ca.crt
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         etcd-cert-file : /etc/ssl/certs/cluster.crt
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         etcd-key-file : /etc/ssl/private/cluster.key
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         hosting-type : basic
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         hostingtype : basic
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         interface : ens224
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         interval : 1000
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         ip : 10.139.128.48
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         manager-type : basic
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         netmask : 23
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         retry-after : 250
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         retry-num : 2
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         trigger-key : /service/postgresql/pgsql/15-main/leader
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         trigger-value : pgsql-01
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         verbose : false
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]:         version : false
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:30 IP address 10.139.128.48/23 state is false, desired false
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:30 set WATCH on /service/postgresql/pgsql/15-main/leader
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:30 Current Leader from DCS: pgsql-01
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:30 IP address 10.139.128.48/23 state is false, desired true
Jul 25 07:51:30 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:30 Problems with producing the arp client: no IPv4 address available for interface
Jul 25 07:51:31 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:31 Problems with producing the arp client: no IPv4 address available for interface
Jul 25 07:51:31 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:31 too many retries
Jul 25 07:51:31 scm-pgsql-01 vip-manager[3859]: 2024/07/25 07:51:31 Couldn't create an Arp client: no IPv4 address available for interface

I have no idea what the problem could be and am hoping for help.

Thanks for your help
André

[pashagolub]

Hello. Do you run vip-manager under sudo?

[munin2k]

vip-manager runs under systemd. However, I have just tried to start the whole thing manually:

sudo /usr/bin/vip-manager --config=/etc/default/vip-manager.yml

But the error persists:

2024/07/25 12:28:35 Using config from file: /etc/default/vip-manager.yml
2024/07/25 12:28:35 This is the config that will be used:
	config : /etc/default/vip-manager.yml
	dcs-endpoints : [https://x.x.x.x:2379 https://x.x.x.x:2379 https://x.x.x.x:2379 https://x.x.x.x:2379]
	dcs-type : etcd
	etcd-ca-file : /etc/ssl/certs/ca.crt
	etcd-cert-file : /etc/ssl/certs/cluster.crt
	etcd-key-file : /etc/ssl/private/cluster.key
	hosting-type : basic
	hostingtype : basic
	interface : ens224
	interval : 1000
	ip : 10.139.128.48
	manager-type : basic
	netmask : 23
	retry-after : 250
	retry-num : 2
	trigger-key : /service/postgresql/pgsql/15-main/leader
	trigger-value : pgsql-01
	verbose : false
	version : false
2024/07/25 12:28:35 IP address 10.139.128.48/23 state is false, desired false
2024/07/25 12:28:35 set WATCH on /service/postgresql/pgsql/15-main/leader
2024/07/25 12:28:35 Current Leader from DCS: pgsql-01
2024/07/25 12:28:35 IP address 10.139.128.48/23 state is false, desired true
2024/07/25 12:28:35 Problems with producing the arp client: no IPv4 address available for interface
2024/07/25 12:28:35 Problems with producing the arp client: no IPv4 address available for interface
2024/07/25 12:28:35 too many retries
2024/07/25 12:28:35 Couldn't create an Arp client: no IPv4 address available for interface

[pashagolub]

I believe, this has todo with the specifics of your networking configuration. The error is produced by system, this is not vip-manager internal error. I would say interface you're trying to use doesn't support IP address you're trying to use.

@markwort have you met such an error?

[munin2k]

A separate network card is used as the network interface. Ubuntu uses netplan by default. Here is the configuration:

cat /etc/netplan/01-netcfg.yaml 
network:
    ethernets:
        ens192:
            dhcp4: false 
            addresses:
            - x.x.x.x/23
            nameservers:
                addresses:
                - x.x.x.x
                - x.x.x.x
                search:
                - domain.com
            routes:
            -   to: default
                via: x.x.x.x
        ens224:
            dhcp4: false
            dhcp6: false
    version: 2
    renderer: networkd

If I assign the IP address manually, everything works.

[pashagolub]

ok, it seesm the duplicate of #231

Do you use IPv6?

[munin2k]

IPv6 is not used, but is also not deactivated. I have now deactivated IPv6 as a test:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

But this does not change the error.

[pashagolub]

Weird. The documentation about error says:

errNoIPv4Addr is returned when an interface does not have an IPv4 address.

OK, let me think about it a little. I think it's possible to convert an error to a warning

[pashagolub]

Could you please check, if #246 solves your issue? It might still produce warnings about ARP client but the main functionality should work as expected

[munin2k]

I have compiled and integrated the binary. Here is the output:

[...]
Jul 25 14:28:07 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:07 IP address 10.139.128.48/23 state is false, desired false
Jul 25 14:28:07 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:07 set WATCH on /service/postgresql/pgsql/15-main/leader
Jul 25 14:28:07 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:07 current leader from DCS: pgsql-01
Jul 25 14:28:07 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:07 IP address 10.139.128.48/23 state is false, desired true
Jul 25 14:28:07 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:07 Problems with producing the arp client: no IPv4 address available for interface
Jul 25 14:28:07 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:07 Problems with producing the arp client: no IPv4 address available for interface
Jul 25 14:28:08 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:08 Couldn't create an Arp client: no IPv4 address available for interface
Jul 25 14:28:08 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:08 Configuring address 10.139.128.48/23 on ens224
Jul 25 14:28:08 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:08 No arp client available, skip send gratuitous ARP
Jul 25 14:28:08 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:08 IP address 10.139.128.48/23 state is true, desired true
Jul 25 14:28:17 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:17 IP address 10.139.128.48/23 state is true, desired true
Jul 25 14:28:27 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:27 IP address 10.139.128.48/23 state is true, desired true
Jul 25 14:28:37 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:37 IP address 10.139.128.48/23 state is true, desired true
Jul 25 14:28:47 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:47 IP address 10.139.128.48/23 state is true, desired true
Jul 25 14:28:57 pgsql-01 vip-manager[8935]: 2024/07/25 14:28:57 IP address 10.139.128.48/23 state is true, desired true

ip -br a
lo               UNKNOWN        127.0.0.1/8 
ens192           UP             10.139.128.49/23 
ens224           UP             10.139.128.48/23

So it seems to work. But what does that mean? Is it expected that the interface already has an IP address when vip-manager starts? What is so special about my network configuration that others don't have this problem?

[pashagolub]

The Address Resolution Protocol (ARP) is a network protocol used to map an IP address to a physical machine address (MAC address) in a local area network (LAN). ARP is essential for the functioning of IPv4 networks.

Purpose of ARP:

IP to MAC Address Mapping: ARP translates 32-bit IPv4 addresses to 48-bit MAC addresses. This is necessary because while IP addresses are used for routing packets across networks, MAC addresses are used for delivering packets within the same local network segment.

Communication within a LAN: When a device wants to communicate with another device on the same network, it needs to know the MAC address of the destination device. ARP helps in discovering this MAC address.

How ARP Works:

ARP Request: When a device wants to know the MAC address associated with an IP address, it broadcasts an ARP request packet to all devices on the local network. This packet contains the IP address for which the MAC address is sought.

ARP Reply: The device with the matching IP address responds with an ARP reply packet, which contains its MAC address. This reply is sent directly to the requesting device.

Caching: Devices typically cache the IP-to-MAC address mappings they learn through ARP to reduce the number of ARP requests on the network.

Example Scenario:

Device A wants to send data to Device B.
Device A knows Device B's IP address but not its MAC address.
Device A broadcasts an ARP request asking "Who has IP address X.X.X.X?"
Device B responds with an ARP reply saying "IP address X.X.X.X is at MAC address YY:YY:YY:YY:YY:YY."
Device A now knows the MAC address of Device B and can send data directly to it.

Code Context:

In the vip-manager ARP is used to send gratuitous ARP requests and replies. Gratuitous ARP is a special type of ARP where a device sends an ARP request or reply without having received an ARP request. This is often used to update other devices' ARP tables with the sender's IP-to-MAC address mapping, which can be useful in scenarios like IP address changes or failover situations.

[pashagolub]

I don't know how your configuration is different and why we have ARP errors. But since everything is working now, I'm closing the issue. Please check the new release with bug fix https://github.com/cybertec-postgresql/vip-manager/releases/tag/v2.6.0