EPMD IP binding doesn't change, but there's a possible workaround

[MaxRepin]

Describe the bug

export ERL_EPMD_ADDRESS=10.150.0.3 directive in rabbitmq-env.conf didn't work on Ubuntu 22.04.4 LTS. I think this is because rabbitmq-server service is set to depend on epmd.socket which starts under epmd user and probably takes its own environmental variables.

To resolve the issue I've removed the dependency in /lib/systemd/system/rabbitmq-server.service as follows:

After=network.target  # removed epmd.socket
Wants=network.target  # removed epmd.socket

Then I killed epmd, did a 'sudo systemctl daemon-reload' and restarted rabbitmq-server. Now it works as expected, and the user is rabbitmq:

COMMAND        PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
beam.smp      3795        rabbitmq   18u  IPv4  41961      0t0  TCP *:25672 (LISTEN)
beam.smp      3795        rabbitmq   19u  IPv4  41963      0t0  TCP 127.0.0.1:34274->127.0.0.1:4369 (ESTABLISHED)
beam.smp      3795        rabbitmq   33u  IPv4  43031      0t0  TCP 10.150.0.3:5672 (LISTEN)
beam.smp      3795        rabbitmq   34u  IPv4  43035      0t0  TCP 127.0.0.1:5672 (LISTEN)
epmd          3833        rabbitmq    3u  IPv4  41941      0t0  TCP 127.0.0.1:4369 (LISTEN)
epmd          3833        rabbitmq    4u  IPv6  41942      0t0  TCP [::1]:4369 (LISTEN)
epmd          3833        rabbitmq    5u  IPv4  41943      0t0  TCP 10.150.0.3:4369 (LISTEN)
epmd          3833        rabbitmq    6u  IPv4  42448      0t0  TCP 127.0.0.1:4369->127.0.0.1:34274 (ESTABLISHED)

Reproduction steps

1. Using default installation set ERL_EPMD_ADDRESS to any specific IP in rabbitmq-env.conf
2. Restart rabbitmq-server
3. Launch ss -l and see that epmd still binds to all IPs

Expected behavior

epmd binds to the IP that's been set using 'export ERL_EPMD_ADDRESS=' directive

Additional context

No response

[lukebakken]

Thanks for your report.

In your environment, please note that epmd is listening on port 4369 on all ipv6 interfaces in your environment:

epmd          3833        rabbitmq    4u  IPv6  41942      0t0  TCP [::1]:4369 (LISTEN)

Probably, the "correct" way to achieve this is through systemd override files. I'm working on the correct overrides to try to achieve what you want. I'll get back to this as time allows.

[MaxRepin]

Hi! Thanks for pointing out at that. I think TCP [::1]:4369 (LISTEN) means it listens on ipv6 loopback address only, which is similar to 127.0.0.1:4369.

[lukebakken]

OK, I finally figured it out.

Note that I'm using this Ubuntu 22.04 VM - https://app.vagrantup.com/generic/boxes/ubuntu2204

After starting it, I did apt update && apt upgrade

Also note that this VM explicitly disables ipv6 for the lo interface:

root@UBUNTU-22-JAMMY:~# sysctl net.ipv6.conf.lo.disable_ipv6
net.ipv6.conf.lo.disable_ipv6 = 1

I'm not sure why it ships this way, but it did throw a loop into some of my tests. Also, you need to ensure that the short host name resolves to 127.0.0.1:

root@UBUNTU-22-JAMMY:~# ping -c1 "$(hostname -s)"
PING UBUNTU-22-JAMMY (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.013 ms

--- UBUNTU-22-JAMMY ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.013/0.013/0.013/0.000 ms

You need to create two override files:

root@UBUNTU-22-JAMMY:~# ls -l /etc/systemd/system/epmd.socket.d/
total 4
-rw-r--r-- 1 root root 73 Apr 18 18:58 override.conf
root@UBUNTU-22-JAMMY:~# ls -l /etc/systemd/system/rabbitmq-server.service.d/
total 4
-rw-r--r-- 1 root root 72 Apr 18 19:03 override.conf

• epmd.socket.d/override.conf

[Socket]
ListenStream=
ListenStream=127.0.0.1:4369

• rabbitmq-server.service.d/override.conf

To be honest, this override isn't strictly necessary, I'm just using it to demonstrate that the [email protected] dependency doesn't really do anything, since epmd.socket isn't templated

[Unit]
After=network.target epmd.socket
Wants=network.target epmd.socket

Then, do a systemctl daemon-reload and start RabbitMQ. On my system, here are the listening sockets:

root@UBUNTU-22-JAMMY:~# ss -ptnl
State          Recv-Q         Send-Q                 Local Address:Port                  Peer Address:Port         Process
LISTEN         0              4096                       127.0.0.1:4369                       0.0.0.0:*             users:(("epmd",pid=2334,fd=3),("systemd",pid=1,fd=124))
LISTEN         0              4096                   127.0.0.53%lo:53                         0.0.0.0:*             users:(("systemd-resolve",pid=588,fd=14))
LISTEN         0              128                          0.0.0.0:22                         0.0.0.0:*             users:(("sshd",pid=738,fd=3))
LISTEN         0              128                          0.0.0.0:25672                      0.0.0.0:*             users:(("beam.smp",pid=2303,fd=19))
LISTEN         0              128                                *:5672                             *:*             users:(("beam.smp",pid=2303,fd=35))
LISTEN         0              128                             [::]:22                            [::]:*             users:(("sshd",pid=738,fd=4))

Let me know how it works in your environment.

[MaxRepin]

I think this one should work as well, but I haven't tried it yet:

epmd.socket.d/override.conf
[Socket]
ListenStream=
ListenStream=127.0.0.1:4369

My point was that epmd starts under the context of 'epmd' user and takes appropriate Environment Variables. And rabbitmq-server starts epmd under its own user, in which case the 'export ERL_EPMD_ADDRESS=10.150.0.3' directive from rabbitmq-env.conf worked as expected.

[lukebakken]

And rabbitmq-server starts epmd under its own user,

To be precise, in the case of systemd, RabbitMQ will start epmd only if the socket activation of epmd fails for some reason, or if the user disables socket activation.