rabbitmq spawning every minute epmd (in okd version 4.11.0-0.okd-2023-01-14-152430)

[ffppmm]

Hi,
we experience a problem after updating to the current 4.11 okd version:

• RabbitMQ 3.11.3
• Erlang: 25.1.2 [jit]
• TLS Library: OpenSSL - OpenSSL 1.1.1k FIPS 25 Mar 2021

pstree -T -c
        ├─conmon───rabbitmq-server─┬─beam.smp───erl_child_setup─┬─inet_gethost───inet_gethost
        │                          │                            └─sh
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          ├─epmd
        │                          └─epmd

And they (epmd) keep growing in number, any idea? "First" epmd has the correct port for LISTENING, all other epmd have a random port. We have a 3 Node Cluster.

Regards Philipp

[michaelklishin]

Something in your environment runs either CLI tools (more likely) or the server (probably not according to the process tree) or causes the runtime start a new epmd instance due to certain periodic tasks that RabbitMQ nodes perform.

My best guess is that the node itself cannot connect to its local epmd in some cases.

epmd -names

and other epmd commands might help.

The RabbitMQ guide I've linked to explains how to pin down the ports used by epmd and a section that follows explains how to pin the port of inter-node communication ports (which is somewhat related).

Without seeing what system calls the node uses I cannot suggest much else. epmd expects remote and local (from RabbitMQ nodes and CLI tools) connections on port 4369 by default, and the runtime will start it if it is not running. Again, my best guess is that the RabbitMQ node cannot connect to it in some cases, e.g. due to a security policy of some kind, a local hostname resolution issue, and so on.

[lukebakken]

https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit_common/src/rabbit_nodes_common.erl#L109-L118

I'm looking into how this could leave behind processes like this.

[cobexer]

We see a new epmdprocess about every 60s ...

rabbitmq-server/deps/rabbit/src/rabbit_epmd_monitor.erl

Line 20 in 1020958

-define(CHECK_FREQUENCY, 60000).

[ffppmm]

Some additional Infos regarding the OS:

[root@HOSTNAME ~]# uname -a
Linux HOSTNAME 6.0.18-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Jan 7 17:08:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@HOSTNAME ~]# cat /etc/os-release
NAME="Fedora Linux"
VERSION="36 (CoreOS)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora CoreOS 36"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
SUPPORT_END=2023-05-16
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='411.36.202301141513-0'
DEFAULT_HOSTNAME=localhost

[michaelklishin]

Yup, nodes register with epmd every 60s. This is only really necessary on Windows but the same process runs on all OSes.

[lukebakken]

#7160 (comment)

[lukebakken]

@ffppmm @cobexer what this code does is start a temporary Erlang VM to ensure that epmd is running. I have run the same command on my local workstation many times, and epmd -daemon is only running once:

lbakken@shostakovich ~
$ killall epmd
lbakken@shostakovich ~
$ pgrep -a epmd
lbakken@shostakovich ~
$ "/home/lbakken/.asdf/installs/erlang/25.2.2/erts-13.1.4/bin/erl" -boot no_dot_erlang -sname "epmd-starter-$RANDOM" -noinput -s erlang halt
lbakken@shostakovich ~
$ "/home/lbakken/.asdf/installs/erlang/25.2.2/erts-13.1.4/bin/erl" -boot no_dot_erlang -sname "epmd-starter-$RANDOM" -noinput -s erlang halt
lbakken@shostakovich ~
$ "/home/lbakken/.asdf/installs/erlang/25.2.2/erts-13.1.4/bin/erl" -boot no_dot_erlang -sname "epmd-starter-$RANDOM" -noinput -s erlang halt
lbakken@shostakovich ~
$ pgrep -a epmd
52510 /home/lbakken/.asdf/installs/erlang/25.2.2/erts-13.1.4/bin/epmd -daemon

It's only running once because epmd knows when another process is running based on port 4369:

lbakken@shostakovich ~
$ epmd -d
epmd: Thu Feb  2 08:58:40 2023: epmd running - daemon = 0
epmd: Thu Feb  2 08:58:40 2023: there is already a epmd running at port 4369 on ipaddr 0.0.0.0

So, in your environment, epmd does NOT detect the already running process and starts a new one. I would begin by investigating why it appears that no processes are listening on port 4369. This is exactly what @michaelklishin stated in this comment:

My best guess is that the node itself cannot connect to its local epmd in some cases.

[ffppmm]

So the epmd-starter-??? tries to connect to the "old" one but can't or "whatever", but spawns a new epmd process with a different port, which is very strange. Magic ;)
I set ERL_EPMD_PORT=40666, restartet the StatefulSet and made an lsof with nsenter on an worker node and it looks like this:

epmd 3874705 10007900 3u IPv4 102082650 0t0 TCP *:40666 (LISTEN)
epmd 3874705 10007900 4u IPv6 102082651 0t0 TCP *:40666 (LISTEN)
epmd 3874705 10007900 5u IPv4 102069216 0t0 TCP 127.0.0.1:40666->127.0.0.1:48860 (ESTABLISHED)
epmd 3876577 10007900 3u IPv4 102079443 0t0 TCP *:34971 (LISTEN)
epmd 3876577 10007900 4u IPv6 102079444 0t0 TCP *:42895 (LISTEN)
epmd 3878350 10007900 3u IPv4 102095476 0t0 TCP *:40779 (LISTEN)
epmd 3878350 10007900 4u IPv6 102095477 0t0 TCP *:45981 (LISTEN)
epmd 3880015 10007900 3u IPv4 102114440 0t0 TCP *:41345 (LISTEN)
epmd 3880015 10007900 4u IPv6 102114441 0t0 TCP *:37983 (LISTEN)
epmd 3881708 10007900 3u IPv4 102116037 0t0 TCP *:39641 (LISTEN)
epmd 3881708 10007900 4u IPv6 102116038 0t0 TCP *:36861 (LISTEN)

First to ports correct and than random. Something happend and I don't know where the ports are comming from. I will try my luck tomorrow with the OKD Team.

Thanks for your help!

[ffppmm]

As promised ;)
okd-project/okd#1488

We did a strace of the network binding, maybe any thoughts on this?
okd-project/okd#1488 (comment)

Regards Philipp

[ffppmm]

Got some News on this:
https://bugzilla.redhat.com/show_bug.cgi?id=2159066

creds go to @vrutkovs from okd project:
okd-project/okd#1488 (reply in thread)