Why is the error "epmd does not know us, re-registering rabbit at port 25672" reported?

[mthgh0818]

I am doing a stress test on the connect performance of Rabbitmq-mqtt. The stress test server configuration is as follows:
2C8G single node, deployed using docker, version RabbitMQ 3.12.12, Erlang 25.3.2.8; and set rabbitmqctl set_vm_memory_high_watermark 0.85

when using jmeter to connect to about 65000, the following error is reported:

2024-01-16 02:16:07.476837+00:00 [error] <0.27725.31> closing MQTT connection <<"192.168.179.147:30963 -> 172.17.0.2:1883">> (login timeout)
2024-01-16 02:16:07.478552+00:00 [info] <0.4841.32> Accepted MQTT connection 192.168.179.147:30113 -> 172.17.0.2:1883 for client ID conn_559f2b1935b34b2ea7
2024-01-16 02:16:36.483083+00:00 [warning] <0.2036.24> epmd does not know us, re-registering rabbit at port 25672
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24> ** Generic server rabbit_epmd_monitor terminating
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24> ** Last message in was check
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24> ** When Server state == {state,{erlang,#Ref<0.3472093683.1429733378.29180>},
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>                                erl_epmd,"rabbit","ef50598f63af",25672}
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24> ** Reason for termination ==
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24> ** {system_limit,
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>        [{erlang,open_port,
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>             [{spawn_executable,"/opt/erlang/lib/erlang/erts-13.2.2.5/bin/erl"},
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>              [{args,
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>                   ["-boot","no_dot_erlang","-sname","epmd-starter-822723895",
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>                    "-noinput","-s","erlang","halt"]},
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>               exit_status,stderr_to_stdout,use_stdio]],
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>             [{error_info,#{module => erl_erts_errors}}]},
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>         {rabbit_nodes_common,ensure_epmd,0,
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>             [{file,"rabbit_nodes_common.erl"},{line,112}]},
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>         {rabbit_epmd_monitor,check_epmd,1,
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>             [{file,"rabbit_epmd_monitor.erl"},{line,82}]},
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>         {rabbit_epmd_monitor,handle_info,2,
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>             [{file,"rabbit_epmd_monitor.erl"},{line,60}]},
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>         {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1123}]},
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>         {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]},
2024-01-16 02:16:36.483280+00:00 [error] <0.2036.24>         {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

At the same time, the web console cannot be opened normally.Check the memory usage as follows：

[mkuratczyk]

You have exceeded the available TCP ports.

[mthgh0818]

Thank you for your reply, but in this example rabbitmq is used as the server, and the above error is reported by rabbitmq, not the client. Under normal circumstances, it acts as a listener and is not limited by the number of connections.
and the /proc/$rabbitmq-pid/limits

[michaelklishin]

The ephemeral port range, which clients and CLI tools use (that's how TCP works), goes from 32768 to 60999 on many Linux distributions by default.

If you run out of ports, nothing will be able to connect to nothing else from this host.

[michaelklishin]

RabbitMQ acts as a TCP client when it connects to epmd, and in certain other cases (e.g. a Shovel or a Federation links are clients that run on RabbitMQ nodes). Therefore RabbitMQ opens a client TCP socket, and if the ephemeral port range is exhausted, it can't, and epmd eventually considers the node to be gone.

[mthgh0818]

Thanks for your reply, I have configured the port range as follows:
sysctl -w net.ipv4.ip_local_port_range='1024 65535'；
I made sure that there was only rabbitmq process running on the server and that there were sufficient free ports.Please help confirm if there are any other questions? I configured it according to the following connection https://www.rabbitmq.com/networking.html#open-file-handle-limit

[mthgh0818]

I want to achieve 100,000 mqtt connections on a single rabbitmq server. Below are all my configurations. Please help me see if there is anything missing.

1、sysctl -w fs.file-max=2097152
2、sysctl -w fs.nr_open=2097152
3、echo 2097152 > /proc/sys/fs/nr_open

2）
1、ulimit -n 1048576
2、vim /etc/sysctl.conf
fs.file-max = 1048576

3、vim /etc/systemd/system.conf
DefaultLimitNOFILE=1048576

3）
vim /etc/security/limits.conf

•  soft   nofile      1048576
  

•  hard   nofile      1048576
  

4）
sysctl -w net.core.somaxconn=32768
sysctl -w net.ipv4.tcp_max_syn_backlog=16384
sysctl -w net.core.netdev_max_backlog=16384
sysctl -w net.ipv4.ip_local_port_range='1024 65535'
sysctl -w net.core.rmem_default=262144
sysctl -w net.core.wmem_default=262144
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.optmem_max=16777216

#sysctl -w net.ipv4.tcp_mem='16777216 16777216 16777216'
sysctl -w net.ipv4.tcp_rmem='1024 4096 16777216'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16777216'
sysctl -w net.ipv4.tcp_max_tw_buckets=1048576

5） install rabbitmq with rpm type;
vim /usr/lib/systemd/system/rabbitmq-server.service
[Service]
LimitNOFILE=16384

[michaelklishin]

I'm afraid our team does not have the capacity to tune the OS for non-paying users.

We have a blog post and some guidelines in the Networking guide.

Perhaps most importantly you must understand that this kind of tests must start clients (or load testing tools) on a separate host, and that host must be configured accordingly to allow for a wide enough ephemeral (client) port range.

Trying to set everything up on the same machine is harder and doesn't really represent how your clients will connect to RabbitMQ nodes (they won't be connecting from the same host).

So, start with 50K connections between two machines then tune both sides gradually to get to 100K or whatever is the number you need. Going straight to 100K with everything on the single host will be unnecessarily frustrating.

[mthgh0818]

Thanks for your reply, I have made sure that the test client and rabbitmq server are on different hosts, this is my server configuration
Client test machine: 24C8G
Rabbitmq:14C8G