Configured IP address gets lost somewhere in ETS_TABLE

[fin-ger]

I am currently investigating on an issue regarding the rabbitmq-diagnostics listeners and check_port_connectivity commands. Both commands always report the binding interface for all ports to be [::] regardless of the configuration. On my setup this leads to failed connectivity checks although the ports are open and running.

I investigated the code and first found that the check_port_connectivity is not using the configured binding interface but instead is resolving the hostname to connect to a service port.

However, this logic is not used to create the console output of listeners which explicitly states that all services are running behind [::]. I investigated where this output is produced and found out that the code reads the interface (or ip-address) from a listener record.

I figured out that this listener record is fetched from the ETS_TABLE.

I checked the contents of the ETS_TABLE with

$ rabbitmqctl eval 'lists:nth(1, ets:tab2list(rabbit_listener_ets)).'
{listener,'rabbit@myhost',http,"myhost.my.domain",
          {0,0,0,0,0,0,0,0},
          15672,
          [{cowboy_opts,[{sendfile,false}]},{ip,"127.0.0.1"},{port,15672}]}

I found the definition for the listener type in the ETS_TABLE which references the listener record. This shows that the IP address is the fourth element in the record, so it should be {0,0,0,0,0,0,0,0} as derived from the output. However, this is wrong. I explicitly set this to 127.0.0.1 (e.g. via prometheus.tcp.ip = 127.0.0.1 and management.tcp.ip = 127.0.0.1). In fact, the value is always {0,0,0,0,0,0,0,0} regardless of configuration.

I searched further and first found that the IP address is missing from the type definition. Second, the listener instance used to create and start a tcp_listener is altered in a couple of functions before being inserted into the ETC_TABLE. After a listener has been successfully started a listener_info is created which somehow fetches the IPAddress from the Port. I did not fully understand this code and don't know whether a fault occurs here. After that, the new listener is being registered in the ETS_TABLE. The data stored after this point seems to be faulty as the data read from the ETS_TABLE does not include the correct binding interface. It has to be noted that all services are running on the correct ports and interfaces (checked with netstat). Only the data in the ETS_TABLE is wrong.

As I do not know Erlang nor Elixir, I am not sure whether my analysis is correct, however I think that the handling of the ip_address field in the listener record is wrong. First during registration of the listener in the ETS_TABLE and later when using the listener record in the diagnostics tools.

[lukebakken]

What version of RabbitMQ and Erlang are you using?

[michaelklishin]

What configuration can we use to reproduce?

[michaelklishin]

CLI tools health check is not aware of the interface configuration. It does what most clients would do: connect using a hostname and a port. What specific interface should be used to connect is out of scope for most clients and this health check.

To analyze the rest of the code flow we need to know what configuration you've used and what your expectations are.

[fin-ger]

On my production system:

$ rabbitmqctl --version
3.11.6
$ aptitude versions erlang | grep i
i  1:25.2-1 bullseye 1000

Minimum configuration file to reproduce:

rabbitmq.conf:

management.tcp.ip = 127.0.0.1

This is really easy to check. Just use the latest official docker image and you can reproduce this. On my production setup I am using the official debian package.

$ docker run -it --rm --name rabbitmq_test \
    -v $(pwd)/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro \
    rabbitmq:3-management

And then run:

$ docker exec -it rabbitmq_test /bin/bash
# rabbitmqctl eval 'lists:nth(1, ets:tab2list(rabbit_listener_ets)).'
{listener,rabbit@afb753a5dbc5,http,"afb753a5dbc5",
          {0,0,0,0,0,0,0,0},
          15672,
          [{cowboy_opts,[{sendfile,false}]},{ip,"127.0.0.1"},{port,15672}]}
# rabbitmq-diagnostics listeners
Asking node rabbit@afb753a5dbc5 to report its protocol listeners ...
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Hope that helps!

[lukebakken]

Thank you. Providing exact reproduction steps saves us time and keeps us from having to guess.

[fin-ger]

What specific interface should be used to connect is out of scope for most clients and this health check.

IMHO this must be documented. Additionally, giving the interface in the console output is misleading as it suggests that the interface address is used to connect to the service. However, the value here will be whatever the hostname resolves to. Additionally, I think this is not what most people do. I think, most people have HTTP APIs behind a reverse proxy. The URL and domain of the reverse proxy are completely unknown to rabbitmq. Using the hostname for connections is IMHO a wild guess. I would expect rabbitmq to just use the IP address that is configured in the configuration file. Any other connectivity checks are out of scope for rabbitmq.

[michaelklishin]

In what console output? It's one thing if it's listed in listener information and another in the check itself.

@fin-ger you are making wild claims about what most people do or do not do. I will make one, too: you are the first person to complain about this check in a couple of years of its existence.

Most people do not override listeners, and listeners can be configured to use :: or 0.0.0.0, what IP address should the check use then?

I'll how realistic it might be to use a specific IP address if it's available but to me
it sounds like you should build your own health check, as a RabbitMQ plugin or an external tool of any kind.

[fin-ger]

In what console output? It's one thing if it's listed in listener information and another in the check itself.

Yes, the console output (of e.g the listeners command, but also of the check_port_connectivity) is using the ETS_TABLE as an information source for getting the IP address. However, the check_port_connectivity check itself (not the console output, but the actual check) is using whatever the hostname resolves to as IP addressing for probing the service.

you are the first person to complain about this check in a couple of years of its existence.

Yes, that seems to be correct as nothing similar arose in neither issues nor discussions.

Most people do not override listeners, and listeners can be configured to use :: or 0.0.0.0, what IP address should the check use then?

First, does this mean that overriding the listener binding interface is somewhat unsupported? Second, this is not a rabbitmq question, but networking in general. When you bind to [::] or 0.0.0.0 you can connect to [::] and 0.0.0.0 and receive an answer from all listeners bound to [::] (see notes in INADDR_ANY) or 0.0.0.0 respectively. So, it is perfectly fine to probe against 0.0.0.0 or [::] (as you are currently doing e.g. in the official docker image).

I'll how realistic it might be to use a specific IP address if it's available but to me

Yes, this is only a check if at least locally the ports are accessible and are answering requests. This ofc does not ensure connectivity throughout the whole network in which rabbitmq is deployed in. As I said, I think such connectivity check would be out-of-scope for rabbitmq.

[michaelklishin]

Overriding listener binding interface is supported but health checks can have limitations. This doc guide mentions multiple times that there is no such thing as
a universal health check. Your environment, configuration or approach to operations may mean
that you would have to develop your own health checks.

[fin-ger]

I understand that my environment may not be covered by the provided health checks. As it is now documented that the health checks connect via the hostname, I'm fine with it.

However, the other part (and actually the main part of my issue, as indicated in the title) is that the ETS_TABLE is holding the wrong IP address for listeners created with the rabbit_web_dispatch_registry, e.g. the management and prometheus plugins.

This leads to the listeners diagnostics command reporting false information on the bound interface address. Additionally, it makes it hard to write my own health check as I would have to parse the configuration file(s) on my own and figure out what address was used to bind the listener. This is very error-prone. I think a more reasonable approach would be to register the correct IP address in the ETS_TABLE. If the check_port_connectivity check is then using this information is up to you. If it doesn't, it's now documented and I know I have to write my own health check. Although, I think it would be reasonable to do so, as the hostname is not guaranteed to resolve to the exact interface you happen to bind the listeners on.

[michaelklishin]

rabbitmq/rabbitmq-website@99e3019 adds a note to the Monitoring guide that explains how the health check decides where to connect.

#6852 and #6853 are the only ideas I have since CLI tools cannot know what IP address to use to connect to listeners bound to :: or 0.0.0.0.

[fin-ger]

#6852 and #6853 are the only ideas I have since CLI tools cannot know what IP address to use to connect to listeners bound to :: or 0.0.0.0.

To my understanding the interface the listener is bound to should be part of the listener record. The IP address field is present in that record, but not properly filled during registration in the ETS_TABLE. The diagnostics commands could then use the IP address from the listener record fetched from the ETS_TABLE to do their checks on the correct interfaces. My bug report is about the rabbit_web_dispatch_registry not correctly writing the IP address of the successfully listened port to the ETS_TABLE. The IP address field in the ETS_TABLE is always {0,0,0,0,0,0,0,0}.

[michaelklishin]

Even if that is addressed, what would the CLI tool do when the interface is ::/0.0.0.0? How does it find out what interface or IP address to use?

[michaelklishin]

#6852 as best effort and #6853 are the only improvements I can think of even when the ETS record is corrected (#6852 would basically require that)

[fin-ger]

Even if that is addressed, what would the CLI tool do when the interface is ::/0.0.0.0? How does it find out what interface or IP address to use?

As I already explained here you can connect to e.g. 0.0.0.0:15672. This is a convention from BSD and its implementation in the Linux kernel is here. Using INADDR_ANY in a connect call is totally supported. As 0.0.0.0 binds all interfaces, this will just work.

[fin-ger]

Either way, you can also implement the convention yourself if you don't trust all OS's implementing this the same way: check for INADDR_ANY and connect to INADDR_LOOPBACK if INADDR_ANY was given. As INADDR_ANY binds all interfaces, a connect to INADDR_LOOPBACK will work.

[lukebakken]

@fin-ger @michaelklishin I created a repository to reproduce this issue:

https://github.com/lukebakken/rabbitmq-server-6851

Clone it and run docker compose up, then in another terminal:

lbakken@shostakovich ~/development/lukebakken/rabbitmq-server-6851 (main =)
$ docker compose exec rabbitmq rabbitmq-diagnostics listeners
Asking node rabbit@7d51c541fd11 to report its protocol listeners ...
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: 127.0.0.1, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

So this appears to be an issue with the management plugin. My guess is that other plugins that start listeners may be affected as well.

check_port_connectivity isn't working as expected. It's running in the same container as RabbitMQ but the port checks fail, probably because it's trying to connect to 0.0.0.0 as @michaelklishin points out in this issue - #6852

lbakken@shostakovich ~/development/lukebakken/rabbitmq-server-6851 (main =)
$ docker compose exec rabbitmq rabbitmq-diagnostics check_port_connectivity
Testing TCP connections to all active listeners on node rabbit@7d51c541fd11 ...
Error:
Connection to ports of the following listeners on node rabbit@7d51c541fd11 failed:
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: 127.0.0.1, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Listening ports within the container:

$ docker compose exec --tty --interactive rabbitmq /bin/bash

root@7d51c541fd11:/# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
rabbitmq       1       0  0 15:28 ?        00:00:00 /bin/sh /opt/rabbitmq/sbin/rabbitmq-server
rabbitmq      21       1  0 15:28 ?        00:00:07 /usr/local/lib/erlang/erts-13.1.2/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio nonerabbitmq      27      21  0 15:28 ?        00:00:00 erl_child_setup 1048576
rabbitmq      67       1  0 15:28 ?        00:00:00 /usr/local/lib/erlang/erts-13.1.2/bin/epmd -daemon
rabbitmq     102      27  0 15:28 ?        00:00:00 /usr/local/lib/erlang/erts-13.1.2/bin/inet_gethost 4
rabbitmq     103     102  0 15:28 ?        00:00:00 /usr/local/lib/erlang/erts-13.1.2/bin/inet_gethost 4
rabbitmq     106      27  0 15:28 ?        00:00:00 /bin/sh -s rabbit_disk_monitor
root        1274       0  0 15:44 pts/0    00:00:00 /bin/bash
root        1285    1274  0 15:45 pts/0    00:00:00 ps -ef

root@7d51c541fd11:/# apt update

root@7d51c541fd11:/# apt install iproute2

root@7d51c541fd11:/# ss -4l
Netid       State        Recv-Q       Send-Q             Local Address:Port              Peer Address:Port      Process
udp         UNCONN       0            0                     127.0.0.11:34125                  0.0.0.0:*
tcp         LISTEN       0            1024                     0.0.0.0:15692                  0.0.0.0:*
tcp         LISTEN       0            4096                     0.0.0.0:4369                   0.0.0.0:*
tcp         LISTEN       0            1024                   127.0.0.1:15672                  0.0.0.0:*
tcp         LISTEN       0            4096                  127.0.0.11:37473                  0.0.0.0:*
tcp         LISTEN       0            128                    127.0.0.1:5672                   0.0.0.0:*
tcp         LISTEN       0            128                      0.0.0.0:25672                  0.0.0.0:*

[michaelklishin]

@lukebakken thanks. I cannot reproduce this outside of Docker, so this must be something specific to container networking.

[fin-ger]

FYI, I'm experiencing this in my production setup on a debian bullseye VM without any container networking involved. I only used the docker container for easy reproduction of the issue.

[michaelklishin]

So the issue seemingly affects listeners that bind to 0.0.0.0 (or store that address in the listener table) and not a specific IP address.

[ikavgo]

I think it related to dns.
If now ip address explicitly given (via new option by Michael) check_port_connectivity uses hostname portion of node name.
The problem is that even on localhost local hostname not necessarily resolved to loopback. So here are two cases:

1. Rabbit bound to 127.0.0.1, check_connectivity uses "hostname" which is resolved to something else (depends on how many interfaces ifconfig shows, and exact dns setup), say 10.0.0.10. Of course the check will fail.
2. Rabbit bound to hostname - sometimes here it won't even start if resolves to some weird bridges (experienced myself) - then for any reason during health check hostname resolves to a different up address.

First case explains why we see this problem more often in docker, the networking setup is more complex. Thought I can repeat it on my machine too, not docker.

This prompts me to wonder when exactly we want to resolve hostnames. Wouldn't it make more sense to not check by hostname at all and always use stored listener ip address if not overriden.

[michaelklishin]

If the target is specified as --node rabbit@hostname, how can we avoid using DNS if --address is not explicitly provided? This part is well understood: we do what we can, if you don't agree with what CLI tools do, provide an --address you want to use or write your own health check as a plugin.

What is not well understood is what happens to the stored listeners, and how they are displayed in some cases, like #8242, for example.

[michaelklishin]

When a listener is bound to a specific address, ideally CLI tools should discover that and use it.

When it is bound to 0.0.0.0 or ::, the best we can do is

• Resolve the hostname (we already do this)
• Allow the user to specify an --address

Ironically, in order to get that specific IP address configured, we must guess how to contact the node. At the very least we can make sure that the correct values are stored in the tables,
which according to the analysis by the OP, is not the case.

By the way, I regret adding the check_port_connectivity health check and think it should be deprecated. It promises what it cannot necessarily deliver, for fundamental reasons (CLI tools do not have any information about the node's networking setup).

[michaelklishin]

FTR, while this is not really a part of this discussion, I have updated https://rabbitmq.com/monitoring.html#health-checks to mention that check_port_connectivity requires an --address or it will do its best: perform a hostname resolution, which may or may not match your expectations.