cannot start server when node name different then hostname

[sergey-safarov]

I have configured a server with multiple IPv6 addresses only and each IPv6 address has different DNS names.
My goal is to separate services by different IP so I can relocate services with IP to a dedicated server and nothing changed for clients.

Example, in my case host have multiple DNS name line

1. ippbx-0a.ca.nga911.com 2605:84c0:48:4::6
2. ippbx-db0a.ca.nga911.com 2605:84c0:48:4::7
3. ippbx-mq0a.ca.nga911.com 2605:84c0:48:4::8

I want start CouchDB as node [email protected] and start RabbitMQ as node [email protected]

I have configured
/etc/rabbitmq/rabbitmq.conf

loopback_users.guest = false
listeners.tcp.default = 5672
default_pass = change_me
default_user = admin
vm_memory_high_watermark.relative = 0.7
cluster_partition_handling = autoheal
tcp_listen_options.keepalive = true
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.1 = [email protected]
cluster_formation.classic_config.nodes.2 = [email protected]
prometheus.tcp.port = 15692
prometheus.tcp.ip = ::
management.tcp.ip = ::
proxy_protocol = true

/etc/rabbitmq/rabbitmq-env.conf

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-proto_dist inet6_tcp -setcookie change_me"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp -setcookie change_me"
RABBITMQ_USE_LONGNAME=true
[email protected]

But when I wat start the server I got an error in the log

ERROR: epmd error for host ippbx-mq0a.ca.nga911.com: nxdomain

I have fond emd and rabbitmq do not try to resolve AAAA DNS records.
Also have started epmd indebug mode and do not see a request related to ippbx-mq0a.ca.nga911.com.
For me looks like an error in the rabbitmq startup script.

[sergey-safarov]

If I add record into /etc/hosts file for rabbitmq hostname with IPv4 loopback IP address then RabbitMQ start properly

127.0.0.1   ippbx-mq0a.ca.nga911.com

[michaelklishin]

RabbitMQ nodes must be able to resolve their own hostname, this has been a documented requirement for many years.

nxdomain is UNIX speak for "cannot resolve domain name".

[sergey-safarov]

You can reproduce the issue by:

1. assigning new IPv6 address to the network interface;
2. add a new record into /etc/hosts file new new IPv6 address and a new hostname for RabbitMQ node;
3. configure RabbitMQ to use IPv6 for name resolution;
4. add NODENAME variable with value like [email protected] into /etc/rabbitmq/rabbitmq-env.conf
5. start rabbitmq.

[michaelklishin]

RabbitMQ nodes are not aware of how you set up your IP stack and what DNS queries you expect them to use.

However, the runtime allows for some very accommodating configuration of all kinds of things related to hostname resolution, all in a single file that can be loaded without any elevated privileges.

[michaelklishin]

In other words, you have a very specific IP stack and hostname resolution configuration and expect RabbitMQ nodes to automagically pick that up and run with it. They can't and they won't. But the good news is that you can tell the runtime a lot about how it should resolve hostnames, even if /etc/hosts and friends are off-limits in your environment.

[sergey-safarov]

I want a simple thing, I want "use AAAA" request for name resolution. This does not work now as expected. Request simple does not send.
Is it too specific a request?

AAAA DNS request, nothing more.

[sergey-safarov]

Moreover /etc/hosts file does not help when erlang node name after @ char does not equal to the hostname.
The test is described in this comment.

[lukebakken]

You have a complicated setup. Please be sure to carefully read the docs - https://www.rabbitmq.com/networking.html#distribution-ipv6

You probably have to set ERL_EPMD_ADDRESS in /etc/rabbitmq/rabbitmq-env.conf as well as the options listed here:

• Create the /etc/rabbitmq/erl_inetrc file with this content:

  {inet6,true}.
  

• Ensure your /etc/rabbitmq/rabbitmq-env.conf file contains the following (note that the RABBITMQ_ prefix is not required in this file):

  ERL_EPMD_ADDRESS='2605:84c0:48:4::8'
  SERVER_ADDITIONAL_ERL_ARGS='-kernel inetrc /etc/rabbitmq/erl_inetrc -proto_dist inet6_tcp'
  CTL_ERL_ARGS='-proto_dist inet6_tcp'
  

Be sure to kill all erl and epmd processes before restarting RabbitMQ.

NOTE: you might be able to use the hostname instead of the IPv6 address. You'll have to test it out yourself.

[sergey-safarov]

Hi @lukebakken
Your suggestion about /etc/rabbitmq/erl_inetrc file fixed DNS name resolution via IPv6 protocol.

Thank you very much.

[lukebakken]

Thanks for letting us know.