Erlang error when running rename_cluster_node

[AntonAleksandrov13]

To whom it may concern,

We are running rabbitmq cluster(3 nodes RabbitMQ 3.8.9 Erlang 23.2.2) in ECS(Docker containers). Our goal is to rename one of the nodes without host restart.

We are using rabbitmqctl rename_cluster_node as indicated in documentation:

1. rabbitmqctl stop_app
2. rabbitmqctl rename_cluster_node rabbit@REDACTED [email protected] -l
3. rabbitmqctl start_app

However, on the second step(rename_cluster_node command), we are encountering an error. Please see logs:

rabbitmqctl rename_cluster_node rabbit@REDACTED [email protected] -l

Renaming cluster nodes:
 rabbit@REDACTED -> [email protected]

  * Impersonating node: [email protected]:
{:rename_failed, {:badmatch, {:error, {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [[:"rabbit@REDACTED", :shortnames], false, :net_sup_dynamic]}, :permanent, 1000, :supervisor, [:erl_distribution]}}}}}

While similar issues are related to networking, I have checked it as well:

epmd -names

epmd: up and running on port 4369 with data:
name rabbit at port 25672

If I try start rabbitmq using start_app, node starts with an old name as expected. It's available almost immediately.
Worth noting, there's a set of variables exported such as:

RABBITMQ_FEATURE_FLAGS_FILE=/var/lib/rabbitmq/mnesia/rabbit@REDACTED-feature_flags
RABBITMQ_MNESIA_DIR=/var/lib/rabbitmq/mnesia/rabbit@REDACTED

This is also posted in https://groups.google.com/g/rabbitmq-users/c/4aKxnmtZggk/m/1BVWi3ZrCAAJ?utm_medium=email&utm_source=footer

[michaelklishin]

From the docs:

The local cluster node must therefore be completely stopped; other nodes can be online or offline.

rabbitmqctl stop_app does not completely stop the node. It stops RabbitMQ but the runtime is still running, and so can
system Erlang applications such as net_kernel. rabbitmqctl shutdown would actually stop the node,
and I'm pretty sure the original use case for rename_cluster_node used that via BOSH upgrade scripts.

[AntonAleksandrov13]

Alright.rabbitmqctl shutdown will actually stop my container. Is there no way to avoid that?

[michaelklishin]

It will stop the node. RabbitMQ CLI tools are entirely unaware of containers.

I assume changing the entrypoint of your container to be an interactive shell you then start RabbitMQ would help. The rename operation only really needs persistent storage, it wouldn't care if a container was restarted.

And then there's always an option to migrate to a brand new cluster.