Performance decrease in RabbitMQ Management and CLI

[GroovyRice]

Describe the bug

Whenever a node within a cluster becomes unreachable on RabbitMQ 3.12.4 due to being blocked by the firewall, the management portal and the CLI will take increasingly lengthy amounts of time to load or display diagnostics. On version 3.11.7 if the node was unreachable speed in management and CLI was not impacted, this means that a change between 3.11.X to 3.12.X caused this performance degradation.

Reproduction steps

1. Create a cluster with 3 nodes on 3.12.4.
2. Turn on the firewall and block all traffic along clustering ports from a specific node.

Expected behavior

Ideally, much like 3.11.7 if the node is unreachable this shouldn't impact the loading times of the management portal or CLI at all, it should maintain its speed and display the node as Off / Unreachable.

Additional context

No response

[michaelklishin]

Both HTTP API operations and CLI tools may contact multiple nodes at once:

• Listing operations such as GET /api/connections in the API
• list_* operations plus cluster status (cluster_status, listing feature flags) commands in CLI tools

This behavior has been around for years.

If that node is down, peers and CLI tools will wait for a timeout from that node. If your firewall rule drop traffic instead of rejecting it, it indeed can take up to the TCP handshake timeout on the host. With a rejected TCP connection, the unreachable node should be skipped very quickly.

We need more information to know how exactly to reproduce this behavior. We do not guess in this community.

[kjnilsson]

@GroovyRice is this the case also if the mgmt metrics and stats collection is turned off? https://www.rabbitmq.com/management.html#disable-stats