RabbitMQ Management and CLI slow speeds #9194
-
Describe the bugRecently upgraded from RabbitMQ 3.11.7 to 3.12.4, in doing so we also upgraded Erlang to 26.0.3. Although the speed of message flow has been increased, viewing the nodes and queues within the RabbitMQ Management Plugin or even just running cluster_status takes increasingly longer durations. For context, cluster_status use to take 3-10 seconds give or take, now it takes upwards of two-three minutes. Further if you navigate around the Management portal, what use to take 0 - 0.5 seconds to navigate between sections now takes close to 10 seconds. I.e. From "Overview" to "Queues and Streams" takes 7-10 seconds until it loads now. Further, the refresh is set to every 5 seconds but it takes 10 - 15 seconds before it refreshes. What could be causing this issue and why is this happening on 3.12.4? Reproduction steps
Expected behaviorLoading times should be far faster, I will happily record my screen to show you what I mean if you want. Additional contextNo response |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
|
ISSUE FOUND: For context the RabbitMQ Cluster has 3 nodes that were all on and one additional one that was offline as it was awaiting firewalls rule updates. That node was still "clustered" but was unreachable by any of the other nodes BUT was still in the cluster, as such it seems as if RabbitMQ was constantly polling the offline node causing the management and CLI operations to be increasingly slow. After forgetting the node from the cluster the performance reverted back to near instantaneous. |
Beta Was this translation helpful? Give feedback.
-
|
Six minutes elapsed between when you opened the issue and found that the cause was due to your environment. Next time, please wait before assuming there is a problem with RabbitMQ (or any software, really). Personally I usually wait a day or longer after thinking I've found a bug in someone else's code. Almost always it's something I'm doing. |
Beta Was this translation helpful? Give feedback.
-
|
@lukebakken your response is as useless as it is arrogant, there is an inherent bug in place, there is no need to deem someone based on raising a problem they experienced and a solution they found. The issue has been present within our system for a week, once I found a fix, I thought it would be best if it was mentioned on GitHub so that if someone experiencing something similar could potentially use this to right the problem (Yes, I put it in the wrong section, but @michaelklishin corrected this, thank you). I have opened the bug properly in a separate instance #9215 if you could look into it that would be tremendously helpful for a lot of people. |
Beta Was this translation helpful? Give feedback.
-
|
Neither CLI tools nor nodes "poll" offline peers but some operations requiring contacting all cluster member:
if one or more cluster members are down, contacting them will only be considered a failure after a timeout (30-60s by default). |
Beta Was this translation helpful? Give feedback.
ISSUE FOUND: For context the RabbitMQ Cluster has 3 nodes that were all on and one additional one that was offline as it was awaiting firewalls rule updates. That node was still "clustered" but was unreachable by any of the other nodes BUT was still in the cluster, as such it seems as if RabbitMQ was constantly polling the offline node causing the management and CLI operations to be increasingly slow. After forgetting the node from the cluster the performance reverted back to near instantaneous.