Identify Root Cause for Partition Issue prematurely

[mohankumar27]

RabbitMQ version : 3.9.8

Erlang version : 23.2.7

Hi everyone.

We recently faced partition issue in our 7-node rabbitmq cluster. A partition occurred due to high CPU usage in one of our RabbitMQ nodes and it became unresponsive. We later fixed the issue manually by killing the rabbitmq process on the unresponsive node and doing a rolling restart of all the nodes.

I was wondering if there was any way to determine the root cause of the partition issue from RabbitMQ logs, Erlang metrics, or any prometheus exposed metrics as to why that specific node had high CPU usage and became unresponsive, resulting in partition. Not only with high CPU usage, but also with any other circumstances that may result in a partition.

Also to add on, the node that became unresponsive had all the shovels in that cluster running. There were nearly 60+ shovels.

Is there any way to identify a partition prematurely so that we can act upon it accordingly as we have not setup any auto heal process?

Any help regarding this would be appreciated.

Thanks

[michaelklishin]

rabbitmq-top can provide a breakdown of CPU/runtime resource use on the node at the current moment but that information is not collected historically. The closest option are 1) Other metrics that might indicate correlating spikes or increases and 2) certain runtime metrics exposed via Prometheus. I think 2) might include scheduler metrics but I haven't looked at them for a while.

[mohankumar27]

Hi Michael, 

 Thank you for your quick response. I will analyse the metrics and let you know if I have any findings there.