Shovel status page in Management UI does not list shovels

[mohankumar27]

RabbitMQ version : 3.9.8
Erlang version : 23.2.7

Hi Everyone,

I am currently facing issue with shovel status in RabbitMQ cluster (7-node) . We had network partition issues which we resolved by doing a rolling restart of all the nodes. After this was done, In the management UI, shovel status page shows no shovels present. We had near to 30 shovels present before. Tried in each of the cluster node too using the query rabbitmqctl shovel_status. It gives the below error

Error:
{:noproc, {:gen_server, :call, [:rabbit_shovel_status, :status, :infinity]}}

Note :

• Shovels are running fine as per the connections tab in the Management UI.
• Shovel Management UI lists all the shovels, its source and destination etc.
• It just does not list the shovels running status in the shovel status page

Any help regarding this issue will be appreciated.

We tried reproducing this too but were unable to do so.

Thanks

[michaelklishin]

What's in the logs? Are you sure you are hitting the node that hosts the Shovel? #4242 can be highly relevant. It will ship in 3.10.0.

[mohankumar27]

Hi Michael,

Thanks for taking time and responding to this issue.

All shovels were present on Node 5 of 7-Node cluster.
Due to high CPU usage, that node got unresponsive which led to partition issue. Below is the origin error in Node 5 logs

The partition issue was resolved by force terminating the rabbit instance on Node 5 and a rolling restart of all the 7 nodes. I think during this restart some plugin issue has happened that might have caused issue in shovel_status management UI.

Even the rabbitmqctl command for shovel status too gives the error as I mentioned in the initial discussions

Error:
{:noproc, {:gen_server, :call, [:rabbit_shovel_status, :status, :infinity]}}

Can this issue be fixed by disabling and enabling both the rabbitmq_shovel and rabbitmq_shovel_management plugin ?

[michaelklishin]

I don't know, you are welcome to try and report back.

But I'm pretty sure that the changes in #4242 would make this a less likely scenario. However, the root cause is all Shovels congregating on the same node, which can exhaust it of bandwidth and CPU resources (and maybe even RAM).

Restarting the plugin on such node then would make Shovels migrate to another node. IIRC Federation
links and Shovels currently reside on a single node.

[mohankumar27]

We tried stopping and starting one of the rabbitmq-server in the 7-Node cluster and shovels status started displaying on the shovel status page in the management UI and via rabbitmqctl commands.

Thank you for your time in responding to our queries.

[michaelklishin]

Most likely because it hosted the Shovels in this cluster and the restart caused them to migrate to another node (and restart in the process).