Prometheus: the rabbitmq_global_publishers metric is reported as a negative value for one node

[yule-sun]

Describe the bug

I am using prometheus to monitor the rabbitmq cluster. When I use the command rabbitmq_global_publishers to obtain the value, the value on one of the nodes is negative. What is the reason?

Reproduction steps

...
none

Expected behavior

none

Additional context

No response

[michaelklishin]

@yule-sun we cannot suggest anything. You haven't shared any relevant information: no RabbitMQ version, no logs, no steps to reproduce.

We do not guess in this community.

[yule-sun]

@yule-sun we cannot suggest anything. You haven't shared any relevant information: no RabbitMQ version, no logs, no steps to reproduce.

We do not guess in this community.

Thank you for your reply. Sorry, I am using the automatically deployed RabbitMQ cluster in the environment deployed by Kolla-ansible.So I'm not sure how to write the reproduction steps, but the version of rabbitmq is 3.9.27,and below is the log of the rabbitmq cluster.

--- 2024-08-05 13:46:49.182445+08:00 [info] <0.23250.161> accepting AMQP connection <0.23250.161> (172.18.218.35:34580 -> 172.18.218.35:5672) 2024-08-05 13:46:49.185943+08:00 [info] <0.23250.161> Connection <0.23250.161> (172.18.218.35:34580 -> 172.18.218.35:5672) has a client-provided name: mod_wsgi:24:eeac4823-7a36-4753-ac4b-1b56fec85c85 2024-08-05 13:46:49.186700+08:00 [info] <0.23250.161> connection <0.23250.161> (172.18.218.35:34580 -> 172.18.218.35:5672 - mod_wsgi:24:eeac4823-7a36-4753-ac4b-1b56fec85c85): user 'openstack' authenticated and granted access to vhost '/' 2024-08-05 13:47:12.933573+08:00 [info] <0.23301.161> accepting AMQP connection <0.23301.161> (172.18.218.34:46174 -> 172.18.218.35:5672) 2024-08-05 13:47:12.938323+08:00 [info] <0.23301.161> Connection <0.23301.161> (172.18.218.34:46174 -> 172.18.218.35:5672) has a client-provided name: mod_wsgi:22:48e591a1-748e-47a9-b50b-e6270619e1f0 2024-08-05 13:47:12.938831+08:00 [info] <0.23301.161> connection <0.23301.161> (172.18.218.34:46174 -> 172.18.218.35:5672 - mod_wsgi:22:48e591a1-748e-47a9-b50b-e6270619e1f0): user 'openstack' authenticated and granted access to vhost '/' 2024-08-05 13:47:45.922649+08:00 [error] <0.10698.161> closing AMQP connection <0.10698.161> (172.18.218.35:37372 -> 172.18.218.35:5672 - mod_wsgi:22:5422ea89-95df-4d97-84a6-15c4044ff9f3): 2024-08-05 13:47:45.922649+08:00 [error] <0.10698.161> missed heartbeats from client, timeout: 60s 2024-08-05 13:48:13.208614+08:00 [info] <0.26523.161> accepting AMQP connection <0.26523.161> (172.18.218.35:60712 -> 172.18.218.35:5672) 2024-08-05 13:48:13.212100+08:00 [info] <0.26523.161> Connection <0.26523.161> (172.18.218.35:60712 -> 172.18.218.35:5672) has a client-provided name: mod_wsgi:22:5422ea89-95df-4d97-84a6-15c4044ff9f3 2024-08-05 13:48:13.212511+08:00 [info] <0.26523.161> connection <0.26523.161> (172.18.218.35:60712 -> 172.18.218.35:5672 - mod_wsgi:22:5422ea89-95df-4d97-84a6-15c4044ff9f3): user 'openstack' authenticated and granted access to vhost '/' 2024-08-05 13:52:12.171718+08:00 [error] <0.23167.161> closing AMQP connection <0.23167.161> (172.18.218.35:59216 -> 172.18.218.35:5672 - mod_wsgi:25:61f87963-0939-4434-83b8-d47cc1e1c861): 2024-08-05 13:52:12.171718+08:00 [error] <0.23167.161> missed heartbeats from client, timeout: 60s 2024-08-05 13:52:12.171717+08:00 [error] <0.23170.161> closing AMQP connection <0.23170.161> (172.18.218.35:59218 -> 172.18.218.35:5672 - mod_wsgi:25:456e92f6-47c1-43b0-a01c-a75476d51b3f): 2024-08-05 13:52:12.171717+08:00 [error] <0.23170.161> missed heartbeats from client, timeout: 60s 2024-08-05 13:52:48.913984+08:00 [info] <0.442.162> accepting AMQP connection <0.442.162> (172.18.218.35:49846 -> 172.18.218.35:5672) 2024-08-05 13:52:48.914007+08:00 [info] <0.439.162> accepting AMQP connection <0.439.162> (172.18.218.35:49838 -> 172.18.218.35:5672) 2024-08-05 13:52:48.916094+08:00 [info] <0.442.162> Connection <0.442.162> (172.18.218.35:49846 -> 172.18.218.35:5672) has a client-provided name: mod_wsgi:25:456e92f6-47c1-43b0-a01c-a75476d51b3f 2024-08-05 13:52:48.916254+08:00 [info] <0.439.162> Connection <0.439.162> (172.18.218.35:49838 -> 172.18.218.35:5672) has a client-provided name: mod_wsgi:25:61f87963-0939-4434-83b8-d47cc1e1c861 2024-08-05 13:52:48.916795+08:00 [info] <0.442.162> connection <0.442.162> (172.18.218.35:49846 -> 172.18.218.35:5672 - mod_wsgi:25:456e92f6-47c1-43b0-a01c-a75476d51b3f): user 'openstack' authenticated and granted access to vhost '/' 2024-08-05 13:52:48.916963+08:00 [info] <0.439.162> connection <0.439.162> (172.18.218.35:49838 -> 172.18.218.35:5672 - mod_wsgi:25:61f87963-0939-4434-83b8-d47cc1e1c861): user 'openstack' authenticated and granted access to vhost '/' 2024-08-05 13:54:49.194011+08:00 [error] <0.23250.161> closing AMQP connection <0.23250.161> (172.18.218.35:34580 -> 172.18.218.35:5672 - mod_wsgi:24:eeac4823-7a36-4753-ac4b-1b56fec85c85): 2024-08-05 13:54:49.194011+08:00 [error] <0.23250.161> missed heartbeats from client, timeout: 60s 2024-08-05 13:55:13.218786+08:00 [error] <0.26523.161> closing AMQP connection <0.26523.161> (172.18.218.35:60712 -> 172.18.218.35:5672 - mod_wsgi:22:5422ea89-95df-4d97-84a6-15c4044ff9f3): 2024-08-05 13:55:13.218786+08:00 [error] <0.26523.161> missed heartbeats from client, timeout: 60s 2024-08-05 13:55:44.678833+08:00 [info] <0.3883.162> accepting AMQP connection <0.3883.162> (172.18.218.35:45332 -> 172.18.218.35:5672) 2024-08-05 13:55:44.679753+08:00 [info] <0.3883.162> Connection <0.3883.162> (172.18.218.35:45332 -> 172.18.218.35:5672) has a client-provided name: mod_wsgi:22:5422ea89-95df-4d97-84a6-15c4044ff9f3 2024-08-05 13:55:44.680193+08:00 [info] <0.3883.162> connection <0.3883.162> (172.18.218.35:45332 -> 172.18.218.35:5672 - mod_wsgi:22:5422ea89-95df-4d97-84a6-15c4044ff9f3): user 'openstack' authenticated and granted access to vhost '/' 2024-08-05 13:56:11.819479+08:00 [info] <0.7023.162> accepting AMQP connection <0.7023.162> (172.18.218.35:40420 -> 172.18.218.35:5672) 2024-08-05 13:56:11.822988+08:00 [info] <0.7023.162> Connection <0.7023.162> (172.18.218.35:40420 -> 172.18.218.35:5672) has a client-provided name: mod_wsgi:24:eeac4823-7a36-4753-ac4b-1b56fec85c85 2024-08-05 13:56:11.823530+08:00 [info] <0.7023.162> connection <0.7023.162> (172.18.218.35:40420 -> 172.18.218.35:5672 - mod_wsgi:24:eeac4823-7a36-4753-ac4b-1b56fec85c85): user 'openstack' authenticated and granted access to vhost '/' 2024-08-05 13:58:11.992907+08:00 [error] <0.16474.161> closing AMQP connection <0.16474.161> (172.18.218.36:42398 -> 172.18.218.35:5672 - mod_wsgi:22:cb34b749-2c03-45dc-8183-cd04bd74aad7): 2024-08-05 13:58:11.992907+08:00 [error] <0.16474.161> missed heartbeats from client, timeout: 60s 2024-08-05 13:58:44.577146+08:00 [info] <0.10428.162> accepting AMQP connection <0.10428.162> (172.18.218.36:54780 -> 172.18.218.35:5672) 2024-08-05 13:58:44.578356+08:00 [info] <0.10428.162> Connection <0.10428.162> (172.18.218.36:54780 -> 172.18.218.35:5672) has a client-provided name: mod_wsgi:22:cb34b749-2c03-45dc-8183-cd04bd74aad7 2024-08-05 13:58:44.578880+08:00 [info] <0.10428.162> connection <0.10428.162> (172.18.218.36:54780 -> 172.18.218.35:5672 - mod_wsgi:22:cb34b749-2c03-45dc-8183-cd04bd74aad7): user 'openstack' authenticated and granted access to vhost '/' 2024-08-05 13:58:48.922782+08:00 [error] <0.439.162> closing AMQP connection <0.439.162> (172.18.218.35:49838 -> 172.18.218.35:5672 - mod_wsgi:25:61f87963-0939-4434-83b8-d47cc1e1c861): 2024-08-05 13:58:48.922782+08:00 [error] <0.439.162> missed heartbeats from client, timeout: 60s 2024-08-05 13:58:48.922829+08:00 [error] <0.442.162> closing AMQP connection <0.442.162> (172.18.218.35:49846 -> 172.18.218.35:5672 - mod_wsgi:25:456e92f6-47c1-43b0-a01c-a75476d51b3f): 2024-08-05 13:58:48.922829+08:00 [error] <0.442.162> missed heartbeats from client, timeout: 60s ---

But I don't know why there is a negative number when grabbing the rabbitmq_global_publishers parameter of the cluster.

[mkuratczyk]

RabbitMQ 3.9 has been out of support for years. This issue is likely already fixed.

[ansd]

Yes, this issue is fixed in supported versions.

[michaelklishin]

So, here is the relevant doc guide on upgrading.