rabbitmq_prometheus_exporter serves no metrics after upgrading to 3.13.0

[headyj]

Describe the bug

After updating rabbitmq to v3.13.0, prometheus-rabbitmq-exporter is not able to get metrics from the cluster:

time="2024-02-26T12:12:50Z" level=error msg="Error while retrieving data from rabbitHost" error="<nil>" host="http://rabbitmq.rabbitmq.svc.cluster.local:15672" statusCode=500

Reproduction steps

1. Setup rabbitmq using the cluster-operator with v3.12
2. Setup prometheus-rabbitmq-exporter so it gets metrics from your rabbitmq cluster:

prometheus-rabbitmq-exporter:
  [...]
  rabbitmq:
    url: http://rabbitmq.rabbitmq.svc.cluster.local:15672
    user: ****
    existingPasswordSecret

3. Update rabbitmq to v3.13.0

Expected behavior

prometheus-rabbitmq-exporter should still get metrics from rabbitmq cluster

Additional context

Rabbitmq cluster installed with cluster operator
Kubernetes v1.27 (EKS)

[mkuratczyk]

Please note that prometheus-rabbitmq-exporter is an external project and generally speaking, making sure it works is not within the scope of the RabbitMQ project. I'd highly recommend moving to the built-in Prometheus plugin which provides similar metrics without requiring an external tool and should also be more performant. If you have specific reasons why you think the exporter is better, please let us know what they are.

Having said that, it causes an error 500 on the Management API so it is an issue that we need to fix. The stack trace is:

2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>   crasher:
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     initial call: cowboy_stream_h:request_process/3
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     pid: <0.3765.0>
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     registered_name: []
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     exception error: no function clause matching
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                      rabbit_mgmt_util:get_dotted_value0([],
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                                                         [{name,<<>>},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                                                          {vhost,<<"/">>},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                                                          {type,direct},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                                                          {durable,true},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                                                          {auto_delete,false},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                                                          {internal,false},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                                                          {arguments,#{}},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                                                          {user_who_performed_action,
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                                                           <<"rmq-internal">>}]) (rabbit_mgmt_util.erl, line 599)
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>       in function  rabbit_mgmt_util:sort_key/2 (rabbit_mgmt_util.erl, line 593)
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>       in call from rabbit_mgmt_util:'-sort_list/4-lc$^0/1-1-'/2 (rabbit_mgmt_util.erl, line 466)
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>       in call from rabbit_mgmt_util:sort_list/4 (rabbit_mgmt_util.erl, line 466)
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>       in call from rabbit_mgmt_util:sort/2 (rabbit_mgmt_util.erl, line 410)
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>       in call from rabbit_mgmt_util:run_augmentation/2 (rabbit_mgmt_util.erl, line 403)
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>       in call from rabbit_mgmt_util:augment_resources0/6 (rabbit_mgmt_util.erl, line 392)
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>       in call from rabbit_mgmt_util:with_valid_pagination/3 (rabbit_mgmt_util.erl, line 311)
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     ancestors: [<0.3761.0>,<0.777.0>,<0.774.0>,<0.773.0>,<0.771.0>,
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                   rabbit_web_dispatch_sup,<0.721.0>]
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     message_queue_len: 0
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     messages: []
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     links: [<0.3761.0>]
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     dictionary: [{{xtype_to_module,topic},rabbit_exchange_type_topic},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                   {{xtype_to_module,direct},rabbit_exchange_type_direct},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                   {{xtype_to_module,headers},rabbit_exchange_type_headers},
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>                   {{xtype_to_module,fanout},rabbit_exchange_type_fanout}]
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     trap_exit: false
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     status: running
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     heap_size: 4185
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     stack_size: 28
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>     reductions: 5800
2024-02-26 14:22:40.492561+01:00 [error] <0.3765.0>   neighbours:

[mkuratczyk]

The issue is that the exporter requests sorted results without specifying which column to sort by. I can reproduce the error without the exporter by starting a fresh node and then:

curl -u guest:guest 'http://localhost:15672/api/nodes/?sort='

same problem with other endpoints (/exchanges/?sort=, etc)

[michaelklishin]

It's very generous of @mkuratczyk to have investigated something with an external Prometheus plugin. Given that the built-in plugin for Prometheus scraping has been around for years, I'd have simply closed it as something that the users of said plugin are supposed to deal with, just like with a lot of 3rd party plugins.

[headyj]

@michaelklishin yes, but users are not supposed to deal with such breaking changes with minor releases, moreover with a plugin which is as written in the documentation: highly recommended option for RabbitMQ monitoring.

[michaelklishin]

@headyj prometheus_rabbitmq_exporter is NOT what is documented in the guide. It is not the plugin that ships with RabbitMQ, and prometheus-rabbitmq-exporter has been effectively unmaintained ever since a different Prometheus plugin was included into the RabbitMQ distribution a few years ago.

[headyj]

@michaelklishin as far as I see, prometheus-rabbitmq-exporter is still maintained: https://artifacthub.io/packages/helm/prometheus-community/prometheus-rabbitmq-exporter

[michaelklishin]

In general, users of 3rd party plugins usually have to wait for them to be updated with every new feature RabbitMQ release: 3.13.0, 3.12.0, and so on. Even if it's just a "compatible version" list update in plugin's metadata.

If you are not happy with that, perhaps become a maintainer of the plugin and make sure it supports both 3.12 and 3.13, or 3.13 and 4.0, at the same time. Demanding that a small team of maintainers does something for you is not how open source works.

I'm not going to explain that strictly following SemVer is not an option for many reasonably large projects, in particular due to contractual obligations to support every major for N years.

[michaelklishin]

@headyj I don't know who it is maintained by. Team RabbitMQ does not maintain it, and the original author of that plugin has been a member of our team for several years now.

In case it is not clear: we DO NOT maintain that plugin, and will not maintain an obsolete plugin that was replaced by a built-in and documented one. If you absolutely must use that plugin, you are welcome to take over its maintenance.

[michaelklishin]

prometheus-erl/prometheus_rabbitmq_exporter#108

[michaelklishin]

To summarize:

• There is a built-in Prometheus plugin in RabbitMQ, and it has been the case for several years
• This older Prometheus plugin, prometheus_rabbitmq_exporter, is not maintained by the RabbitMQ core team
• This external exporter is not maintained by our team. Use default sorting when ?sort= in management/HTTP API #10610 addresses an issue that it was tripped by
• rabbitmq_exporter says at the top of the README that it is an unofficial external scraper. Its authors or users could have tested RabbitMQ 3.13 RCs (the RC phase lasted for about 4-5 months), then we wouldn't even have this conversation