Visualize communication between producers and consumers

[mawilms]

Hello everyone,

is it possible retrieve information which producer sends a msg to an exchange and how many consumers are listening on the queue that gets the msg from the exchange? I can use Elixir or Erlang for this task, so maybe there is a plugin based solution?

I try to visualize these communication streams in a .dot graph with graphviz.

I'm really thankful for any input.

[mkuratczyk]

In the simplest case, this could probably be retrieved from the exchange/queue/channel metrics. However, this would break as soon as you have more things going on because there is no per message tracing. For example, with two publishers publishing to the same exchange/queue and two consumers consuming from the that queue, we could for example tell that each publisher published 1 messages and each consumer received one message, but we wouldn't be able to tell, based purely on metrics, which consumer received which message. Things complicate even further with fan-out exchanges, messages unacked by consumers and therefore returned to the queue and perhaps delivered to another consumers, etc. If your goal is to trace a message in a production env, this is something @deadtrickster has some interest in I believe but it's not something you can really do right now. If your goal is to build a tool to visualize this in a controlled environment, you'd need to include some IDs in the headers and/or payload to be able to trace where a given message was routed and delivered. Some distributed tracing tools already support RabbitMQ by injecting SPAN IDs into the headers but I can't personally say much more about that.

[mawilms]

Hello!

Thanks for your really detailed answer :)

"you'd need to include some IDs in the headers and/or payload to be able to trace where a given message was routed and delivered." Is this something that can be traced withing RabbitMQ? Through the exchange/queue/channel metrics? Can you give me a starting point where I can learn more on how to retrieve these metrics?

I think for my requirement, it's enough if I can say something like: Producer 1 sends a message to Exchange 1 which propagates the message to Queue 1 and on Queue 1 are currently listening Consumer 1 and Consumer 2.

The message itself and the content isn't important.

[mkuratczyk]

My point was that RabbitMQ doesn't track messages so precisely - it'd be your apps or some external systems doing that (this might change in the future but definitely nothing short-term). If your apps assign tracing IDs to messages when they publish them, they can then also track where these messages appeared on the other side. But that's it.

As for the metrics - enable the Prometheus plugin, declare an exchange e1, queues q1 and q2, bind them and then publish a message. Then you can do curl -s localhost:15692/metrics/per-object | grep -E 'e1|q1|q2' to see what metrics are available (most interesting probably being rabbitmq_queue_messages_published_total which will look like rabbitmq_channel_messages_published_total{channel="<0.1587.0>",vhost="/",exchange="e1"} 2). However:

1. These are only counters - in a system with many messages, you won't know which message was routed where - just how many
2. /metrics/per-object are only available as long as these objects exists (eg. when a connection is terminated, all stats about it disappear)

[mawilms]

Ok I see.

We're currently not using Prometheus. So I'm wondering if it's possible to get these metrics from RabbitMQ directly. Is it possible to inject your own gen server into the supervision tree of the RabbitMQ server? Would be nice if you could extract the metrics via Erlang or much better Elixir.

[mkuratczyk]

The Prometheus plugin is part of RabbitMQ and you don't need Prometheus - it's just an HTTP endpoint. You can build a RabbitMQ plugin so you may want to see if that's an option for you.

[mawilms]

Ah great! Thanks for all the input. I'll check how I can continue my journey :)