Missing publish confirms on default exchange

[motmot80]

Hello everyone,

during high load scenarios in our RMQ cluster we are seeing missing publish confirms on messages published (queue-routing by routing key/subject) to the default exchange.
At the same time other exchanges with less messages on the same cluster are not showing the problem.
The producing apps are equally distributed on the cluster nodes.

I know that there's a capacity limit for queues (1 CPU max per queue) but is the default exchange limited, too.

• Cluster size: 5 Nodes - 8 vCPUs 64 GB RAM
• Protocol: amqpv1
• QoS: At-least-once (Publish-Confirms)
• Queue type: Classic queue, Non-Persistent Messages
• Message rate when the missing publish-confirms are starting.: ~> 15.000 Msg/Sec on the default exchange but many different queues

Any idea how to measure if an exchange has reached a capacity limit?
Which erlang process has the exchange logic / compute time for answering the publish (inter-node publish-confirm)?
Is it the queue itself?

Thanks in advance.

Best regards
Thomas

[mkuratczyk]

Routing is performed in the channel process:
https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_channel.erl#L1329

Therefore there is no way of measuring anything about the exchange directly, because exchanges are just lookups (there is no exchange process). However, concurrent publishers will put additional load on the ETS tables used for routing.

Since 3.11, there is a different (simpler/faster) code path) for direct exchange: #4606 but to be honest I don't see how the exchange type could affect the publisher confirm. It seems more likely to be related with the queue/channel being overloaded.

1. What version are you on right now?
2. Can you explain the message flow a bit more? How many publishers? How many bindings on that exchange? How many queues is a given message routed to, etc?
3. Any chance for an executable test case we can use to reproduce? It's ok if it doesn't always trigger the issue - I can keep it running until it does.

Thanks,

[michaelklishin]

Since default exchange uses a separate code path, it can be missing some things that all other exchanges go through via exchange type modules. But that's just a guess, we need an executable way to reproduce.

[lukebakken]

@motmot80 by amqpv1 you mean you're using the AMQP 1.0 protocol, correct?