Shovel not respecting prefetch for "no-ack" and "on-publish"?

[johanrhodin]

Consider the following scenario, two clusters connected by a shovel. The shovel is using these parameters:

    "ack-mode": "on-publish",
    "src-prefetch-count": 1

If the destination cluster reaches the high watermark the messages at the source cluster will accumulate. Unfortunately they don't accumulate in the source queue but in the connection writer.

This can be seen in the Erlang mailbox for the rabbit_writer that keeps on growing.

The exact shovel(s) this happens to can be found by using the technique outlined here: https://groups.google.com/g/rabbitmq-users/c/bJqiqyFuiUE/m/tJNZaUKyBwAJ

The only workaround that I've found for this is to use ack-mode on-confirm, but it would be good to have a way for less important data (that do not require a confirm) to not accumulate in the RAM of the sending node.

If this is considered to be "by design" I'd be happy to include that in the documentation for shovel. This issue comes up from time to time and is hard to reason about and to track down.

A more detailed reproduction can be seen in this gist: https://gist.github.com/johanrhodin/87a54361ddebc2c09c015d991385e329

[carlhoerberg]

How come erlang-rabbitmq-client can have an unbounded write buffer? connection_writer's mailbox seems to just grow and grow when the server is blocking publishes (not reading from its side of the socket).

[johanrhodin]

This is being addressed in #5715