Customize back-pressure mechanisms

[flefebure]

Hello,

We have been using Rabbit for a few years for our process-mining platform and we have experienced several episodes of panic due to bad configuration of limits (RAM/Disk watermarks) (Rabbit really doesn't like disk overruns).

But well configured, it poses other problems:

Indeed, our processing pipeline has several stages, all of which use Rabbit queues. So we have primary producers who post in the queue of the first stage, we don't control their traffic. But subsequently, during processing, each stage will consume the queue of its stage and publish in the queue of the following stage, also then can publish in "retries" queues which allow us to manage replays following transient errors.

This means that if, for example following an overrun of the "disk-low-watermark", due to a swelling of the queue of the first stage, if Rabbit blocks the publications, there will be no chance in being able to unstack this queue and left the back pressure mechanism.

So I was wondering if it was possible to customize the back-pressure mechanism. Ideally, we would like 2 thresholds:

• A first where we want to block the publication on certain specific queues
• a second corresponding to the standard

I specify that coding does not scare us, we just want opinions on the subject.

Franck

[michaelklishin]

In environments where alarms are likely, we recommend using separate connections for publishers and consumers. This way consumer connections won't be blocked.

There is no way to configure back pressure. Blocking only some publishers won't achieve the goal
the alarm mechanism was created to achieve.

[michaelklishin]

If you don't control your publishers then a "DMZ" cluster sounds reasonable. It's the only way to isolate physical resources since virtual hosts only offer "logical" (access) separation.

[michaelklishin]

Actually now that I think of it, since quorum queues in 3.10 have much lower footprint and higher throughput than in 3.9, and support message TTL, you might be able to avoid memory usage spikes with uncontrolled publishers (up to a point). Even then, isolating this cluster sounds like a good idea still.

[flefebure]

After careful consideration, I think we are going to implement this DMZ/MAIN dual Rabbit cluster pattern (thanks for the name "DMZ") .

• The DMZ cluster will only have one exchange/queue ("rawdata"), producers are public, we want to throttle them. Consumers are the first stage of our platform.
• Little disk / conservative watermarks configurations for this DMZ cluster to be sure to throttle producers as soon as possible (and not put the MAIN cluster at risk)
• Big disk for the MAIN cluster, to avoid as much as possible producers blockage. As explained theses cause whole platform deadlock.
• On the MAIN cluster, message that flow to the "rawdata" queue from the rawdata.retries* queues will be brought back to the DMZ "rawdata" queue with a shovel

... It should work..
We will use it to upgrade rabbit version, actually 3.6.5 since several years in prod. If it can enhance memory footprint, its good for us..

[mkuratczyk]

Great to hear you have a plan. I'd just mention that disk-level limits is not the only option. You can set overflow=reject-publish on the queue level to set a per-queue length limit and reject messages above the limit. This could also be used in the DMZ cluster as an additional mechanism, on top of disk alarms. It could give you more flexibility - if there is a queue per publisher or per a group of publishers, you could have different limits for different publisher (groups).

[michaelklishin]

That's one feature I always forget about, even though it has been around for three years or so? Thank you, @mkuratczyk.

[flefebure]