Quorum cpu iowait considerations on AWS

[andrii29]

Hello,
we recently started to use quorum queues for data safety, and found high CPU iowait, caused by disk.
Cluster setup:

• 3 x c6a.xlarge instances
• gp3 disks (3k iops, 125Mbps)
• rabbitmq:3.12.12-management
• queues: ~150 (stable number, arguments: x-queue-type: quorum, durable: true)
• message rates are 0.4 - 1k per second
• amqproxy for apps with high connection churn

Any suggestions or recommendations for raft/cluster tuning to optimize disk write pressure?

Cluster oprates normally, but we are worried about high cpu iowait

disk stats

All apps are symfony messenger, which does not rely on consume()
https://symfony.com/doc/5.x/messenger.html#amqp-transport

The consumers do not show up in an admin panel as this transport does not rely on \AmqpQueue::consume() which is blocking. Having a blocking receiver makes the --time-limit/--memory-limit options of the messenger:consume command as well as the messenger:stop-workers command inefficient, as they all rely on the fact that the receiver returns immediately no matter if it finds a message or not. The consume worker is responsible for iterating until it receives a message to handle and/or until one of the stop conditions is reached. Thus, the worker's stop logic cannot be reached if it is stuck in a blocking call.

rabbitmq-diagnostics runtime_thread_stats

Average thread real-time    : 5000322 us
Accumulated system run-time : 8774165 us
Average scheduler run-time  : 1046773 us

        Thread    alloc      aux      bifbusy_wait check_io emulator      ets       gc  gc_full      nif    other     port     send    sleep   timers

Stats per thread:
     async( 0)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
       aux( 1)    0.01%    0.02%    0.00%    0.00%    0.04%    0.00%    0.00%    0.00%    0.00%    0.00%    0.01%    0.00%    0.00%   99.93%    0.00%
dirty_cpu_( 1)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
dirty_cpu_( 2)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
dirty_cpu_( 3)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
dirty_cpu_( 4)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.01%    0.00%    0.00%    0.00%    0.00%   99.99%    0.00%
dirty_io_s( 1)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
dirty_io_s( 2)    0.02%    0.00%    0.00%    0.01%    0.00%    0.20%    0.00%    0.00%    0.00%   15.58%    0.04%    0.00%    0.00%   84.14%    0.00%
dirty_io_s( 3)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
dirty_io_s( 4)    0.01%    0.00%    0.00%    0.01%    0.00%    0.28%    0.00%    0.00%    0.00%   47.31%    0.06%    0.00%    0.00%   52.32%    0.00%
dirty_io_s( 5)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
dirty_io_s( 6)    0.00%    0.00%    0.00%    0.00%    0.00%    0.06%    0.00%    0.00%    0.00%   10.00%    0.01%    0.00%    0.00%   89.91%    0.00%
dirty_io_s( 7)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
dirty_io_s( 8)    0.01%    0.00%    0.00%    0.01%    0.00%    0.16%    0.00%    0.00%    0.00%   17.33%    0.04%    0.00%    0.00%   82.46%    0.00%
dirty_io_s( 9)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
dirty_io_s(10)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
      poll( 0)    0.04%    0.00%    0.00%    0.00%    0.45%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%   99.51%    0.00%
 scheduler( 1)    2.14%    1.22%    1.31%    0.50%    0.96%    4.43%    2.04%    0.43%    0.03%    0.02%    1.92%    4.99%    1.62%   78.35%    0.04%
 scheduler( 2)    2.18%    1.21%    1.33%    0.50%    0.96%    4.48%    2.10%    0.44%    0.03%    0.02%    1.92%    4.98%    1.67%   78.16%    0.03%
 scheduler( 3)    1.84%    1.07%    1.18%    0.50%    0.93%    3.86%    1.75%    0.38%    0.03%    0.02%    1.75%    3.87%    1.45%   81.34%    0.03%
 scheduler( 4)    2.10%    1.18%    1.35%    0.53%    0.99%    4.48%    2.11%    0.44%    0.03%    0.02%    1.90%    4.72%    1.67%   78.42%    0.04%

Stats per type:
         async    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
           aux    0.01%    0.02%    0.00%    0.00%    0.04%    0.00%    0.00%    0.00%    0.00%    0.00%    0.01%    0.00%    0.00%   99.93%    0.00%
dirty_cpu_sche    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%    0.00%
dirty_io_sched    0.00%    0.00%    0.00%    0.00%    0.00%    0.07%    0.00%    0.00%    0.00%    9.02%    0.02%    0.00%    0.00%   90.88%    0.00%
          poll    0.04%    0.00%    0.00%    0.00%    0.45%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%   99.51%    0.00%
     scheduler    2.06%    1.17%    1.29%    0.51%    0.96%    4.32%    2.00%    0.42%    0.03%    0.02%    1.87%    4.64%    1.60%   79.07%    0.04%

[kjnilsson]

If you use a "polling consumer" or basic.get as the frame is called in the AMQP 0.9.1 protocol you are going to see some disk use even when there are no messages in the system. Every basic.get to a quorum queue is modelled as a single message consumer command so it is a command being written to the raft log, replicated and replied to. It is very inefficient and wasteful but most likely why you see this overhead.

I have no further advice apart from using a long running consumer.

[kjnilsson]

Confirmed

[michaelklishin]

basic.get has always been recommended against. The docs these days are very explicit.

I'm afraid we cannot convince maintainers not use old features recommended against.

[lukebakken]

get is the natural way to consume messages via symfony framework

Symfony needs to address this.

[andrii29]

Thank you all for your advice!
We'll try to use consumers instead of basic.get in our apps. In the meantime, as an interim solution, we've replaced our quorum queues with mirrored classic queues version 2 (deprecated). This adjustment has notably reduced disk I/O while still ensuring data replication.

[michaelklishin]

@andrii29 sorry but using classic mirrored queues voids your warranty at this point. Yes, they are less disk I/O heavy, and also inferior in every other way, most notably data safety.

Our team would not recommend this solution to anyone. Two wrongs don't make a right.