5x higher cpu usage after update from 3.11.4 to 3.11.13 on low loaded cluster

[oprudkyi]

related to bitnami/charts#16141

• on exactly the same config I have increase in CPU from 200m to 1CPU (in gke cpu units)
• both configs - with classic queues in single-instance rabbitmq and quorum queues with 3 replica ha cluster are affected
• some replicas isn't affected
• the cpu use increases in an hour or few hours without any issues/pikes/etc - just start to use more cpu

the load is small - mostly up to 50-100 messages per second and 3.11.4 handles it without any issues

I tried to compare rabbitmq-diagnostics observer on affected and non-affected replicas and don't see any meaningful data
the only outstanding value is erl_prim_loader - 1.7MB on non-affected and 4.1478 MB on affected one

when comparing with 3.11.4 the 'Gc Count' is 2-3x higher in 3.11.13 most of time (150-250 vs 250-600)

[mkuratczyk]

Could you try the versions in-between to identify the exact version that introduced the regression?
Could you try to reproduce using https://perftest.rabbitmq.com/ and provide the flags used?
Could you start observer and sort by reductions (rr[ENTER]) and see if there are differences in what's on top?

Ideally, capturing flamegraphs (https://blog.rabbitmq.com/posts/2022/05/flame-graphs/ or https://parca.dev/) would provide useful info, but this would require a bit more set up.

[oprudkyi]

Hi @mkuratczyk

thank you.
I checked observer and seems values the same, the only difference in TotalHeapSize - node with higher cpu has erl_prim_loader 520Kb while low load node has 220.0586 KB

command line used is

/opt/bitnami/erlang/lib/erlang/erts-13.2/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -S 2:2 -B i -- -root /opt/bitnami/erlang/lib/erlang -bindir /opt/bitnami/erlang/lib/erlang/erts-13.2/bin -progname erl -- -home /opt/bitnami/rabbitmq/.rabbitmq -- -pa  -noshell -noinput -s rabbit boot -boot start_sasl -syslog logger [] -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false

I will try to gather more data later

[oprudkyi]

downgrading to 3.11.4 didn't help either. seems there are some other sort of problems

[oprudkyi]

fixed by updating liveness probes (in k8) bitnami/charts#11116 (comment)