CPU footprint pattern changed after upgrading from GKE 1.23 to 1.24

[Jojoooo1]

Describe the bug

After upgrading from GKE 1.23 to 1.24, CPU use seems instable as shown in the picture below (the peak just started after the reboot on the new node). It is only happening for one node.

I am currently using bitnami helm chart 11.14.4 which uses RabbitMQ 3.11.15

Reproduction steps

1. create cluster with GKE 1.23 and upgrade to GKE 1.24
2. follow rabbitmq resources.

Expected behavior

CPU use should be stable.

Additional context

No response

[michaelklishin]

If you upgraded GKE and observed a difference, why do you expect the RabbitMQ core team to be looking into it?

We cannot suggest much with the amount of information provided. Consider collecting additional
metrics from your Kubernetes and RabbitMQ clusters first.

RabbitMQ docs provide a set of recommendations for reducing CPU footprint of mostly idle clusters.

[Jojoooo1]

Trying to investigate

[michaelklishin]

This looks like a node that has just booted, and that's it. A code loader module and async task work pool are the top contributors. Then application controller and the code (cache) server are number 3 and number 4.

This node has booted and is doing nothing, which is further confirmed by its grand memory footprint of 80 MiB.

[michaelklishin]

This is exactly the kind of environment that these recommendations are for.

What in the new GKE version may have caused a CPU spike to 0.4 of one core, I won't speculate, we don't have any evidence that this condition is permanent.

[michaelklishin]

@Jojoooo1 are you sure you have any reasons to file this and ask for our attention? This node is seemingly doing nothing at all, both according to my observations above #8142, and the fact that 99% of runtime scheduler's time is spent sleeping.

[Jojoooo1]

I thought that It might have been related to GKE 1.24 sorry If got to the wrong direction. Having worked with the same cluster (and many upgrade) for more than 3 years and never had been any similar situations. This is why I got pretty surprise, even after starting to receive messages it was not getting stabilized.

[Jojoooo1]

@michaelklishin sorry for the convenience seems it might related to bitnami/charts@83827ce