OOMKilled Brokers since Java17 Update

[siegenthalerroger]

Hi,

We've identified an issue with our strimzi powered kafka cluster that is causing regular (ca 5. day intervals) OOMKilled restarts of our brokers. We've been able to locate the issue as being caused by our upgrade from Strimzi 0.32.0 to 0.33.1 and/or the upgrade from Kafka 3.3.1 to 3.3.2. We are also still observing this behaviour on the latest strimzi 0.33.2 and kafka 3.4.0.

The images below show a distinct change in GC behaviour and an analysis of the changes in the updates leads us to believe that the java17 update has led to the encountered issue. I am wondering if there's any specific GC properties that could be set to tame the new behaviour or if this is potentially a memory leak on the kafka side (see last image).

Before (strimzi 0.32.0, kafka 3.3.1, java 11):

After (strimzi 0.33.1, kafka 3.3.2, java 17):

Memory Leak? Note the run-up of memory with GC unable to bring it back down (apparently)

[scholzj]

Is the container killed by the OOM killer? I think in that case, maybe you should tune the resources of the container / Java memory settings as that suggests that Java tries to consume more memory than it should. You can also use the jvmOptions section to tune the various JVM settings if you want.

[siegenthalerroger]

Yeah it was. Thanks for the tip about the options, just saw that they were misconfigured (not enough free space at all). I am left wondering about the endless growth of the jvm memory, but I'll keep an eye on it with the much lowered heap maxima.

[scholzj]

The GC behavior changes are interesting as well. But to be honest, my Java knowledge is not deep enough to know the details of what changed.