weird OOMs after update to EKS 1.33

[makarov-roman]

Hi guys,
I've updated EKS cluster from 1.32 to 1.33 and noticed a strange behaviour. Around 20% of workloads went into crashloops with OOMs. Some of them I consider very stable, which didn't have any changes for months, if not years. also in some cases I had to increase the requests/limits by 2 times. e.g. 3->6, 2->4 to fix them. Nodes have plenty of spare memory, does it point at containerd?

I've checked containerd, bottlerocket and EKS/k8s 1.33 changelogs and didn't find a good enough explanation yet. Do you have any ideas what should I check to understand what's going on?

Thanks in advance

[piyush-jena]

Hi @makarov-roman !

We made several changes in the BR variants with k8s-1.33 like containerd-2.0 and kernel-6.12. The full changelog is here. This is concerning and we will create an issue and investigate it.

[piyush-jena]

Link to the issue: #4551

[piyush-jena]

@makarov-roman ,
Can you please let us know here or in the issue linked above the answers to these questions:

1. which bottlerocket version are you using? You can run apiclient get os and paste the output.
2. which type of instance are you using?
3. Can you paste the output of kubectl describe node?
4. Can you also attach the logdog output? The steps are described here
5. Can you please describe the kind of workload you are running?

[larvacea]

Issue #4549 is an OOM issue, and may be related, especially if your workloads expect to use cgroups v1 to govern their memory use (as the JVM does in #4549).

[makarov-roman]

@piyush-jena I checked the failing jobs. All of them are java.

[makarov-roman]

updated the Bottlerocket to 1.40 and the issue is resolved. thanks both of you for the help