kswapd0 eating cpu on control plane when running out of RAM

[valkenburg-prevue-ch]

Hi all, I have a question. In a simple dev setup (I tear it down daily, no need for HA etc), I have a single CX21 control plane node (2cpu, 4GB RAM) with several agent nodes.

While playing around with a tool which launches a shipload of jobs, I got to the point that the k3s-server process is at 80% of the available memory on the control plane node. Next, the node becomes unresponsive (the k8s api is unresponsive, more exactly). Ssh-ing into the machine, I observe that k3s-server is hitting the limits (that's ok, I know what I did and that I need a bigger machine if I want to continue doing this), but also that kswapd0 is at 80%-100% of cpu usage. About a few thousand linux fora teach me that this happens what kswapd0 starts looking for swap space when there isn't any. Indeed, on our microos setup, swapping is turned off. (Running commands like echo 1 > /proc/sys/vm/drop_caches and swapoff -a and sysctl vm.swappiness=0 have absolutely no impact here.)

Is there anything that could/should be done to avoid kswapd0 from making things worse? Is there any setting which tells kswapd0 to never even look for swap space to begin with?

Just to repeat: I know that this is probably never an issue if I use a bigger control-plane machine, but I'm raising this here as someone else might also run into this for other more legitimate reasons, and perhaps one of the more linux-savvy people here actually knows what to do with this.

[mysticaltech]

@valkenburg-prevue-ch FYI (From GPT-4):

kswapd0 is a kernel process that is responsible for freeing up memory pages when the system is running out of physical memory. When your k3s-server process is consuming a lot of memory, kswapd0 tries to find pages that can be moved into swap space to free up physical memory. If there's no swap space configured, kswapd0 can be stuck in a loop trying to free up memory, causing high CPU usage.

The fact that setting swappiness to 0 or trying to disable swapping (with swapoff -a) doesn't impact your situation is expected. These commands would only be effective if you had swap space configured. The swappiness parameter controls the tendency of the kernel to move processes out of physical memory and onto the swap disk. Because you don't have any swap, the setting has no real effect.

In a situation where there is no swap space, and memory is being heavily used, the kernel can start killing processes to free up memory, guided by the Out-Of-Memory (OOM) killer. This could explain why your k3s-server process becomes unresponsive.

There's no setting to tell kswapd0 to not look for swap space because that's fundamentally what it's designed to do. Instead, there are a few options to address your problem:

1. Increase physical memory: Upgrading your machine to one with more RAM may solve the issue if your workload is memory-intensive.

2. Configure swap space: Even a small swap partition can help alleviate memory pressure and give kswapd0 someplace to move pages when physical memory is low. However, remember that swapping can lead to significant I/O and consequent performance degradation.

3. Limit memory usage of your pods: Kubernetes allows you to specify memory requests and limits for your pods. By appropriately configuring these, you can prevent a single pod from consuming all the available memory.

4. Optimize your application: If your tool is consuming a lot of memory, there may be opportunities to optimize it to reduce its memory usage. This could involve things like tuning garbage collection (in languages that support it), reducing caching, or optimizing data structures and algorithms.

5. Use a Kubernetes distribution designed for low-resource environments: If you're running in a resource-constrained environment, you might want to consider using a Kubernetes distribution designed for such scenarios. For example, K3s is a lightweight distribution that requires only 512MB of RAM to run.

Remember that not having enough memory for your workloads can lead to unpredictable behavior and can make your system unstable. It's always a good idea to monitor your memory usage and ensure you have enough resources for your workloads.

[valkenburg-prevue-ch]

Thanks