Skip to content

Change container workload's default OOM Score#522

Merged
alexmwu merged 1 commit intogoogle:mainfrom
alexmwu:oomscoreadj
Jan 6, 2025
Merged

Change container workload's default OOM Score#522
alexmwu merged 1 commit intogoogle:mainfrom
alexmwu:oomscoreadj

Conversation

@alexmwu
Copy link
Contributor

@alexmwu alexmwu commented Jan 3, 2025

We currently rely on the random OOM Killer to delete processes when we have out of memory issues. This can lead to nondeterministic and potentially exploitable behavior where the OOM Killer can delete other sensitive processes. This change now makes the container workload a very likely candidate for the OOMKiller when the CS VM is experiencing out of memory issues.

Manually tested:

Before

wuale@cs-dbg-241000 ~ $ ps -ef --forest
root        1238       1  0 04:54 ?        00:00:00 /usr/bin/containerd-shim-run
root        1258    1238  0 04:54 ?        00:00:00  \_ nginx: master process ng
101         1295    1258  0 04:54 ?        00:00:00      \_ nginx: worker proces
101         1296    1258  0 04:54 ?        00:00:00      \_ nginx: worker proces
wuale@cs-dbg-241000 ~ $ cat /proc/1258/oom_score_adj
-998
wuale@cs-dbg-241000 ~ $ cat /proc/1295/oom_score_adj
-998
wuale@cs-dbg-241000 ~ $ cat /proc/1296/oom_score_adj
-998

After

wuale@cs-dbg-20250103-oomscoreadj ~ $ ps -ef --forest
root        1134       1  0 04:29 ?        00:00:04 /usr/bin/fluent-bit -c /etc/
root        2885       1  0 06:29 ?        00:00:00 /usr/bin/containerd-shim-run
root        2904    2885  0 06:29 ?        00:00:00  \_ nginx: master process ng
101         2938    2904  0 06:29 ?        00:00:00      \_ nginx: worker proces
101         2939    2904  0 06:29 ?        00:00:00      \_ nginx: worker proces
wuale@cs-dbg-20250103-oomscoreadj ~ $ cat /proc/2904/oom_score_adj
1000
wuale@cs-dbg-20250103-oomscoreadj ~ $ cat /proc/2938/oom_score_adj
1000
wuale@cs-dbg-20250103-oomscoreadj ~ $ cat /proc/2939/oom_score_adj
1000
wuale@cs-dbg-20250103-oomscoreadj ~ $

@alexmwu
Copy link
Contributor Author

alexmwu commented Jan 3, 2025

/gcbrun

@alexmwu
Copy link
Contributor Author

alexmwu commented Jan 3, 2025

/gcbrun

@alexmwu
Copy link
Contributor Author

alexmwu commented Jan 3, 2025

/gcbrun

@alexmwu
Copy link
Contributor Author

alexmwu commented Jan 3, 2025

/gcbrun

This makes the container workload a very likely candidate for the
OOMKiller when the CS VM is experiencing out of memory issues.
@alexmwu
Copy link
Contributor Author

alexmwu commented Jan 3, 2025

/gcbrun

@alexmwu alexmwu requested a review from jkl73 January 3, 2025 06:39
@alexmwu alexmwu merged commit ef8a29b into google:main Jan 6, 2025
11 checks passed
@alexmwu alexmwu deleted the oomscoreadj branch January 6, 2025 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants