Proposal: Disable default debug logging for gVisor extension to prevent disk exhaustion #12825
Closed
pioklopk-gif
started this conversation in
Ideas
Replies: 2 comments 1 reply
-
|
Talos doesn't ship debug settings by default: https://github.com/siderolabs/extensions/blob/main/container-runtime/gvisor/runsc.toml Are you sure you're not using |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
10.10.10.10 runtime ExtensionStatus 1 1 gvisor 20260202.0 So maybe should I delete the gvisor-debug extension right ? Piotr Kloc |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Current behavior:
The Talos gVisor extension currently defaults to debug = "true" and log_level = "debug" in /etc/cri/conf.d/runsc.toml. When combined with a mutating admission controller (like Kyverno) that runs many pods under gVisor, it generates a massive amount of logs in /var/log/runsc/.
In my case, this led to 35GB of logs per node in just two days, causing critical DiskPressure and crashing the entire cluster.
Proposed change:
Set the default log_level to error and debug to false in the extension's default template. Debugging should be an opt-in feature, not a default, to ensure cluster stability out of the box.
Temporary fix applied:
I managed to stabilize my nodes by applying the following MachineConfig patch:
YAML
machine:
files:
- content: |
log_path = "/var/log/runsc/%ID%/shim.log"
log_level = "error"
Thanks
Piotr Kloc
Beta Was this translation helpful? Give feedback.
All reactions