[ws-manager-mk2] do cleanup of failed workspace with unknown status #20829
Conversation
| return fmt.Sprintf("container %s completed; containers of a workspace pod are not supposed to do that", cs.Name), nil | ||
| } | ||
| } else if !isPodBeingDeleted(pod) && terminationState.ExitCode != containerUnknownExitCode { | ||
| } else if !isPodBeingDeleted(pod) && terminationState.ExitCode == containerUnknownExitCode { |
There was a problem hiding this comment.
In very old versions of Kubernetes, if there was a network issue with the node, the kubelet could not report pod status for a while, and the API server will mark the container status of the pod would be marked as "unknown". Therefore, our previous approach was to treat this situation as a temporary problem and expect it to recover automatically.
But in the recent version, the API server does not change the status of the pod; instead, it modifies the status of the node to mark it as NotReady. Therefore, this unknown situation is actually returned by containerd.
There was a problem hiding this comment.
In very old versions of Kubernetes
How old, are you able to tell where the cut off is?
There was a problem hiding this comment.
Adding approval to unblock.
Please check K3s version for gitpod.io is not impacted before removing hold.
There was a problem hiding this comment.
Pull Request Overview
This PR addresses the cleanup of failed workspaces with an unknown exit status by updating the controller’s status extraction logic and adding a corresponding test case.
- Updated the logic in workspace status extraction to handle an unknown container exit code.
- Added a new test case to simulate a workspace failure with an unknown exit code.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| components/ws-manager-mk2/controllers/workspace_controller_test.go | Added a test to simulate and verify cleanup behavior for workspaces failing with an unknown exit code. |
| components/ws-manager-mk2/controllers/status.go | Adjusted the exit code check and updated the error message for workspaces terminating with an unknown exit code. |
|
testing now, will then review code |
|
stop and restart are fine. |
|
With a running workspace, after forcing reboot I see: After services start we see: And for the workspace I see: And if I try restarting the failed workspace: |
|
Unit tests run (had to bump timeout to 5m), otherwise we panic at 30s mark: |
There was a problem hiding this comment.
👋 @iQQBot prior to removing hold, can you double check K3s version for gitpod.io is not impacted? Ref: #20829 (comment)
|
I was wrong, we know this version of K3s isn't impacted (as the preview env uses the same version). However, it might be interesting, to restart the CNI pod, and measure/gauge impact. If it is fast, I expect no change, and if it is slow I expect similar behavior. Please check beforehand? |
/unhold CNI does not affect this logic |
Description
[ws-manager-mk2] do cleanup of failed workspace with unknown status
Related Issue(s)
Fixes CLC-1372
How to test
workspace container xxxx terminated for an unknown reasonin ws-manager-mk2 podDocumentation
Preview status
Gitpod was successfully deployed to your preview environment.
Build Options
Build
Run the build with werft instead of GHA
Run Leeway with
--dont-testPublish
Installer
Add desired feature flags to the end of the line above, space separated
Preview Environment / Integration Tests
If enabled this will build
install/previewIf enabled this will create the environment on GCE infra
Saves cost. Untick this only if you're really sure you need a non-preemtible machine.
Valid options are
all,workspace,webapp,ide,jetbrains,vscode,ssh. If enabled,with-previewandwith-large-vmwill be enabled./hold