feat: Implement SandboxWarmPool recreate on template updates by shrutiyam-glitch · Pull Request #347 · kubernetes-sigs/agent-sandbox

shrutiyam-glitch · 2026-02-26T00:40:29Z

Fixes #323

This PR implements the rollout logic for SandboxWarmPool when its associated SandboxTemplate is updated.

This change adds two update strategies—Recreate and OnReplenish—allowing users to control how stale pods are handled.

Changes:

1. Update Strategy Definition
Added a new UpdateStrategy field to the SandboxWarmPool specification.

Recreate (Default): Ensures the pool contains only fresh sandboxes by immediately deleting stale sandboxes when the template is updated.
OnReplenish: Retains existing sandboxes even if they are stale; they are only replaced after being manually deleted or claimed from the pool. This is applicable for any changes in the associated SandboxTemplate as well as the change of name of sandboxTemplateRef in SandboxWarmpool.

2. Template Versioning and Tracking

Template Hashing: Implemented a hashing mechanism for name and spec.podTemplate of the SandboxTemplate content to detect changes accurately.
Semantic Equality Check: If the template hashes of the current template and the label value in the pod are not matching, then the Semantic equality between the spec.podTemplate.spec of the current template and spec.podTemplate.spec of the current sandbox are compared. This comparison boolean value is stored with the hash as the key, to avoid this comparison for every sandbox in the warmpool.
Pod Labeling: Every pod created by the warm pool is now labeled with a hash of the specific template version used during its creation.

3. Controller Logic Updates

Template Watches: The SandboxWarmPool controller now watches SandboxTemplate resources. It triggers a reconciliation for any SandboxWarmPool that references a modified template. It uses an EnqueueRequestsFromMapFunc to identify and reconcile all warmpools referencing a modified template.
Staleness Detection: During reconciliation, the controller identifies "stale" pods by comparing their template hash label against the current template's hash.
Automated Rollout: If the strategy is set to Recreate, the controller deletes identified stale pods, allowing the standard reconciliation loop to replace them with pods based on the new template.

Testing Performed:

Verified that updating a SandboxTemplate image triggers the deletion of idle sandboxes in the associated SandboxWarmPool.
Verified that sandboxes already bound to a SandboxClaim are NOT deleted during a template update.
Verified that changing the templateRef (with new spec) on the SandboxWarmPool itself triggers a full pool rotation.
Verified that changing the templateRef (with old spec) on the SandboxWarmPool itself triggers a full pool rotation.

Unit tests added:

TestReconcilePool_TemplateUpdateRollout
TestFindWarmPoolsForTemplate

Steps followed:

Created SandboxTemplate, SandboxWarmpool with default Recreate strategy and SandboxClaim

$ kubectl get sandbox,sandboxclaim,sandboxtemplate -n sandbox-test
NAME                                                AGE
sandbox.agents.x-k8s.io/python-sdk-warmpool-kvjrz   3m
sandbox.agents.x-k8s.io/python-sdk-warmpool-wltr6   3m11s
sandbox.agents.x-k8s.io/python-sdk-warmpool-xhxwg   3m11s

NAME                                                          AGE
sandboxclaim.extensions.agents.x-k8s.io/sandbox-claim-new-1   3m2s

NAME                                                                 AGE
sandboxtemplate.extensions.agents.x-k8s.io/pct                       111m
sandboxtemplate.extensions.agents.x-k8s.io/python-counter-template   23h

Sandbox python-sdk-warmpool-wltr6 is adopted by the sandbox-claim.

Updated the spec of python-counter-template and applied.

$ kubectl get sandbox -n sandbox-test
NAME                        AGE
python-sdk-warmpool-wk89c   22s
python-sdk-warmpool-wltr6   4m24s
python-sdk-warmpool-zt7mk   21s

The unclaimed warmpool sandboxes are recreated with the new updated template spec.

Updated the spec.sandboxTemplateRef in the SandboxWarmPool manifest to use the sandbox template pct. All the unclaimed sandboxes are by default recreated.

$ kubectl get sandbox -n sandbox-test
NAME                        AGE
python-sdk-warmpool-2pdfp   5s
python-sdk-warmpool-5cqbv   4s
python-sdk-warmpool-wltr6   5m59s

netlify · 2026-02-26T00:40:35Z

✅ Deploy Preview for agent-sandbox canceled.

Name	Link
🔨 Latest commit	`1af3e86`
🔍 Latest deploy log	https://app.netlify.com/projects/agent-sandbox/deploys/69bda4a74b99310008cfe0e5

k8s-ci-robot · 2026-02-26T00:40:39Z

Hi @shrutiyam-glitch. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dhenkel92

Thank you for working on this feature. It's painful to roll out changes to a warm pool right now 🙂

extensions/controllers/sandboxwarmpool_controller.go

shrutiyam-glitch · 2026-03-03T18:18:09Z

/retest

igooch

Overall good work automating the warmpool pod recreation via the new spec hash label.

A few minor suggestions for performance and edge cases in the inline comments.

extensions/controllers/sandboxwarmpool_controller.go

lyj7890 · 2026-03-04T15:35:04Z

This feature would be really helpful for our use case! Looking forward to the merge. 👍

k8s-ci-robot · 2026-03-10T01:41:45Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shrutiyam-glitch
Once this PR has been reviewed and has the lgtm label, please ask for approval from janetkuo. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

igooch

Good work! A few comments, but the bulk of the logic looks good.

It would be good to have an e2e test for this, although that does not need to be a part of this PR.

extensions/api/v1alpha1/sandboxwarmpool_types.go

igooch · 2026-03-12T06:22:34Z

extensions/controllers/utils.go

Since the contents of this file are only used within the Warmpool controller I'd recommend moving them back into that file.

igooch · 2026-03-12T06:28:06Z

extensions/controllers/utils.go

+
+	// SandboxWarmPool update strategies
+	RecreateStrategy = "Recreate"
+	OnDeleteStrategy = "OnDelete"


Recommend putting these in the api/v1alpha1 type definition, with similar naming pattern as StatefulSet, so that API consumers can use the values.

https://github.com/kubernetes/kubernetes/blob/802b3f744b2378b8dc4e880e326b8ad620ffc2fe/pkg/apis/apps/types.go#L77-L92

igooch · 2026-03-12T17:05:37Z

extensions/controllers/sandboxwarmpool_controller.go

+		}
+
+		if isOrphan {
 			// Pod has no controller - adopt it


This only happens if the strategy is OnDelete, correct? Could you update the comment to reflect this?

So, if the strategy is Recreate and if the orphaned pod is not stale, the controller will go ahead and adopt the pod.

igooch · 2026-03-12T18:27:02Z

extensions/controllers/sandboxwarmpool_controller_test.go

 	}
 }
+
+func TestReconcilePool_TemplateUpdateRollout(t *testing.T) {


Could you update these to use the testCases := []struct{} ... for _, tc := range testCases {} pattern here?

janetkuo · 2026-03-13T00:00:56Z

extensions/controllers/sandboxwarmpool_controller.go

 }

 // reconcilePool ensures the correct number of pods exist in the pool
 func (r *SandboxWarmPoolReconciler) reconcilePool(ctx context.Context, warmPool *extensionsv1alpha1.SandboxWarmPool) error {


This function is getting really long and hard to review/maintain. Break it up into smaller helper functions. See go/small-functions

janetkuo · 2026-03-13T00:06:08Z

extensions/controllers/sandboxwarmpool_controller.go

+	template, tmplErr := r.getTemplate(ctx, warmPool)
+	var currentTemplateHash string
+	if tmplErr == nil {
+		currentTemplateHash = computeTemplateHash(template)


It's not safe to directly compute template hash for comparison, because pod schema changes in upstream will affect the value even if the template hasn't changed. This will end up deleting all warmed resources after a cluster upgrade. The value should only be computed once and added as a label, and later get the hash value from labels to compare. The same pattern is used in Deployment with pod-template-hash label.

Per discussion, if the Warmpool can create sandboxes directly (#390), we can use sandbox spec to compare directly.

brandonroyal · 2026-03-18T14:58:33Z

This is a relatively small nit but I'd suggest we reconsider the name OnDelete here as it implies that manual deletion is the event that triggers the new SandboxTemplate version to be used. Instead, we might consider something like OnReplenish which means that the new SandboxTemplate pods are deployed in the pool as they're replenished. That could be from a delete event or from the pod being claimed.

shrutiyam-glitch · 2026-03-19T17:46:31Z

/hold
dependent on #395

shrutiyam-glitch · 2026-03-20T20:27:01Z

/unhold

shrutiyam-glitch added 2 commits February 25, 2026 17:58

Enable sandboxwarmpool on template updates

6eeb647

Fetch template once

b1b115b

k8s-ci-robot requested review from janetkuo and justinsb February 26, 2026 00:40

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 26, 2026

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 26, 2026

shrutiyam-glitch changed the title ~~feat: Implement SandboxWarmPool rollout on template updates~~ feat: Implement SandboxWarmPool recreate on template updates Feb 26, 2026

dhenkel92 reviewed Feb 27, 2026

View reviewed changes

extensions/controllers/sandboxwarmpool_controller.go Outdated Show resolved Hide resolved

janetkuo added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 27, 2026

Fix lint

9171db5

igooch reviewed Mar 4, 2026

View reviewed changes

Added additional test case checks

8e7c8b1

janetkuo self-assigned this Mar 9, 2026

Add UpdateStrategy

9d6e10a

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 10, 2026

Merge branch 'main' into swp-rollout

7271c40

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Mar 10, 2026

shrutiyam-glitch requested a review from igooch March 10, 2026 17:53

shrutiyam-glitch added 2 commits March 10, 2026 10:56

Merge branch 'main' into swp-rollout

f5eba0a

fix:lint

d19f5ba

igooch reviewed Mar 12, 2026

View reviewed changes

janetkuo reviewed Mar 13, 2026

View reviewed changes

shrutiyam-glitch added 2 commits March 13, 2026 02:29

Address comments

acac0a3

Merge branch 'main' into swp-rollout

592e271

janetkuo mentioned this pull request Mar 16, 2026

Revisit KEP - Warmpool creates Sandboxes vs Pods #390

Closed

dhenkel92 mentioned this pull request Mar 17, 2026

Support RollingUpdate strategy for SandboxWarmPool template rollouts #426

Closed

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Mar 19, 2026

shrutiyam-glitch added 2 commits March 20, 2026 17:57

Merge branch 'main' into swp-rollout

bc85cc0

Update isSandboxStale

774211b

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 20, 2026

fix: ut

1af3e86

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 20, 2026

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 20, 2026

Conversation

shrutiyam-glitch commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for agent-sandbox canceled.

Uh oh!

k8s-ci-robot commented Feb 26, 2026

Uh oh!

dhenkel92 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shrutiyam-glitch commented Mar 3, 2026

Uh oh!

igooch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lyj7890 commented Mar 4, 2026

Uh oh!

k8s-ci-robot commented Mar 10, 2026

Uh oh!

igooch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

igooch Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

igooch Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

igooch Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

shrutiyam-glitch Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

igooch Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

janetkuo Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janetkuo Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janetkuo Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

brandonroyal commented Mar 18, 2026

Uh oh!

shrutiyam-glitch commented Mar 19, 2026

Uh oh!

shrutiyam-glitch commented Mar 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

shrutiyam-glitch commented Feb 26, 2026 •

edited

Loading

netlify bot commented Feb 26, 2026 •

edited

Loading

janetkuo Mar 13, 2026 •

edited

Loading

janetkuo Mar 13, 2026 •

edited

Loading