Bug: SandboxClaim Never Enters a Failure State

[igooch]

When a SandboxClaim cannot successfully provision a Sandbox or Pod, the controller continues to reconcile the resource indefinitely. In large-scale environments, this results in:

• Stale SandboxClaims accumulate in etcd and consume kube-apiserver bandwidth.
• There is no clear way to distinguish between a transient error and a persistent failure without a time-bound threshold.
• Users must manually identify and delete reconciling claims that have stalled.

Proposed Solution:

Introduce a timeout mechanism for the SandboxClaim lifecycle:

• Add a configurable timeout after which a claim is considered failed.
• If the Sandbox and Pod are not ready within the timeout period, the controller should take action to stop the reconciliation loop.

Open Questions:

Possible options for how to configure a timeout:

• Controller Flag: A global timeout applied to all SandboxClaims managed by the controller. This is simpler to implement and manage globally but lacks flexibility for specific workloads.
• CRD Field: Adding a timeout field to the SandboxClaim specification. This allows users to define custom timeouts per request, which may be useful for workloads with varying startup characteristics.

Possible options for desired behavior once a timeout is reached:

• Hard Deletion: The controller deletes the Sandbox, Pod, and the SandboxClaim to immediately free up resources.
• Failed Status: The controller retains the SandboxClaim but updates its status to Failed to provide a clear signal for debugging and observability.
• Failed Status to Deletion: Same as the "Failed Status", but the SandboxClaim is deleted after X period of time in a Failed status.

[aditya-shantanu]

My 2c:

1. We can do both. Have a controller flag but that is override-able through a CRD field.
   2.Failed status.

[barney-s]

Valid point on a claim unable to provision.
What are the possible conditions for this. I can think of a few :

• No pod capacity (compute)
• Pending pods for a long time (compute capacity).

Any other reason ?

Claim is like a sandbox, the creator owns the lifecycle.
To prevent accumulation of sandboxes we have expirytime.
Claims also support expirytime.

How is a timeout different from expirytime ?
How does the client reason about these ?

[janetkuo]

A similar pattern exists in Kubernetes Deployments via Progress Deadline Seconds which allows users to set a deadline before the controller marks progress as failed.

The controller shouldn't delete the SandboxClaim. Controllers should not mutate the user's intent, as this violates the declarative model.

[aditya-shantanu]

The controller shouldn't delete the SandboxClaim. Controllers should not mutate the user's intent, as this violates the declarative model

+1

[igooch]

Valid point on a claim unable to provision. What are the possible conditions for this. I can think of a few :

• No pod capacity (compute)
• Pending pods for a long time (compute capacity).

Any other reason ?

The main case that I ran into while testing was a matching SandboxTemplate was not found.

There also could be an issue with the SandboxTemplate itself, such as referencing a non-existent ServiceAccount, Secret, or ConfigMap.

And this issue would also apply when a Pod can be created, but never reaches a Ready state because of image pull errors, app misconfiguration, etc.

Claim is like a sandbox, the creator owns the lifecycle. To prevent accumulation of sandboxes we have expirytime. Claims also support expirytime.

How is a timeout different from expirytime ? How does the client reason about these ?

expirytime is derived from the ShutdownTime.

For instance a user might want a long-running Sandbox with an ShutdownTime of 4 hours from now. If a SandboxClaim cannot create a healthy running Sandbox and Pod, the user would likely want it to fail much faster than that and get a clear signal that there is some sort of creation issue. However, they can't change the ShutdownTime of the SandboxClaim to say 3 minutes, because then the SandboxClaim would delete its Sandbox and Pod after 3 minutes even if they were running and healthy.

agent-sandbox/extensions/api/v1alpha1/sandboxclaim_types.go

Lines 43 to 59 in dc3be48

	// Lifecycle defines the lifecycle management for the SandboxClaim.
	type Lifecycle struct {
	// ShutdownTime is the absolute time when the SandboxClaim expires.
	// This time governs the lifecycle of the claim. It is not propagated to the
	// underlying Sandbox. Instead, the SandboxClaim controller enforces this
	// expiration by deleting the Sandbox resources when the time is reached.
	// If this field is omitted or set to nil, the SandboxClaim itself won't expire.
	// This implies unsetting a Sandbox's ShutdownTime via SandboxClaim isn't supported.
	// +kubebuilder:validation:Format="date-time"
	// +optional
	ShutdownTime *metav1.Time `json:"shutdownTime,omitempty"`
	// ShutdownPolicy determines the behavior when the SandboxClaim expires.
	// +kubebuilder:default=Retain
	// +optional
	ShutdownPolicy ShutdownPolicy `json:"shutdownPolicy,omitempty"`
	}

So we want to differentiate between the expiry of a healthy SandboxClaim, and the expiry of a SandboxClaim that cannot ready a Ready running Sandbox and Pod within a configurable period of time.

[janetkuo]

@igooch would you clarify your goal? If you just want to be able to tell whether there's an issue when sandboxclaims create sandboxes/pods, this could be done by improving the sandboxclaim status, without adding progress deadline.

[igooch]

[@igooch](https://github.com/igooch) would you clarify your goal? If you just want to be able to tell whether there's an issue when sandboxclaims create sandboxes/pods, this could be done by improving the sandboxclaim status, without adding progress deadline.

There are three related goals:

1. Enforce Loop Termination: Transition SandboxClaim creation requests to a definitive terminal state—either success or failure—to prevent resources from entering infinite reconciliation loops.
2. Detect Stalled Requests: Provide clear diagnostic signals to reliably identify when a creation attempt has stalled or is cycling indefinitely without making progress.
3. Surface Fulfillment Failures: Effectively communicate failed creation attempts to both users and automated monitoring systems by providing actionable status updates and high-resolution metrics.