Draft
Conversation
shayasoolin
requested review from
Ronkahn21,
gflarity,
sanjaychatterjee and
unmarshall
as code owners
kangclzjc
reviewed
Mar 3, 2026
| - Introduce a `workloadType` field to distinguish `Inference` (default, no behavior change) from `Training` workloads. | ||
| - Add a `phase` field to `PodCliqueSetStatus` representing the workload lifecycle: `Pending`, `Running`, `Succeeded`, `Failed`. | ||
| - Add `maxRuntime` and `maxRestarts` controls under a `trainingSpec` stanza. | ||
| - Default `terminationDelay` to `0` for training workloads for immediate failure response. |
Contributor
There was a problem hiding this comment.
If we set terminationDelay default to 0, does it means that gang termination be triggered when one pod fail/deleted? Wouldn't we wait for the failed/deleted pod to be created again?
kangclzjc
reviewed
Mar 3, 2026
|
|
||
| A new `workloadType` field is added to `PodCliqueSetSpec`. When set to `Training`, the operator changes its behavior in the following ways: | ||
|
|
||
| - Pods that exit with code 0 are not recreated. When all pods in a replica complete successfully, the replica is marked `Succeeded`. |
Contributor
There was a problem hiding this comment.
What happens if we support in-place update in the near future. If we in-place update the pod, would we consider it as success or fail? For this kind of action/job, we shouldn't block the recreate, right?
kangclzjc
reviewed
Mar 3, 2026
| maxRestarts: 2 | ||
| ``` | ||
|
|
||
| A "restart" means deleting and recreating all pods across all PodCliques in the affected replica. The counter is a single total across all replicas. The PodCliqueSet is marked `Failed` when the total restart count exceeds `maxRestarts`. |
Contributor
There was a problem hiding this comment.
Can we provide restart action also for PCSG and PCLQ level ?
kangclzjc
reviewed
Mar 3, 2026
|
|
||
| ### terminationDelay | ||
|
|
||
| `terminationDelay` (default: 4 hours) is the grace period grove gives after `MinAvailableBreached` before tearing down the gang. For inference workloads, this allows time for transient pod failures to recover. For training workloads, a missing worker means the job cannot make progress — a 4-hour wait wastes reserved GPU capacity and delays the retry. |
Contributor
There was a problem hiding this comment.
A training job can have several part, each part fail may don't affect other parts, all we need to do is to restart the part of job. A missing worker which may not mean the job can't make progress.
kangclzjc
reviewed
Mar 3, 2026
|
|
||
| `terminationDelay` (default: 4 hours) is the grace period grove gives after `MinAvailableBreached` before tearing down the gang. For inference workloads, this allows time for transient pod failures to recover. For training workloads, a missing worker means the job cannot make progress — a 4-hour wait wastes reserved GPU capacity and delays the retry. | ||
|
|
||
| When `workloadType: Training`, `terminationDelay` defaults to `0`. The operator immediately tears down the replica and begins a restart (or marks it `Failed` if `maxRestarts` is exhausted). Users can override this with an explicit value if their framework tolerates brief pod gaps. |
Contributor
There was a problem hiding this comment.
Could we add multiple level of Tear down the replica of PCS/PCSG/PCLQ
kangclzjc
reviewed
Mar 3, 2026
|
|
||
| #### Validating Webhook | ||
|
|
||
| - Reject pod template spec changes that would produce a new generation hash (i.e., would trigger a rolling update) for Training workloads. |
Contributor
There was a problem hiding this comment.
We should support in-place update which could change spec. Maybe we should allow this action
kangclzjc
reviewed
Mar 3, 2026
|
|
||
| - **Phase computation** — on every reconcile, aggregate PodClique statuses to derive the phase: all pods pending → `Pending`; any replica running → `Running`; all PodCliques have `Succeeded` condition → `Succeeded`; restart budget exhausted or `maxRuntime` exceeded → `Failed`. | ||
| - **maxRuntime enforcement** — record `startTime` in PCS status when phase first transitions to `Running`. `startTime` is never reset on restarts — `maxRuntime` is a wall-clock deadline from first start, inclusive of any time spent failing and restarting. This matches the behavior of Kubernetes Job's `activeDeadlineSeconds` and Kubeflow's `runPolicy.activeDeadlineSeconds`, and bounds total GPU consumption regardless of retry count. On each reconcile, check elapsed time; if elapsed > `maxRuntime`, terminate all pods and set phase `Failed`. Use `requeueAfter(remainingDuration)` to avoid busy-polling. | ||
| - **Restart orchestration** — when a PodClique has `MinAvailableBreached: True`, check `status.restartCount` vs `maxRestarts`. If budget remains, delete all pods across all PodCliques in the replica, increment `restartCount`, and emit a `ReplicaRestarting` event. If exhausted, terminate all remaining pods, set phase `Failed`, and emit `MaxRestartsExceeded`. |
Contributor
There was a problem hiding this comment.
One PodClique has MinAvailableBreached: True should avoid delete all PodCliques's all pods
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/kind documentation
/kind feature
What this PR does / why we need it:
Adds GREP-285 for training job support in
PodCliqueSet. The document covers theproposed API changes, lifecycle phase semantics,
trainingSpecfields (maxRuntime,maxRestarts),terminationDelaydefaulting, webhook enforcement, conditions, and events needed to support finite trainingworkloads alongside existing inference workloads.
Which issue(s) this PR fixes:
Fixes #285
Special notes for your reviewer:
This is a design/proposal PR only — no code changes. Feedback welcome on API naming,
maxRestartscountingsemantics, and the
terminationDelaydefaulting behavior for training workloads.Does this PR introduce a API change?
Extends the
PodCliqueSetAPI with aworkloadTypefield (Inference/Training),a
phasefield in status, and atrainingSpecstanza (maxRuntime,maxRestarts)to support finite training workloads alongside existing inference workloads.