add drawbacks and call DRA out of scope

Author: kannon92

keps/sig-apps/5440-mutable-job-pod-resource-updates/README.md

@@ -88,6 +88,7 @@ tags, and then generate with `hack/update-toc.sh`.
     - [Story 1](#story-1)
   - [Risks and Mitigations](#risks-and-mitigations)
 - [Design Details](#design-details)
+  - [DRA Support](#dra-support)
   - [Test Plan](#test-plan)
     - [Unit tests](#unit-tests)
     - [Integration tests](#integration-tests)
@@ -107,6 +108,7 @@ tags, and then generate with `hack/update-toc.sh`.
 - [Implementation History](#implementation-history)
 - [Drawbacks](#drawbacks)
 - [Alternatives](#alternatives)
+  - [Delete and Recreate Jobs](#delete-and-recreate-jobs)
 - [Infrastructure Needed (Optional)](#infrastructure-needed-optional)
 <!-- /toc -->
 
@@ -128,11 +130,11 @@ checklist items _must_ be updated for the enhancement to be released.
 
 Items marked with (R) are required *prior to targeting to a milestone / release*.
 
-- [] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
-- [] (R) KEP approvers have approved the KEP status as `implementable`
-- [] (R) Design details are appropriately documented
-- [] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
-  - [ ] e2e Tests for all Beta API Operations (endpoints)
+- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [x] (R) KEP approvers have approved the KEP status as `implementable`
+- [x] (R) Design details are appropriately documented
+- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+  - [] e2e Tests for all Beta API Operations (endpoints)
   - [ ] (R) Ensure GA e2e tests for meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
   - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
 - [] (R) Graduation criteria is in place
@@ -159,7 +161,7 @@ to allow suspending jobs to control when the Pods of a Job get created by contro
 This was proposed as a primitive to allow a higher-level queue controller to implement
 job queuing: the queue controller unsuspends the job when resources become available.
 
-To complement the above capability, a queue controller may also want to control the
+To complement the above capability, a secondary controller may also want to control the
 resource requirements of a job based on current cluster capacity or resource availability.
 For example, it may want to adjust CPU, memory, and GPU requests/limits based on available
 node capacity, allocate specific extended resources like TPUs or FPGAs, optimize resource
@@ -168,7 +170,7 @@ priority and cluster load.
 
 This is a proposal to relax update validation on suspended jobs to allow mutating
 resource specifications in the job's pod template, specifically CPU, memory, GPU,
-and other extended resource requests and limits. This enables a higher-level queue
+and other extended resource requests and limits. This enables a higher-level
 controller to optimize resource allocation before un-suspending a job based on
 current cluster conditions and resource availability.
 
@@ -187,11 +189,10 @@ there's no way to optimize them based on actual cluster conditions when the job
 ready to run.
 
 Adding the ability to mutate a job's resource requirements while it's suspended gives
-a queue controller the ability to optimize resource allocation based on real-time
+a controller the ability to optimize resource allocation based on real-time
 cluster conditions, improve overall cluster utilization, and ensure jobs are sized
 appropriately for current capacity constraints.
 
-
 ### Goals
 
 - Allow mutating CPU, memory, GPU, and extended resource requests and limits of a container within a PodTemplate of a suspended jobs.
@@ -207,6 +208,7 @@ appropriately for current capacity constraints.
 - Allow mutating other job specifications beyond container resource requirements.
 - Support in-place pod resource updates (this is covered by separate KEPs).
 - Allow mutating of Pod Resources.
+- Allow mutating of ResourceClaims.
 
 ## Proposal
 
@@ -262,6 +264,15 @@ We will allow updates to the following fields in container specifications within
 - `resources.limits.memory`
 - `resources.limits.*` (for extended resources like `nvidia.com/gpu`, `amd.com/gpu`, `tpu-v4` etc.)
 
+### DRA Support
+
+DRA does not allow changing ResourceClaimTemplates once they are created.
+At the moment, relaxing mutability constraits of ResourceClaimTemplates or ResourceClaims is not in scope.
+To add support for this feature with DRA, the recommendation is to recreate ResourceClaimTemplates that match the
+desired resources.
+
+One does not have to modify claims in the PodTemplate so one can still assume claims are immutable also.
+
 ### Test Plan
 
 - Unit and integration tests verifying that:
@@ -531,7 +542,20 @@ This allows for more mutability of Jobs, particularly around resource specificat
 
 ## Alternatives
 
-NA
+
+### Delete and Recreate Jobs
+
+One option is to keep this immutability and any modification of a Job should require a delete and create.
+
+If there is a higher level controller controlling this Jobs with its own Owner References, then deletion is required on all resources that this job
+originally referenced.
+This is a common use case for JobSet which manages multiple jobs and services.
+Recreation would require deleting all existing jobs on an update if JobSet wanted to add support on updating JobTemplates while suspended.
+
+In a multicluster scenario, deletion and recreation may require dispatching amoung different clusters.
+Think of a scenario where you have 1 hub cluster and many worker clusters. If uses the hub to dispatch to a worker cluster, then this would require one to delete
+the Job and propogate that deletion to the clusters. A patch is only a single operation so this would be faster.
+
 ## Infrastructure Needed (Optional)
 
 NA