Merge pull request kubernetes#3110 from denkensk/promote-non-preempting-to-GA

Graduate NonPreemptingPriority to GA

Author: k8s-ci-robot

keps/prod-readiness/sig-scheduling/902.yaml

@@ -0,0 +1,3 @@
+kep-number: 902
+stable:
+  approver: "@wojtek-t"

keps/sig-scheduling/902-non-preempting-priorityclass/README.md

@@ -16,9 +16,12 @@
   - [Graduation Criteria](#graduation-criteria)
     - [Alpha (v1.15):](#alpha-v115)
     - [Beta (v1.19):](#beta-v119)
+    - [Stable (v1.24):](#stable-v124)
 - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
   - [Feature enablement and rollback](#feature-enablement-and-rollback)
   - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+  - [Monitoring Requirements](#monitoring-requirements)
+  - [Dependencies](#dependencies)
   - [Scalability](#scalability)
   - [Troubleshooting](#troubleshooting)
 - [Implementation History](#implementation-history)
@@ -164,15 +167,19 @@ Ensure existing tests (for preempting PriorityClasses) do not break.
 ### Graduation Criteria
 #### Alpha (v1.15):
 
-- [x] Support NonPreemptingPriority in PriorityClasses
+- Support NonPreemptingPriority in PriorityClasses
 
 #### Beta (v1.19):
 
-- [ ] Add integration test for NonPreemptingPriority.
-- [ ] Graduate NonPreemptingPriority to Beta.
-- [ ] Update documents to reflect the changes.
-
+- Add integration test for NonPreemptingPriority.
+- Graduate NonPreemptingPriority to Beta.
+- Update documents to reflect the changes.
 
+#### Stable (v1.24):
+- No negative feedback.
+- Enhance the message of the existing event for scheduling failed to include details about preemption.
+- Graduate NonPreemptingPriority to GA.
+- Update documents to reflect the changes.
 
 ## Production Readiness Review Questionnaire
 
@@ -189,7 +196,7 @@ Ensure existing tests (for preempting PriorityClasses) do not break.
 
 * **Can the feature be disabled once it has been enabled (i.e. can we rollback
   the enablement)?**
-  Yes, the feature can be disabled if the PreemptionPolicy isn't set.
+  Yes. This feature can be disabled by restarting kube-apiserver and kube-scheduler with feature-gate turned off.
 
 * **What happens if we reenable the feature if it was previously rolled back?**
   If we reenable the feature, the Pod with high priority and NonPreemptionPolicy will be eligible to preempt other pods with low priority when cluster resources are tight.
@@ -199,18 +206,126 @@ Ensure existing tests (for preempting PriorityClasses) do not break.
 
 ### Rollout, Upgrade and Rollback Planning
 * **How can a rollout fail? Can it impact already running workloads?**
-  The scheduler errors and exits during start up. Existing workloads are not
-  affected.
+  If a rollout fails, kube-scheduler will keep crashing. Running workloads won't be affected by kube-scheduler.
 
 * **What specific metrics should inform a rollback?**
-  N/A.
+Check the following indicators to determine if there are any exceptions:
+  - pod_preemption_victims
+  - total_preemption_attempts
+  - scheduling_algorithm_preemption_evaluation_seconds
 
 * **Were upgrade and rollback tested? Was upgrade->downgrade->upgrade path tested?**
-  N/A.
+  Manually tested successfully. The test environment version is v1.23. We tested enabling and disabling this
+  feature. After each change in the feature-gate, 3 separate priorityclasses will be recreated (One
+  high-priorityclass with preemptionPolicy as Never, other high-priorityclass with preemptionPolicy not be
+  set, one low-priorityclass with preemptionPolicy not be set). Create multiple pods with the above 3
+  priorityclasses to verify that the preemption results are as expected.
 
 * **Is the rollout accompanied by any deprecations and/or removals of features?**
   N/A.
 
+### Monitoring Requirements
+
+<!--
+This section must be completed when targeting beta to a release.
+-->
+
+###### How can an operator determine if the feature is in use by workloads?
+The operator can determine if the workload is using the feature by checking if the priorityclass's preemptionPolicy is set to "Never".
+<!--
+Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
+checking if there are objects with field X set) may be a last resort. Avoid
+logs or events for this purpose.
+-->
+
+###### How can someone using this feature know that it is working for their instance?
+
+<!--
+For instance, if this is a pod-related feature, it should be possible to determine if the feature is functioning properly
+for each individual pod.
+Pick one more of these and delete the rest.
+Please describe all items visible to end users below with sufficient detail so that they can verify correct enablement
+and operation of this feature.
+Recall that end users cannot usually observe component logs or access metrics.
+-->
+
+- [x] Events
+  - Event Reason: There is an event sent by kube-scheduler if the pod preempts other pods. If the feature is working and the pod with the priorityclass'preemptionPolicy as Never, there won't be a preemption related event for this pod.
+- [ ] API .status
+  - Condition name:
+  - Other field:
+- [x] Other (treat as last resort)
+  - Details: Check if pods with preemptionPolicy set to Never can preempt other low-priority pods when the cluster resources cannot be met.  
+
+###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?
+N/A
+
+<!--
+This is your opportunity to define what "normal" quality of service looks like
+for a feature.
+
+It's impossible to provide comprehensive guidance, but at the very
+high level (needs more precise definitions) those may be things like:
+  - per-day percentage of API calls finishing with 5XX errors <= 1%
+  - 99% percentile over day of absolute value from (job creation time minus expected
+    job creation time) for cron job <= 10%
+  - 99.9% of /health requests per day finish with 200 code
+
+These goals will help you determine what you need to measure (SLIs) in the next
+question.
+-->
+
+###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+<!--
+Pick one more of these and delete the rest.
+-->
+
+- [x] Metrics
+  - Metric name: preemption_victims
+  - [Optional] Aggregation method:
+  - Components exposing the metric: kube-scheduler
+- [ ] Other (treat as last resort)
+  - Details:
+
+###### Are there any missing metrics that would be useful to have to improve observability of this feature? 
+We currently only have events that describe a pod being preempted by another pod. But we don't
+have an event that describes why sometimes the preemption is not successful. We can enhance the
+message of the existing event for scheduling failed to include details about preemption. This
+will help us to improve observability for this feature and other scenarios.
+
+In addition to events, we can add metrics about how many pods have stopped preempting other pods because of this no-preemption option. However, since the probability of this metric being used is likely to be small, it was not added.
+
+<!--
+Describe the metrics themselves and the reasons why they weren't added (e.g., cost,
+implementation difficulties, etc.).
+-->
+
+
+### Dependencies
+
+<!--
+This section must be completed when targeting beta to a release.
+-->
+
+###### Does this feature depend on any specific services running in the cluster?
+No.
+
+<!--
+Think about both cluster-level services (e.g. metrics-server) as well
+as node-level agents (e.g. specific version of CRI). Focus on external or
+optional services that are needed. For example, if this feature depends on
+a cloud provider API, or upon an external software-defined storage or network
+control plane.
+
+For each of these, fill in the following—thinking about running existing user workloads
+and creating new ones, as well as about cluster-level services (e.g. DNS):
+  - [Dependency name]
+    - Usage description:
+      - Impact of its outage on the feature:
+      - Impact of its degraded performance or high-error rates on the feature:
+-->
+
 ### Scalability
 * **Will enabling / using this feature result in any new API calls?**
   No
@@ -249,9 +364,6 @@ Ensure existing tests (for preempting PriorityClasses) do not break.
   - scheduling_algorithm_preemption_evaluation_seconds
 
 ## Implementation History
-
-[Original Github issue](https://github.com/kubernetes/kubernetes/issues/67671)
-
-Pod Priority and Preemption are tracked as part of [enhancement#564](https://github.com/kubernetes/enhancements/issues/564).
-The proposal for Pod Priority can be [found here](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/pod-priority-api.md)
-and Preemption proposal is [here](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/pod-preemption.md).
+- 2019-03-17: Initial KEP
+- 2020-05-19: Graduate the feature to Beta
+- 2022-01-15: Graduate the feature to GA

keps/sig-scheduling/902-non-preempting-priorityclass/kep.yaml

@@ -15,11 +15,12 @@ reviewers:
 approvers:
   - "bsalamat"
   - "Huang-Wei"
-stage: beta
-latest-milestone: "v1.19"
+stage: stable
+latest-milestone: "v1.24"
 milestone:
   alpha: "v1.15"
   beta: "v1.19"
+  stable: "v1.24"
 feature-gates:
   - name: NonPreemptingPriority
     components: