Merge pull request kubernetes#5128 from nokia/4540-strict-cpu-reservation-beta

KEP-4540: Move to beta

Author: k8s-ci-robot

keps/prod-readiness/sig-node/4540.yaml

@@ -1,4 +1,5 @@
 kep-number: 4540
 alpha:
   approver: "@soltysh"
-
+beta:
+  approver: "@soltysh"

keps/sig-node/4540-strict-cpu-reservation/README.md

@@ -42,20 +42,20 @@
 
 Items marked with (R) are required *prior to targeting to a milestone / release*.
 
-- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
 - [x] (R) KEP approvers have approved the KEP status as `implementable`
 - [x] (R) Design details are appropriately documented
 - [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
-  - [ ] e2e Tests for all Beta API Operations (endpoints)
-  - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) 
-  - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
+  - [x] e2e Tests for all Beta API Operations (endpoints)
+  - [x] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+  - [x] (R) Minimum Two Week Window for GA e2e tests to prove flake free
 - [ ] (R) Graduation criteria is in place
   - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) 
-- [ ] (R) Production readiness review completed
+- [x] (R) Production readiness review completed
 - [ ] (R) Production readiness review approved
 - [x] "Implementation History" section is up-to-date for milestone
-- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
-- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+- [x] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [x] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
 
 [kubernetes.io]: https://kubernetes.io/
 [kubernetes/enhancements]: https://git.k8s.io/enhancements
@@ -73,15 +73,14 @@ With this KEP, a new `CPUManager` policy option `strict-cpu-reservation` is intr
 
 The static policy is used to reduce latency or improve performance. If you want to move system daemons or interrupt processing to dedicated cores, the obvious way is use the `reservedSystemCPUs` option. But in current implementation this isolation is implemented only for guaranteed pods with integer CPU requests not for burstable and best-effort pods (and guaranteed pods with fractional CPU requests).
 Admission is only comparing the cpu requests against the allocatable cpus. Since the cpu limit are higher than the request, it allows burstable and best-effort pods to use up the capacity of `reservedSystemCPUs` and cause host OS services to starve in real life deployments.
-Custom CPU allocation policies deployed as NRI plugins (e.g. Balloons) can separate infrastructure and workload into different CPU pools but they require extra software, additional tuning and reduced CPU pool size could affect performance of multi-threaded processes.
 
 ### Goals
  * Align scheduler and node view for Node Allocatable (total - reserved).
  * Ensure `reservedSystemCPUs` is only used by system daemons or interrupt processing not by workloads.
  * Ensure no breaking changes for the `static` policy of `CPUManager`.
 
 ### Non-Goals
- * Change scheduler interface to sub-partition `cpu` resource (as described in the archived Risk Mitigation Option 1).
+ * Change interface between node and scheduler.
 
 ## Proposal
 
@@ -109,10 +108,6 @@ With the following Kubelet configuration:
 ```yaml
 kind: KubeletConfiguration
 apiVersion: kubelet.config.k8s.io/v1beta1
-featureGates:
-  ...
-  CPUManagerPolicyOptions: true
-  CPUManagerPolicyAlphaOptions: true
 cpuManagerPolicy: static
 cpuManagerPolicyOptions:
   strict-cpu-reservation: "true"
@@ -123,7 +118,7 @@ reservedSystemCPUs: "0,32,1,33,16,48"
 When `strict-cpu-reservation` is disabled:
 ```console
 # cat /var/lib/kubelet/cpu_manager_state
-{"policyName":"static","defaultCpuSet":"0-79","checksum":1241370203}
+{"policyName":"static","defaultCpuSet":"0-63","checksum":1058907510}
 ```
 
 When `strict-cpu-reservation` is enabled:
@@ -134,7 +129,7 @@ When `strict-cpu-reservation` is enabled:
 
 ### Risks and Mitigations
 
-The feature is isolated to a specific policy option `strict-cpu-reservation` under `cpuManagerPolicyOptions` and is protected by feature gate `CPUManagerPolicyAlphaOptions` or `CPUManagerPolicyBetaOptions` before the feature graduates to `Stable` i.e. enabled by default.
+The feature is isolated to a specific policy option `strict-cpu-reservation` under `cpuManagerPolicyOptions` and is protected by feature gate `CPUManagerPolicyBetaOptions` before the feature graduates to `Stable` i.e. always enabled.
 
 Concern for feature impact on best-effort workloads, the workloads that do not have resource requests, is brought up.
 
@@ -144,11 +139,11 @@ The concern is, when the feature graduates to `Stable`, it will be enabled by de
 
 However, this is exactly the feature intent, best-effort workloads have no KPI requirement, they are meant to consume whatever CPU resources left on the node including starving from time to time. Best-effort workloads are not scheduled to run on the `reservedSystemCPUs` so they shall not be run on the `reservedSystemCPUs` to destablize the whole node.
 
-Nevertheless, risk mitigation has been discussed in details (see archived options below) and we agree to start with the following node metrics of cpu pool sizes in Alpha stage to assess the actual impact in real deployment before revisiting if we need risk mitigation.
+Nevertheless, risk mitigation has been discussed in details (see archived options below) and we agree to start with the following node metrics of cpu pool sizes in Alpha and Beta stages to assess the actual impact in real deployment. The plan is to move the current implementation to Stable stage if no field issue is observed for one year.
 
 https://github.com/kubernetes/kubernetes/pull/127506
-- `cpu\_manager\_shared\_pool\_size\_millicores`: report shared pool size, in millicores (e.g. 13500m), expected to be non-zone otherwise best-effort pods will starve
-- `cpu\_manager\_exclusive\_cpu\_allocation\_count`: report exclusively allocated cores, counting full cores (e.g. 16)
+- `cpu_manager_shared_pool_size_millicores`: report shared pool size, in millicores (e.g. 13500m), expected to be non-zone otherwise best-effort pods will starve
+- `cpu_manager_exclusive_cpu_allocation_count`: report exclusively allocated cores, counting full cores (e.g. 16)
 
 
 #### Archived Risk Mitigation (Option 1)
@@ -184,7 +179,6 @@ kind: KubeletConfiguration
 apiVersion: kubelet.config.k8s.io/v1beta1
 featureGates:
   ...
-  CPUManagerPolicyOptions: true
   CPUManagerPolicyAlphaOptions: true
 cpuManagerPolicy: static
 cpuManagerPolicyOptions:
@@ -298,7 +292,7 @@ No new integration tests for kubelet are planned.
   - CPU Manager works with `strict-cpu-reservation` policy option
 
 - Basic functionality
-1. Enable `CPUManagerPolicyAlphaOptions` feature gate and `strict-cpu-reservation` policy option.
+1. Enable `strict-cpu-reservation` policy option.
 2. Create a simple pod of Burstable QoS type.
 3. Verify the pod is not using the reserved CPU cores.
 4. Delete the pod.
@@ -313,8 +307,9 @@ No new integration tests for kubelet are planned.
 
 #### Beta
 
-- [ ] Gather feedback from consumers of the new policy option.
-- [ ] Verify no major bugs reported in the previous cycle.
+- [X] Gather feedback from consumers of the new policy option.
+- [X] Verify no major bugs reported in the previous cycle.
+- [X] Ensure proper e2e tests are in place.
 
 #### GA
 
@@ -333,33 +328,32 @@ No changes needed.
 
 ### Feature Enablement and Rollback
 
-The `/var/lib/kubelet/cpu\_manager\_state` needs to be removed when enabling or disabling the feature.
+The `/var/lib/kubelet/cpu_manager_state` needs to be removed when enabling or disabling the feature.
 
 ###### How can this feature be enabled / disabled in a live cluster?
 
 - [X] Feature gate (also fill in values in `kep.yaml`)
-  - Feature gate name: `CPUManagerPolicyAlphaOptions`
+  - Feature gate name: `CPUManagerPolicyBetaOptions`
   - Components depending on the feature gate: `kubelet`
 - [X] Change the kubelet configuration to set a `CPUManager` policy of `static` and a `CPUManager` policy option of `strict-cpu-reservation`
   - Will enabling / disabling the feature require downtime of the control plane? No
-  - Will enabling / disabling the feature require downtime or reprovisioning of a node?  No -- removing `/var/lib/kubelet/cpu\_manager\_state` and restarting kubelet are enough.
+  - Will enabling / disabling the feature require downtime or reprovisioning of a node?  No -- removing `/var/lib/kubelet/cpu_manager_state` and restarting kubelet are enough.
 
 
 ###### Does enabling the feature change any default behavior?
 
 Yes. Reserved CPU cores will be strictly used for system daemons and interrupt processing no longer available for workloads.
 
 The feature is only enabled when all following conditions are met:
-1. The `CPUManagerPolicyAlphaOptions` feature gate must be enabled
-2. The `static` `CPUManager` policy must be selected
-3. The new `strict-cpu-reservation` policy option must be selected
-4. The `reservedSystemCPUs` is not empty
+1. The `static` `CPUManager` policy is selected
+2. The `CPUManagerPolicyBetaOptions` feature gate is enabled and the `strict-cpu-reservation` policy option is selected
+3. The `reservedSystemCPUs` is not empty
 
 ###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
 
-Yes, the feature can be disabled by either:
-1. Disable feature gate `CPUManagerPolicyAlphaOptions` or remove `strict-cpu-reservation` from the list of `CPUManager` policy options
-2. Remove `/var/lib/kubelet/cpu\_manager\_state` and restart kubelet
+Yes, the feature can be disabled by:
+1. Disable feature gate `CPUManagerPolicyBetaOptions` or remove `strict-cpu-reservation` from the list of `CPUManager` policy options
+2. Remove `/var/lib/kubelet/cpu_manager_state` and restart kubelet
 
 ###### What happens if we reenable the feature if it was previously rolled back?
 
@@ -381,7 +375,7 @@ If the feature rollout fails, burstable and best-efforts continue to run on the
 If the feature rollback fails, burstable and best-efforts continue not to run on the reserved CPU cores.
 In either case, existing workload will not be affected.
 
-When enabling or disabling the feature, make sure `/var/lib/kubelet/cpu\_manager\_state` is removed before restarting kubelet otherwise kubelet restart could fail.
+When enabling or disabling the feature, make sure `/var/lib/kubelet/cpu_manager_state` is removed before restarting kubelet otherwise kubelet restart could fail.
 
 <!--
 Try to be as paranoid as possible - e.g., what if some components will restart
@@ -410,8 +404,54 @@ Describe manual testing that was done and the outcomes.
 Longer term, we may want to require automated upgrade/rollback tests, but we
 are missing a bunch of machinery and tooling and can't do that now.
 -->
+If you have this feature enabled in v1.32 under `CPUManagerPolicyAlphaOptions` (default to false) you will continue to have the feature enabled in v1.33 under `CPUManagerPolicyBetaOptions` (default to true) automatically i.e. no extra action is needed.
+To enable or disable this feature in v1.33, follow the feature activation and de-activation procedures described above.
+
+Manual upgrade->downgrade->upgrade testing from v1.32 to v1.33 is as follows:
+
+With the following Kubelet configuration and `cpu_manager_state` v1.32:
+
+```yaml
+kind: KubeletConfiguration
+apiVersion: kubelet.config.k8s.io/v1beta1
+featureGates:
+  CPUManagerPolicyAlphaOptions: true
+  ...
+cpuManagerPolicy: static
+cpuManagerPolicyOptions:
+  strict-cpu-reservation: "true"
+reservedSystemCPUs: "0,32,1,33,16,48"
+...
+```
 
-We manually test it in our internal environment and it works.
+```console
+# cat /var/lib/kubelet/cpu_manager_state
+{"policyName":"static","defaultCpuSet":"2-15,17-31,34-47,49-63","checksum":4141502832}
+```
+
+The same Kubelet `cpu_manager_state` will be seen after upgrading to v1.33:
+```console
+# cat /var/lib/kubelet/cpu_manager_state
+{"policyName":"static","defaultCpuSet":"2-15,17-31,34-47,49-63","checksum":4141502832}
+```
+
+You are recommended to remove the `CPUManagerPolicyAlphaOptions` feature gate after upgrading to v1.33 for operational integrity, but it is not mandatory.
+
+If you want to disable the feature in v1.33, you can either disable the `CPUManagerPolicyBetaOptions` feature gate, or remove the `strict-cpu-reservation` policy option. Remember to remove the `/var/lib/kubelet/cpu_manager_state` file before restarting kubelet.
+
+The following `cpu_manager_state` will be seen after the feature is disabled:
+```console
+# cat /var/lib/kubelet/cpu_manager_state
+{"policyName":"static","defaultCpuSet":"0-63","checksum":1058907510}
+```
+
+If you want to enable the feature in v1.33, you need to make sure the `CPUManagerPolicyBetaOptions` feature gate is not disabled and add the `strict-cpu-reservation` policy option. Remember to remove the `/var/lib/kubelet/cpu_manager_state` file before restarting kubelet.
+
+The following `cpu_manager_state` will be seen after the feature is enabled:
+```console
+# cat /var/lib/kubelet/cpu_manager_state
+{"policyName":"static","defaultCpuSet":"2-15,17-31,34-47,49-63","checksum":4141502832}
+```
 
 ###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
 
@@ -425,7 +465,7 @@ No.
 
 ###### How can an operator determine if the feature is in use by workloads?
 
-Inspect the `defaultCpuSet` in `/var/lib/kubelet/cpu\_manager\_state`:
+Inspect the `defaultCpuSet` in `/var/lib/kubelet/cpu_manager_state`:
 - When the feature is disabled, the reserved CPU cores are included in the `defaultCpuSet`.
 - When the feature is enabled, the reserved CPU cores are not included in the `defaultCpuSet`.
 
@@ -447,9 +487,9 @@ This feature allows users to protect infrastructure services from bursty workloa
 
 ###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
 
-https://github.com/kubernetes/kubernetes/pull/127506:
-- `cpu\_manager\_shared\_pool\_size\_millicores`: report shared pool size, in millicores (e.g. 13500m), expected to be non-zone otherwise best-effort pods will starve
-- `cpu\_manager\_exclusive\_cpu\_allocation\_count`: report exclusively allocated cores, counting full cores (e.g. 16)
+Monitor the following kubelet counters:
+- `cpu_manager_shared_pool_size_millicores`: report shared pool size, in millicores (e.g. 13500m), expected to be non-zone otherwise best-effort pods will starve
+- `cpu_manager_exclusive_cpu_allocation_count`: report exclusively allocated cores, counting full cores (e.g. 16)
 
 ###### Are there any missing metrics that would be useful to have to improve observability of this feature?
 
@@ -520,6 +560,7 @@ You can safely disable the feature.
 
 - 2024-03-08: Initial KEP created
 - 2024-10-07: KEP gets LGTM and Approval
+- 2025-02-03: KEP updated with Beta criteria
 
 
 ## Drawbacks

keps/sig-node/4540-strict-cpu-reservation/kep.yaml

@@ -8,20 +8,19 @@ status: implementable
 creation-date: 2024-03-06
 reviewers:
   - "@ffromani"
-  - "@klueska"                                                                          
   - "@swatisehgal"                                                                      
 approvers:
-  - "@sig-node-tech-leads"                                                              
+  - "@ffromani"
 see-also: []
 replaces: []
 
 # The target maturity stage in the current dev cycle for this KEP.
-stage: alpha
+stage: beta
 
 # The most recent milestone for which work toward delivery of this KEP has been
 # done. This can be the current (upcoming) milestone, if it is being actively
 # worked on.
-latest-milestone: "v1.32"
+latest-milestone: "v1.33"
 
 # The milestone at which this feature was, or is targeted to be, at each stage.
 milestone:
@@ -32,9 +31,10 @@ milestone:
 # The following PRR answers are required at alpha release
 # List the feature gate name and the components for which it must be enabled
 feature-gates:
-  - name: "CPUManagerPolicyAlphaOptions"                                                
-    components:                                                                         
+  - name: "CPUManagerPolicyBetaOptions"
+    components:
       - kubelet
+
 disable-supported: true
 
 # The following PRR answers are required at beta release