Merge pull request #49004 from michellengnx/merged-main-dev-1.32

Merge main branch into dev-1.32

Author: k8s-ci-robot

content/bn/_index.html

@@ -12,7 +12,7 @@
 {{% blocks/feature image="flower" id="feature-primary" %}}
 [কুবারনেটিস]({{< relref "/docs/concepts/overview/" >}}), K8s নামেও পরিচিত, কনটেইনারাইজড অ্যাপ্লিকেশনের স্বয়ংক্রিয় ডিপ্লয়মেন্ট, স্কেলিং এবং পরিচালনার জন্য একটি ওপেন-সোর্স সিস্টেম।
 
-এটি সহজ ব্যবস্থাপনা এবং আবিষ্কারের জন্য লজিক্যাল ইউনিটে একটি অ্যাপ্লিকেশন তৈরি করে এমন কন্টেইনারগুলিকে গোষ্ঠীভুক্ত করে। কুবারনেটিস [Google-এ প্রোডাকশন ওয়ার্কলোড চালানোর 15 বছরের অভিজ্ঞতার ভিত্তিতে](http://queue.acm.org/detail.cfm?id=2898444) তৈরি করে, কমিউনিটির সেরা ধারণা এবং অনুশীলনের সাথে মিলিত ভাবে।
+এটি সহজ ব্যবস্থাপনা এবং আবিষ্কারের জন্য লজিক্যাল ইউনিটে একটি অ্যাপ্লিকেশন তৈরি করে এমন কন্টেইনারগুলিকে গোষ্ঠীভুক্ত করে। কুবারনেটিস [Google-এ প্রোডাকশন ওয়ার্কলোড চালানোর 15 বছরের অভিজ্ঞতার ভিত্তিতে](https://queue.acm.org/detail.cfm?id=2898444) তৈরি করে, কমিউনিটির সেরা ধারণা এবং অনুশীলনের সাথে মিলিত ভাবে।
 {{% /blocks/feature %}}
 
 {{% blocks/feature image="scalable" %}}

content/en/blog/_posts/2024-12-11-Kubernetes-v1-32-Release/index.md

@@ -0,0 +1,92 @@
+---
+layout: blog
+title: 'Kubernetes v1.32 Adds A New CPU Manager Static Policy Option For Strict CPU Reservation'
+draft: true
+date: 2024-12-11
+slug: cpumanager-strict-cpu-reservation
+author: >
+  [Jing Zhang](https://github.com/jingczhang) (Nokia)
+---
+
+In Kubernetes v1.32, after years of community discussion, we are excited to introduce a
+`strict-cpu-reservation` option for the [CPU Manager static policy](/docs/tasks/administer-cluster/cpu-management-policies/#static-policy-options).
+This feature is currently in alpha, with the associated policy hidden by default. You can only use the
+policy if you explicitly enable the alpha behavior in your cluster.
+
+
+## Understanding the feature
+
+The CPU Manager static policy is used to reduce latency or improve performance. The `reservedSystemCPUs` defines an explicit CPU set for OS system daemons and kubernetes system daemons. This option is designed for Telco/NFV type use cases where uncontrolled interrupts/timers may impact the workload performance. you can use this option to define the explicit cpuset for the system/kubernetes daemons as well as the interrupts/timers, so the rest CPUs on the system can be used exclusively for workloads, with less impact from uncontrolled interrupts/timers. More details of this parameter can be found on the [Explicitly Reserved CPU List](/docs/tasks/administer-cluster/reserve-compute-resources/#explicitly-reserved-cpu-list) page.
+
+If you want to protect your system daemons and interrupt processing, the obvious way is to use the `reservedSystemCPUs` option.
+
+However, until the Kubernetes v1.32 release, this isolation was only implemented for guaranteed
+pods that made requests for a whole number of CPUs. At pod admission time, the kubelet only
+compares the CPU _requests_ against the allocatable CPUs. In Kubernetes, limits can be higher than
+the requests; the previous implementation allowed burstable and best-effort pods to use up
+the capacity of `reservedSystemCPUs`, which could then starve host OS services of CPU - and we
+know that people saw this in real life deployments.
+The existing behavior also made benchmarking (for both infrastructure and workloads) results inaccurate.
+
+When this new `strict-cpu-reservation` policy option is enabled, the CPU Manager static policy will not allow any workload to use the reserved system CPU cores.
+
+
+## Enabling the feature
+
+To enable this feature, you need to turn on both the `CPUManagerPolicyAlphaOptions` feature gate and the `strict-cpu-reservation` policy option. And you need to remove the `/var/lib/kubelet/cpu_manager_state` file if it exists and restart kubelet.
+
+With the following kubelet configuration:
+
+```yaml
+kind: KubeletConfiguration
+apiVersion: kubelet.config.k8s.io/v1beta1
+featureGates:
+  ...
+  CPUManagerPolicyOptions: true
+  CPUManagerPolicyAlphaOptions: true
+cpuManagerPolicy: static
+cpuManagerPolicyOptions:
+  strict-cpu-reservation: "true"
+reservedSystemCPUs: "0,32,1,33,16,48"
+...
+```
+
+When `strict-cpu-reservation` is not set or set to false:
+```console
+# cat /var/lib/kubelet/cpu_manager_state
+{"policyName":"static","defaultCpuSet":"0-63","checksum":1058907510}
+```
+
+When `strict-cpu-reservation` is set to true:
+```console
+# cat /var/lib/kubelet/cpu_manager_state
+{"policyName":"static","defaultCpuSet":"2-15,17-31,34-47,49-63","checksum":4141502832}
+```
+
+
+## Monitoring the feature
+
+You can monitor the feature impact by checking the following CPU Manager counters:
+- `cpu_manager_shared_pool_size_millicores`: report shared pool size, in millicores (e.g. 13500m)
+- `cpu_manager_exclusive_cpu_allocation_count`: report exclusively allocated cores, counting full cores (e.g. 16)
+
+Your best-effort workloads may starve if the `cpu_manager_shared_pool_size_millicores` count is zero for prolonged time.
+
+We believe any pod that is required for operational purpose like a log forwarder should not run as best-effort, but you can review and adjust the amount of CPU cores reserved as needed.
+
+## Conclusion
+
+Strict CPU reservation is critical for Telco/NFV use cases. It is also a prerequisite for enabling the all-in-one type of deployments where workloads are placed on nodes serving combined control+worker+storage roles.
+
+We want you to start using the feature and looking forward to your feedback.
+
+
+## Further reading
+
+Please check out the [Control CPU Management Policies on the Node](/docs/tasks/administer-cluster/cpu-management-policies/)
+task page to learn more about the CPU Manager, and how it fits in relation to the other node-level resource managers.
+
+
+## Getting involved
+
+This feature is driven by the [SIG Node](https://github.com/Kubernetes/community/blob/master/sig-node/README.md). If you are interested in helping develop this feature, sharing feedback, or participating in any other ongoing SIG Node projects, please attend the SIG Node meeting for more details.

content/en/blog/_posts/2024-12-11-cpumanager-strict-cpu-reservation.md

@@ -0,0 +1,131 @@
+---
+layout: blog
+title: "Kubernetes v1.32: QueueingHint Brings a New Possibility to Optimize Pod Scheduling"
+date: 2024-12-12
+slug: scheduler-queueinghint
+draft: true
+Author: >
+  [Kensei Nakada](https://github.com/sanposhiho) (Tetrate.io)
+---
+
+The Kubernetes [scheduler](/docs/concepts/scheduling-eviction/kube-scheduler/) is the core
+component that selects the nodes on which new Pods run. The scheduler processes
+these new Pods **one by one**. Therefore, the larger your clusters, the more important
+the throughput of the scheduler becomes.
+
+Over the years, the Kubernetes project (and SIG Scheduling in particular) has improved the throughput
+of the scheduler in multiple enhancements. This blog post describes a major improvement to the
+scheduler in Kubernetes v1.32: a 
+[scheduling context element](/docs/concepts/scheduling-eviction/scheduling-framework/#extension-points)
+named _QueueingHint_. This page provides background knowledge of the scheduler and explains how
+QueueingHint improves scheduling throughput.
+
+## Scheduling queue
+
+The scheduler stores all unscheduled Pods in an internal component called the _scheduling queue_. 
+
+The scheduling queue consists of the following data structures:
+- **ActiveQ**: holds newly created Pods or Pods that are ready to be retried for scheduling.
+- **BackoffQ**: holds Pods that are ready to be retried but are waiting for a backoff period to end. The
+   backoff period depends on the number of unsuccessful scheduling attempts performed by the scheduler on that Pod.
+- **Unschedulable Pod Pool**: holds Pods that the scheduler won't attempt to schedule for one of the
+   following reasons:
+   - The scheduler previously attempted and was unable to schedule the Pods. Since that attempt, the cluster
+      hasn't changed in a way that could make those Pods schedulable.
+   - The Pods are blocked from entering the scheduling cycles by PreEnqueue Plugins, 
+for example, they have a [scheduling gate](/docs/concepts/scheduling-eviction/pod-scheduling-readiness/#configuring-pod-schedulinggates),
+and get blocked by the scheduling gate plugin.
+
+## Scheduling framework and plugins
+
+The Kubernetes scheduler is implemented following the Kubernetes
+[scheduling framework](/docs/concepts/scheduling-eviction/scheduling-framework/).
+
+And, all scheduling features are implemented as plugins
+(e.g., [Pod affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity)
+is implemented in the `InterPodAffinity` plugin.)
+
+The scheduler processes pending Pods in phases called _cycles_ as follows:
+1. **Scheduling cycle**: the scheduler takes pending Pods from the activeQ component of the scheduling
+    queue  _one by one_. For each Pod, the scheduler runs the filtering/scoring logic from every scheduling plugin. The
+    scheduler then decides on the best node for the Pod, or decides that the Pod can't be scheduled at that time.
+    
+    If the scheduler decides that a Pod can't be scheduled, that Pod enters the Unschedulable Pod Pool
+    component of the scheduling queue. However, if the scheduler decides to place the Pod on a node, 
+    the Pod goes to the binding cycle.
+    
+1. **Binding cycle**: the scheduler communicates the node placement decision to the Kubernetes API
+    server. This operation bounds the Pod to the selected node. 
+    
+Aside from some exceptions, most unscheduled Pods enter the unschedulable pod pool after each scheduling
+cycle. The Unschedulable Pod Pool component is crucial because of how the scheduling cycle processes Pods one by one. If the scheduler had to constantly retry placing unschedulable Pods, instead of offloading those
+Pods to the Unschedulable Pod Pool, multiple scheduling cycles would be wasted on those Pods.
+
+## Improvements to retrying Pod scheduling with QueuingHint
+
+Unschedulable Pods only move back into the ActiveQ or BackoffQ components of the scheduling
+queue if changes in the cluster might allow the scheduler to place those Pods on nodes. 
+
+Prior to v1.32, each plugin registered which cluster changes could solve their failures, an object creation, update, or deletion in the cluster (called _cluster events_),
+with `EnqueueExtensions` (`EventsToRegister`),
+and the scheduling queue retries a pod with an event that is registered by a plugin that rejected the pod in a previous scheduling cycle.
+
+Additionally, we had an internal feature called `preCheck`, which helped further filtering of events for efficiency, based on Kubernetes core scheduling constraints;
+For example, `preCheck` could filter out node-related events when the node status is `NotReady`. 
+
+However, we had two issues for those approaches:
+- Requeueing with events was too broad, could lead to scheduling retries for no reason.
+   - A new scheduled Pod _might_ solve the `InterPodAffinity`'s failure, but not all of them do.
+For example, if a new Pod is created, but without a label matching `InterPodAffinity` of the unschedulable pod, the pod wouldn't be schedulable.
+- `preCheck` relied on the logic of in-tree plugins and was not extensible to custom plugins,
+like in issue [#110175](https://github.com/kubernetes/kubernetes/issues/110175). 
+
+Here QueueingHints come into play; 
+a QueueingHint subscribes to a particular kind of cluster event, and make a decision about whether each incoming event could make the Pod schedulable.
+
+For example, consider a Pod named `pod-a` that has a required Pod affinity. `pod-a` was rejected in
+the scheduling cycle by the `InterPodAffinity` plugin because no node had an existing Pod that matched
+the Pod affinity specification for `pod-a`.
+
+{{< figure src="queueinghint1.svg" alt="A diagram showing the scheduling queue and pod-a rejected by InterPodAffinity plugin" caption="A diagram showing the scheduling queue and pod-a rejected by InterPodAffinity plugin" >}}
+
+`pod-a` moves into the Unschedulable Pod Pool. The scheduling queue records which plugin caused
+the scheduling failure for the Pod. For `pod-a`, the scheduling queue records that the `InterPodAffinity`
+plugin rejected the Pod.
+
+`pod-a` will never be schedulable until the InterPodAffinity failure is resolved. 
+There're some scenarios that the failure could be resolved, one example is an existing running pod gets a label update and becomes matching a Pod affinity.
+For this scenario, the `InterPodAffinity` plugin's `QueuingHint` callback function checks every Pod label update that occurs in the cluster. 
+Then, if a Pod gets a label update that matches the Pod affinity requirement of `pod-a`, the `InterPodAffinity`,
+plugin's `QueuingHint` prompts the scheduling queue to move `pod-a` back into the ActiveQ or
+the BackoffQ component.
+
+{{< figure src="queueinghint2.svg" alt="A diagram showing the scheduling queue and pod-a being moved by InterPodAffinity QueueingHint" caption="A diagram showing the scheduling queue and pod-a being moved by InterPodAffinity QueueingHint" >}}
+
+## QueueingHint's history and what's new in v1.32
+
+At SIG Scheduling, we have been working on the development of QueueingHint since
+Kubernetes v1.28.
+
+While QueuingHint isn't user-facing, we implemented the `SchedulerQueueingHints` feature gate as a
+safety measure when we originally added this feature. In v1.28, we implemented QueueingHints with a
+few in-tree plugins experimentally, and made the feature gate enabled by default.
+
+However, users reported a memory leak, and consequently we disabled the feature gate in a
+patch release of v1.28.  From v1.28 until v1.31, we kept working on the QueueingHint implementation
+within the rest of the in-tree plugins and fixing bugs.
+
+In v1.32, we made this feature enabled by default again. We finished implementing QueueingHints
+in all plugins and also identified the cause of the memory leak!
+
+We thank all the contributors who participated in the development of this feature and those who reported and investigated the earlier issues.
+
+## Getting involved
+
+These features are managed by Kubernetes [SIG Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling).
+
+Please join us and share your feedback. 
+
+## How can I learn more?
+
+- [KEP-4247: Per-plugin callback functions for efficient requeueing in the scheduling queue](https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/4247-queueinghint/README.md)

content/en/blog/_posts/2024-12-12-scheduler-queueinghint/index.md

@@ -486,6 +486,7 @@ Here are some examples of device plugin implementations:
 * [Akri](https://github.com/project-akri/akri), which lets you easily expose heterogeneous leaf devices (such as IP cameras and USB devices).
 * The [AMD GPU device plugin](https://github.com/ROCm/k8s-device-plugin)
 * The [generic device plugin](https://github.com/squat/generic-device-plugin) for generic Linux devices and USB devices
+* The [HAMi](https://github.com/Project-HAMi/HAMi) for heterogeneous AI computing virtualization middleware (for example, NVIDIA, Cambricon, Hygon, Iluvatar, MThreads, Ascend, Metax)
 * The [Intel device plugins](https://github.com/intel/intel-device-plugins-for-kubernetes) for
   Intel GPU, FPGA, QAT, VPU, SGX, DSA, DLB and IAA devices
 * The [KubeVirt device plugins](https://github.com/kubevirt/kubernetes-device-plugins) for