Merge pull request #36682 from ktothill/BZ-1903943

BZ-1903943: low latency cpu partitioning chapter lacks info

Author: bergerhoffer

modules/cnf-cpu-infra-container.adoc

@@ -5,17 +5,55 @@
 [id="cnf-cpu-infra-container_{context}"]
 = Restricting CPUs for infra and application containers
 
-You can reserve cores (threads) from a single NUMA node for operating system housekeeping tasks and put your workloads on another NUMA node. Partitioning the CPUs this way can prevent the housekeeping processes from impacting latency-sensitive application processes. By default, CRI-O uses all online CPUs to run infra containers on nodes, which can result in context switches and spikes in latency. 
+Generic housekeeping and workload tasks use CPUs in a way that may impact latency-sensitive processes. By default, the container runtime uses all online CPUs to run all containers together, which can result in context switches and spikes in latency. Partitioning the CPUs prevents noisy processes from interfering with latency-sensitive processes by separating them from each other. The following table describes how processes run on a CPU after you have tuned the node using the Performance Add-On Operator:
 
-You can ensure that housekeeping tasks and workloads run on separate NUMA nodes by specifying two groups of CPUs in the `spec` section of the performance profile. 
+.Process' CPU assignments
+[%header,cols=2*] 
+|===
+|Process type
+|Details
 
-* `isolated` - The CPUs for the application container workloads. These CPUs have the lowest latency. Processes in this group have no interruptions and can, for example, reach much higher DPDK zero packet loss bandwidth.
+|Burstable and best-effort pods
+|Runs on any CPU except where low latency workload is running
 
-* `reserved` - The CPUs for the cluster and operating system housekeeping duties, including pod infra containers. Threads in the `reserved` group tend to be very busy, so latency-sensitive applications should be run in the `isolated` group. See link:https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed[Create a pod that gets assigned a QoS class of `Guaranteed`].
+|Infrastructure pods
+|Runs on any CPU except where low latency workload is running
+
+|Interrupts
+|Redirects to reserved CPUs (optional in {product-title} {product-version} and later)
+
+|Kernel processes
+|Pins to reserved CPUs
+
+|Latency-sensitive workload pods
+|Pins to a specific set of exclusive CPUs from the isolated pool
+
+|OS processes/systemd services
+|Pins to reserved CPUs
+|===
+
+The exact partitioning pattern to use depends on many factors like hardware, workload characteristics and the expected system load. Some sample use cases are as follows:
+
+* If the latency-sensitive workload uses specific hardware, such as a network interface card (NIC), ensure that the CPUs in the isolated pool are as close as possible to this hardware. At a minimum, you should place the workload in the same Non-Uniform Memory Access (NUMA) node.
+
+* The reserved pool is used for handling all interrupts. When depending on system networking, allocate a sufficiently-sized reserve pool to handle all the incoming packet interrupts. In {product-version} and later versions, workloads can optionally be labeled as sensitive.
+
+The decision regarding which specific CPUs should be used for reserved and isolated partitions requires detailed analysis and measurements. Factors like NUMA affinity of devices and memory play a role. The selection also depends on the workload architecture and the specific use case.
+
+[IMPORTANT]
+====
+The reserved and isolated CPU pools must not overlap and together must span all available cores in the worker node.
+====
+
+To ensure that housekeeping tasks and workloads do not interfere with each other, specify two groups of CPUs in the `spec` section of the performance profile. 
+
+* `isolated` - Specifies the CPUs for the application container workloads. These CPUs have the lowest latency. Processes in this group have no interruptions and can, for example, reach much higher DPDK zero packet loss bandwidth.
+
+* `reserved` - Specifies the CPUs for the cluster and operating system housekeeping duties. Threads in the `reserved` group are often busy. Do not run latency-sensitive applications in the `reserved` group. Latency-sensitive applications run in the `isolated` group.
 
 .Procedure
 
-. Create a performance profile that is appropriate for your hardware and topology.
+. Create a performance profile appropriate for the environment's hardware and topology.
 
 . Add the `reserved` and `isolated` parameters with the CPUs you want reserved and isolated for the infra and application containers:
 +
@@ -34,5 +72,4 @@ spec:
 ----
 <1> Specify which CPUs are for infra containers to perform cluster and operating system housekeeping duties.
 <2> Specify which CPUs are for application containers to run workloads. 
-<3> Optional: Specify a node selector to apply the performance profile to specific nodes.
-
+<3> Optional: Specify a node selector to apply the performance profile to specific nodes.

scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.adoc

@@ -35,6 +35,12 @@ include::modules/cnf-allocating-multiple-huge-page-sizes.adoc[leveloffset=+2]
 
 include::modules/cnf-cpu-infra-container.adoc[leveloffset=+2]
 
+.Additional resources
+
+* xref:../scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.adoc#managing-device-interrupt-processing-for-guaranteed-pod-isolated-cpus_{context}[Managing device interrupt processing for guaranteed pod isolated CPUs]
+
+* link:https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed[Create a pod that gets assigned a QoS class of Guaranteed]
+
 include::modules/cnf-reducing-netqueues-using-pao.adoc[leveloffset=+1]
 
 include::modules/cnf-adjusting-nic-queues-with-the-performance-profile.adoc[leveloffset=+2]