Skip to content

Commit 05afa58

Browse files
authored
Merge pull request #48458 from windsonsea/polom
Clean up topology-manager.md
2 parents f962ee6 + 3eecc6e commit 05afa58

File tree

1 file changed

+50
-51
lines changed

1 file changed

+50
-51
lines changed

content/en/docs/tasks/administer-cluster/topology-manager.md

Lines changed: 50 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,11 @@
11
---
22
title: Control Topology Management Policies on a node
3-
43
reviewers:
54
- ConnorDoyle
65
- klueska
76
- lmdaly
87
- nolancon
98
- bg-chun
10-
119
content_type: task
1210
min-kubernetes-server-version: v1.18
1311
weight: 150
@@ -26,7 +24,7 @@ In order to extract the best performance, optimizations related to CPU isolation
2624
device locality are required. However, in Kubernetes, these optimizations are handled by a
2725
disjoint set of components.
2826

29-
_Topology Manager_ is a Kubelet component that aims to coordinate the set of components that are
27+
_Topology Manager_ is a kubelet component that aims to coordinate the set of components that are
3028
responsible for these optimizations.
3129

3230
## {{% heading "prerequisites" %}}
@@ -38,24 +36,24 @@ responsible for these optimizations.
3836
## How topology manager works
3937

4038
Prior to the introduction of Topology Manager, the CPU and Device Manager in Kubernetes make
41-
resource allocation decisions independently of each other. This can result in undesirable
42-
allocations on multiple-socketed systems, performance/latency sensitive applications will suffer
43-
due to these undesirable allocations. Undesirable in this case meaning for example, CPUs and
44-
devices being allocated from different NUMA Nodes thus, incurring additional latency.
39+
resource allocation decisions independently of each other. This can result in undesirable
40+
allocations on multiple-socketed systems, and performance/latency sensitive applications will suffer
41+
due to these undesirable allocations. Undesirable in this case meaning, for example, CPUs and
42+
devices being allocated from different NUMA Nodes, thus incurring additional latency.
4543

46-
The Topology Manager is a Kubelet component, which acts as a source of truth so that other Kubelet
44+
The Topology Manager is a kubelet component, which acts as a source of truth so that other kubelet
4745
components can make topology aligned resource allocation choices.
4846

4947
The Topology Manager provides an interface for components, called *Hint Providers*, to send and
50-
receive topology information. Topology Manager has a set of node level policies which are
48+
receive topology information. The Topology Manager has a set of node level policies which are
5149
explained below.
5250

53-
The Topology manager receives Topology information from the *Hint Providers* as a bitmask denoting
51+
The Topology Manager receives topology information from the *Hint Providers* as a bitmask denoting
5452
NUMA Nodes available and a preferred allocation indication. The Topology Manager policies perform
5553
a set of operations on the hints provided and converge on the hint determined by the policy to
56-
give the optimal result, if an undesirable hint is stored the preferred field for the hint will be
54+
give the optimal result. If an undesirable hint is stored, the preferred field for the hint will be
5755
set to false. In the current policies preferred is the narrowest preferred mask.
58-
The selected hint is stored as part of the Topology Manager. Depending on the policy configured
56+
The selected hint is stored as part of the Topology Manager. Depending on the policy configured,
5957
the pod can be accepted or rejected from the node based on the selected hint.
6058
The hint is then stored in the Topology Manager for use by the *Hint Providers* when making the
6159
resource allocation decisions.
@@ -64,28 +62,28 @@ resource allocation decisions.
6462

6563
The Topology Manager currently:
6664

67-
- Aligns Pods of all QoS classes.
68-
- Aligns the requested resources that Hint Provider provides topology hints for.
65+
- aligns Pods of all QoS classes.
66+
- aligns the requested resources that Hint Provider provides topology hints for.
6967

7068
If these conditions are met, the Topology Manager will align the requested resources.
7169

72-
In order to customise how this alignment is carried out, the Topology Manager provides two
73-
distinct knobs: `scope` and `policy`.
70+
In order to customize how this alignment is carried out, the Topology Manager provides two
71+
distinct options: `scope` and `policy`.
7472

75-
The `scope` defines the granularity at which you would like resource alignment to be performed
76-
(e.g. at the `pod` or `container` level). And the `policy` defines the actual strategy used to
77-
carry out the alignment (e.g. `best-effort`, `restricted`, `single-numa-node`, etc.).
73+
The `scope` defines the granularity at which you would like resource alignment to be performed,
74+
for example, at the `pod` or `container` level. And the `policy` defines the actual policy used to
75+
carry out the alignment, for example, `best-effort`, `restricted`, and `single-numa-node`.
7876
Details on the various `scopes` and `policies` available today can be found below.
7977

8078
{{< note >}}
8179
To align CPU resources with other requested resources in a Pod spec, the CPU Manager should be
8280
enabled and proper CPU Manager policy should be configured on a Node.
83-
See [control CPU Management Policies](/docs/tasks/administer-cluster/cpu-management-policies/).
81+
See [Control CPU Management Policies on the Node](/docs/tasks/administer-cluster/cpu-management-policies/).
8482
{{< /note >}}
8583

8684
{{< note >}}
8785
To align memory (and hugepages) resources with other requested resources in a Pod spec, the Memory
88-
Manager should be enabled and proper Memory Manager policy should be configured on a Node. Examine
86+
Manager should be enabled and proper Memory Manager policy should be configured on a Node. Refer to
8987
[Memory Manager](/docs/tasks/administer-cluster/memory-manager/) documentation.
9088
{{< /note >}}
9189

@@ -116,7 +114,8 @@ scope, for example the `pod` scope.
116114

117115
### `pod` scope
118116

119-
To select the `pod` scope, set `topologyManagerScope` in the [kubelet configuration file](/docs/tasks/administer-cluster/kubelet-config-file/) to `pod`.`
117+
To select the `pod` scope, set `topologyManagerScope` in the
118+
[kubelet configuration file](/docs/tasks/administer-cluster/kubelet-config-file/) to `pod`.
120119

121120
This scope allows for grouping all containers in a pod to a common set of NUMA nodes. That is, the
122121
Topology Manager treats a pod as a whole and attempts to allocate the entire pod (all containers)
@@ -127,8 +126,8 @@ alignments produced by the Topology Manager on different occasions:
127126
* all containers can be and are allocated to a shared set of NUMA nodes.
128127

129128
The total amount of particular resource demanded for the entire pod is calculated according to
130-
[effective requests/limits](/docs/concepts/workloads/pods/init-containers/#resources) formula, and
131-
thus, this total value is equal to the maximum of:
129+
[effective requests/limits](/docs/concepts/workloads/pods/init-containers/#resource-sharing-within-containers)
130+
formula, and thus, this total value is equal to the maximum of:
132131

133132
* the sum of all app container requests,
134133
* the maximum of init container requests,
@@ -147,12 +146,12 @@ is present among possible allocations. Reconsider the example above:
147146
* whereas a set containing more NUMA nodes - it results in pod rejection (because instead of one
148147
NUMA node, two or more NUMA nodes are required to satisfy the allocation).
149148

150-
To recap, Topology Manager first computes a set of NUMA nodes and then tests it against Topology
149+
To recap, the Topology Manager first computes a set of NUMA nodes and then tests it against the Topology
151150
Manager policy, which either leads to the rejection or admission of the pod.
152151

153152
## Topology manager policies
154153

155-
Topology Manager supports four allocation policies. You can set a policy via a Kubelet flag,
154+
The Topology Manager supports four allocation policies. You can set a policy via a kubelet flag,
156155
`--topology-manager-policy`. There are four supported policies:
157156

158157
* `none` (default)
@@ -161,7 +160,7 @@ Topology Manager supports four allocation policies. You can set a policy via a K
161160
* `single-numa-node`
162161

163162
{{< note >}}
164-
If Topology Manager is configured with the **pod** scope, the container, which is considered by
163+
If the Topology Manager is configured with the **pod** scope, the container, which is considered by
165164
the policy, is reflecting requirements of the entire pod, and thus each container from the pod
166165
will result with **the same** topology alignment decision.
167166
{{< /note >}}
@@ -175,21 +174,21 @@ This is the default policy and does not perform any topology alignment.
175174
For each container in a Pod, the kubelet, with `best-effort` topology management policy, calls
176175
each Hint Provider to discover their resource availability. Using this information, the Topology
177176
Manager stores the preferred NUMA Node affinity for that container. If the affinity is not
178-
preferred, Topology Manager will store this and admit the pod to the node anyway.
177+
preferred, the Topology Manager will store this and admit the pod to the node anyway.
179178

180179
The *Hint Providers* can then use this information when making the
181180
resource allocation decision.
182181

183182
### `restricted` policy {#policy-restricted}
184183

185184
For each container in a Pod, the kubelet, with `restricted` topology management policy, calls each
186-
Hint Provider to discover their resource availability. Using this information, the Topology
185+
Hint Provider to discover their resource availability. Using this information, the Topology
187186
Manager stores the preferred NUMA Node affinity for that container. If the affinity is not
188-
preferred, Topology Manager will reject this pod from the node. This will result in a pod in a
187+
preferred, the Topology Manager will reject this pod from the node. This will result in a pod entering a
189188
`Terminated` state with a pod admission failure.
190189

191190
Once the pod is in a `Terminated` state, the Kubernetes scheduler will **not** attempt to
192-
reschedule the pod. It is recommended to use a ReplicaSet or Deployment to trigger a redeploy of
191+
reschedule the pod. It is recommended to use a ReplicaSet or Deployment to trigger a redeployment of
193192
the pod. An external control loop could be also implemented to trigger a redeployment of pods that
194193
have the `Topology Affinity` error.
195194

@@ -199,16 +198,16 @@ resource allocation decision.
199198
### `single-numa-node` policy {#policy-single-numa-node}
200199

201200
For each container in a Pod, the kubelet, with `single-numa-node` topology management policy,
202-
calls each Hint Provider to discover their resource availability. Using this information, the
203-
Topology Manager determines if a single NUMA Node affinity is possible. If it is, Topology
201+
calls each Hint Provider to discover their resource availability. Using this information, the
202+
Topology Manager determines if a single NUMA Node affinity is possible. If it is, Topology
204203
Manager will store this and the *Hint Providers* can then use this information when making the
205-
resource allocation decision. If, however, this is not possible then the Topology Manager will
204+
resource allocation decision. If, however, this is not possible then the Topology Manager will
206205
reject the pod from the node. This will result in a pod in a `Terminated` state with a pod
207206
admission failure.
208207

209208
Once the pod is in a `Terminated` state, the Kubernetes scheduler will **not** attempt to
210-
reschedule the pod. It is recommended to use a Deployment with replicas to trigger a redeploy of
211-
the Pod.An external control loop could be also implemented to trigger a redeployment of pods
209+
reschedule the pod. It is recommended to use a Deployment with replicas to trigger a redeployment of
210+
the Pod. An external control loop could be also implemented to trigger a redeployment of pods
212211
that have the `Topology Affinity` error.
213212

214213
## Topology manager policy options
@@ -218,6 +217,7 @@ Support for the Topology Manager policy options requires `TopologyManagerPolicyO
218217
(it is enabled by default).
219218

220219
You can toggle groups of options on and off based upon their maturity level using the following feature gates:
220+
221221
* `TopologyManagerPolicyBetaOptions` default enabled. Enable to show beta-level options.
222222
* `TopologyManagerPolicyAlphaOptions` default disabled. Enable to show alpha-level options.
223223

@@ -230,42 +230,42 @@ this policy option is visible by default provided that the `TopologyManagerPolic
230230
`TopologyManagerPolicyBetaOptions` [feature gates](/docs/reference/command-line-tools-reference/feature-gates/)
231231
are enabled.
232232

233-
The topology manager is not aware by default of NUMA distances, and does not take them into account when making
233+
The Topology Manager is not aware by default of NUMA distances, and does not take them into account when making
234234
Pod admission decisions. This limitation surfaces in multi-socket, as well as single-socket multi NUMA systems,
235235
and can cause significant performance degradation in latency-critical execution and high-throughput applications
236-
if the topology manager decides to align resources on non-adjacent NUMA nodes.
236+
if the Topology Manager decides to align resources on non-adjacent NUMA nodes.
237237

238238
If you specify the `prefer-closest-numa-nodes` policy option, the `best-effort` and `restricted`
239239
policies favor sets of NUMA nodes with shorter distance between them when making admission decisions.
240240

241241
You can enable this option by adding `prefer-closest-numa-nodes=true` to the Topology Manager policy options.
242242

243-
By default (without this option), Topology Manager aligns resources on either a single NUMA node or,
243+
By default (without this option), the Topology Manager aligns resources on either a single NUMA node or,
244244
in the case where more than one NUMA node is required, using the minimum number of NUMA nodes.
245245

246246
### `max-allowable-numa-nodes` (beta) {#policy-option-max-allowable-numa-nodes}
247247

248-
The `max-allowable-numa-nodes` option is beta since Kubernetes 1.31. In Kubernetes {{< skew currentVersion >}}
248+
The `max-allowable-numa-nodes` option is beta since Kubernetes 1.31. In Kubernetes {{< skew currentVersion >}},
249249
this policy option is visible by default provided that the `TopologyManagerPolicyOptions` and
250250
`TopologyManagerPolicyBetaOptions` [feature gates](/docs/reference/command-line-tools-reference/feature-gates/)
251251
are enabled.
252252

253253
The time to admit a pod is tied to the number of NUMA nodes on the physical machine.
254-
By default, Kubernetes does not run a kubelet with the topology manager enabled, on any (Kubernetes) node where
254+
By default, Kubernetes does not run a kubelet with the Topology Manager enabled, on any (Kubernetes) node where
255255
more than 8 NUMA nodes are detected.
256256

257257
{{< note >}}
258-
If you select the the `max-allowable-numa-nodes` policy option, nodes with more than 8 NUMA nodes can
259-
be allowed to run with the topology manager enabled. The Kubernetes project only has limited data on the impact
260-
of using the topology manager on (Kubernetes) nodes with more than 8 NUMA nodes. Because of that
258+
If you select the `max-allowable-numa-nodes` policy option, nodes with more than 8 NUMA nodes can
259+
be allowed to run with the Topology Manager enabled. The Kubernetes project only has limited data on the impact
260+
of using the Topology Manager on (Kubernetes) nodes with more than 8 NUMA nodes. Because of that
261261
lack of data, using this policy option with Kubernetes {{< skew currentVersion >}} is **not** recommended and is
262262
at your own risk.
263263
{{< /note >}}
264264

265265
You can enable this option by adding `max-allowable-numa-nodes=true` to the Topology Manager policy options.
266266

267267
Setting a value of `max-allowable-numa-nodes` does not (in and of itself) affect the
268-
latency of pod admission, but binding a Pod to a (Kubernetes) node with many NUMA does does have an impact.
268+
latency of pod admission, but binding a Pod to a (Kubernetes) node with many NUMA does have an impact.
269269
Future, potential improvements to Kubernetes may improve Pod admission performance and the high
270270
latency that happens as the number of NUMA nodes increases.
271271

@@ -296,10 +296,10 @@ spec:
296296

297297
This pod runs in the `Burstable` QoS class because requests are less than limits.
298298

299-
If the selected policy is anything other than `none`, Topology Manager would consider these Pod
299+
If the selected policy is anything other than `none`, the Topology Manager would consider these Pod
300300
specifications. The Topology Manager would consult the Hint Providers to get topology hints.
301301
In the case of the `static`, the CPU Manager policy would return default topology hint, because
302-
these Pods do not have explicitly request CPU resources.
302+
these Pods do not explicitly request CPU resources.
303303

304304
```yaml
305305
spec:
@@ -320,7 +320,6 @@ spec:
320320
This pod with integer CPU request runs in the `Guaranteed` QoS class because `requests` are equal
321321
to `limits`.
322322

323-
324323
```yaml
325324
spec:
326325
containers:
@@ -380,10 +379,10 @@ assignments.
380379

381380
## Known limitations
382381

383-
1. The maximum number of NUMA nodes that Topology Manager allows is 8. With more than 8 NUMA nodes
382+
1. The maximum number of NUMA nodes that Topology Manager allows is 8. With more than 8 NUMA nodes,
384383
there will be a state explosion when trying to enumerate the possible NUMA affinities and
385384
generating their hints. See [`max-allowable-numa-nodes`](#policy-option-max-allowable-numa-nodes)
386385
(beta) for more options.
387386

388-
2. The scheduler is not topology-aware, so it is possible to be scheduled on a node and then fail
389-
on the node due to the Topology Manager.
387+
1. The scheduler is not topology-aware, so it is possible to be scheduled on a node and then fail
388+
on the node due to the Topology Manager.

0 commit comments

Comments
 (0)