Skip to content

Commit f217a54

Browse files
marossetmarosset
committed
Moving windows resource management to seperate page
Signed-off-by: Mark Rossetti <[email protected]>
1 parent aef1728 commit f217a54

File tree

2 files changed

+84
-71
lines changed

2 files changed

+84
-71
lines changed

‎content/en/docs/concepts/configuration/windows-resource-management.md

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
---
2+
reviewers:
3+
- jayunit100
4+
- jsturtevant
5+
- marosset
6+
- perithompson
7+
title: Resource Management for Windows nodes
8+
content_type: concept
9+
weight: 75
10+
---
11+
12+
<!-- overview -->
13+
14+
This page outlines the differences in how resources are managed between Linux and Windows.
15+
16+
<!-- body -->
17+
18+
On Linux nodes, {{< glossary_tooltip text="cgroups" term_id="cgroup" >}} are used
19+
as a pod boundary for resource control. Containers are created within that boundary
20+
for network, process and file system isolation. The Linux cgroup APIs can be used to
21+
gather CPU, I/O, and memory use statistics.
22+
23+
In contrast, Windows uses a [_job object_](https://docs.microsoft.com/windows/win32/procthread/job-objects) per container with a system namespace filter
24+
to contain all processes in a container and provide logical isolation from the
25+
host.
26+
(Job objects are a Windows process isolation mechanism and are different from
27+
what Kubernetes refers to as a {{< glossary_tooltip term_id="job" text="Job" >}}).
28+
29+
There is no way to run a Windows container without the namespace filtering in
30+
place. This means that system privileges cannot be asserted in the context of the
31+
host, and thus privileged containers are not available on Windows.
32+
Containers cannot assume an identity from the host because the Security Account Manager
33+
(SAM) is separate.
34+
35+
## Memory reservations {#resource-management-memory}
36+
37+
Windows does not have an out-of-memory process killer as Linux does. Windows always
38+
treats all user-mode memory allocations as virtual, and pagefiles are mandatory.
39+
40+
Windows nodes do not overcommit memory for processes running in containers. The
41+
net effect is that Windows won't reach out of memory conditions the same way Linux
42+
does, and processes page to disk instead of being subject to out of memory (OOM)
43+
termination. If memory is over-provisioned and all physical memory is exhausted,
44+
then paging can slow down performance.
45+
46+
You can place bounds on memory use for workloads using the kubelet
47+
parameters `--kubelet-reserve` and/or `--system-reserve`; these account
48+
for memory usage on the node (outside of containers), and reduce
49+
[NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable).
50+
As you deploy workloads, set resource limits on containers. This also subtracts from
51+
`NodeAllocatable` and prevents the scheduler from adding more pods once a node is full.
52+
53+
{{< note >}}
54+
When you set memory resource limits for Windows containers, you should either set a
55+
limit and leave the memory request unspecified, or set the request equal to the limit.
56+
{{< /note >}}
57+
58+
On Windows, good practice to avoid over-provisioning is to configure the kubelet
59+
with a system reserved memory of at least 2GiB to account for Windows, Kubernetes
60+
and container runtime overheads.
61+
62+
## CPU reservations {#resource-management-cpu}
63+
64+
To account for CPU use by the operating system, the container runtime, and by
65+
Kubernetes host processes such as the kubelet, you can (and should) reserve a
66+
percentage of total CPU. You should determine this CPU reservation taking account of
67+
to the number of CPU cores available on the node. To decide on the CPU percentage to
68+
reserve, identify the maximum pod density for each node and monitor the CPU usage of
69+
the system services running there, then choose a value that meets your workload needs.
70+
71+
You can place bounds on CPU usage for workloads using the
72+
kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to
73+
account for CPU usage on the node (outside of containers).
74+
This reduces `NodeAllocatable`.
75+
The cluster-wide scheduler then takes this reservation into account when determining
76+
pod placement.
77+
78+
On Windows, the kubelet supports a command-line flag to set the priority of the
79+
kubelet process: `--windows-priorityclass`. This flag allows the kubelet process to get
80+
more CPU time slices when compared to other processes running on the Windows host.
81+
More information on the allowable values and their meaning is available at
82+
[Windows Priority Classes](https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities#priority-class).
83+
To ensure that running Pods do not starve the kubelet of CPU cycles, set this flag to `ABOVE_NORMAL_PRIORITY_CLASS` or above.

‎content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md

Lines changed: 1 addition & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -42,76 +42,6 @@ This document uses the term *Windows containers* to mean Windows containers with
4242
process isolation. Kubernetes does not support running Windows containers with
4343
[Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container).
4444

45-
## Resource management
46-
47-
On Linux nodes, {{< glossary_tooltip text="cgroups" term_id="cgroup" >}} are used
48-
as a pod boundary for resource control. Containers are created within that boundary
49-
for network, process and file system isolation. The Linux cgroup APIs can be used
50-
to gather CPU, I/O, and memory use statistics.
51-
52-
In contrast, Windows uses a _job object_ per container with a system namespace filter
53-
to contain all processes in a container and provide logical isolation from the
54-
host.
55-
(Job objects are a Windows process isolation mechanism and are different from
56-
what Kubernetes refers to as a {{< glossary_tooltip term_id="job" text="Job" >}}).
57-
58-
There is no way to run a Windows container without the namespace filtering in
59-
place. This means that system privileges cannot be asserted in the context of the
60-
host, and thus privileged containers are not available on Windows.
61-
Containers cannot assume an identity from the host because the Security Account Manager
62-
(SAM) is separate.
63-
64-
#### Memory reservations {#resource-management-memory}
65-
66-
Windows does not have an out-of-memory process killer as Linux does. Windows always
67-
treats all user-mode memory allocations as virtual, and pagefiles are mandatory
68-
(on Linux, the kubelet will by default not start with swap space enabled).
69-
70-
Windows nodes do not overcommit memory for processes running in containers. The
71-
net effect is that Windows won't reach out of memory conditions the same way Linux
72-
does, and processes page to disk instead of being subject to out of memory (OOM)
73-
termination. If memory is over-provisioned and all physical memory is exhausted,
74-
then paging can slow down performance.
75-
76-
You can place bounds on memory use for workloads using the kubelet
77-
parameters `--kubelet-reserve` and/or `--system-reserve`; these account
78-
for memory usage on the node (outside of containers), and reduce
79-
[NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable).
80-
As you deploy workloads, set resource limits on containers. This also subtracts from
81-
`NodeAllocatable` and prevents the scheduler from adding more pods once a node is full.
82-
83-
{{< note >}}
84-
When you set memory resource limits for Windows containers, you should either set a
85-
limit and leave the memory request unspecified, or set the request equal to the limit.
86-
{{< /note >}}
87-
88-
On Windows, good practice to avoid over-provisioning is to configure the kubelet
89-
with a system reserved memory of at least 2GiB to account for Windows, Kubernetes
90-
and container runtime overheads.
91-
92-
#### CPU reservations {#resource-management-cpu}
93-
94-
To account for CPU use by the operating system, the container runtime, and by
95-
Kubernetes host processes such as the kubelet, you can (and should) reserve a
96-
percentage of total CPU. You should determine this CPU reservation taking account of
97-
to the number of CPU cores available on the node. To decide on the CPU percentage to
98-
reserve, identify the maximum pod density for each node and monitor the CPU usage of
99-
the system services running there, then choose a value that meets your workload needs.
100-
101-
You can place bounds on CPU usage for workloads using the
102-
kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to
103-
account for CPU usage on the node (outside of containers).
104-
This reduces `NodeAllocatable`.
105-
The cluster-wide scheduler then takes this reservation into account when determining
106-
pod placement.
107-
108-
On Windows, the kubelet supports a command-line flag to set the priority of the
109-
kubelet process: `--windows-priorityclass`. This flag allows the kubelet process to get
110-
more CPU time slices when compared to other processes running on the Windows host.
111-
More information on the allowable values and their meaning is available at
112-
[Windows Priority Classes](https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities#priority-class).
113-
To ensure that running Pods do not starve the kubelet of CPU cycles, set this flag to `ABOVE_NORMAL_PRIORITY_CLASS` or above.
114-
11545
## Compatibility and limitations {#limitations}
11646

11747
Some node features are only available if you use a specific
@@ -490,7 +420,7 @@ For more details, refer to the deployment guide of the CSI plugin you wish to de
490420

491421
The behavior of some kubelet command line options behave differently on Windows, as described below:
492422

493-
* The `--windows-priorityclass` lets you set the scheduling priority of the kubelet process (see [CPU resource management](#resource-management-cpu))
423+
* The `--windows-priorityclass` lets you set the scheduling priority of the kubelet process (see [CPU resource management](/docs/concepts/configuration/windows-resource-management/#resource-management-cpu))
494424
* The `--kubelet-reserve`, `--system-reserve` , and `--eviction-hard` flags update [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable)
495425
* Eviction by using `--enforce-node-allocable` is not implemented
496426
* Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented

0 commit comments

Comments
 (0)