Improve "Reserve Compute Resources for System Daemons" doc (#45771)

* Improve "Reserve Compute Resources for System Daemons" doc

Remove deprecated CLI flags and replace by KubeletConfiguration settings

* Apply suggestions from code review

Co-authored-by: Qiming Teng <[email protected]>

* keep the heading

---------

Co-authored-by: Qiming Teng <[email protected]>

Author: sathieu

content/en/docs/tasks/administer-cluster/reserve-compute-resources.md

@@ -5,7 +5,6 @@ reviewers:
 - dashpole
 title: Reserve Compute Resources for System Daemons
 content_type: task
-min-kubernetes-server-version: 1.8
 weight: 290
 ---
 
@@ -25,10 +24,10 @@ on each node.
 
 ## {{% heading "prerequisites" %}}
 
-{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
-Your Kubernetes server must be at or later than version 1.17 to use
-the kubelet command line option `--reserved-cpus` to set an
-[explicitly reserved CPU list](#explicitly-reserved-cpu-list).
+{{< include "task-tutorial-prereqs.md" >}}
+
+You can configure below kubelet [configuration settings](/docs/reference/config-api/kubelet-config.v1beta1/)
+using the [kubelet configuration file](/docs/tasks/administer-cluster/kubelet-config-file/).
 
 <!-- steps -->
 
@@ -48,15 +47,14 @@ Resources can be reserved for two categories of system daemons in the `kubelet`.
 ### Enabling QoS and Pod level cgroups
 
 To properly enforce node allocatable constraints on the node, you must
-enable the new cgroup hierarchy via the `--cgroups-per-qos` flag. This flag is
+enable the new cgroup hierarchy via the `cgroupsPerQOS` setting. This setting is
 enabled by default. When enabled, the `kubelet` will parent all end-user pods
 under a cgroup hierarchy managed by the `kubelet`.
 
 ### Configuring a cgroup driver
 
 The `kubelet` supports manipulation of the cgroup hierarchy on
-the host using a cgroup driver. The driver is configured via the
-`--cgroup-driver` flag.
+the host using a cgroup driver. The driver is configured via the `cgroupDriver` setting.
 
 The supported values are the following:
 
@@ -73,41 +71,41 @@ be configured to use the `systemd` cgroup driver.
 
 ### Kube Reserved
 
-- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi][,][pid=1000]`
-- **Kubelet Flag**: `--kube-reserved-cgroup=`
+- **KubeletConfiguration Setting**: `kubeReserved: {}`. Example value `{cpu: 100m, memory: 100Mi, ephemeral-storage: 1Gi, pid=1000}`
+- **KubeletConfiguration Setting**: `kubeReservedCgroup: ""`
 
-`kube-reserved` is meant to capture resource reservation for kubernetes system
-daemons like the `kubelet`, `container runtime`, `node problem detector`, etc.
+`kubeReserved` is meant to capture resource reservation for kubernetes system
+daemons like the `kubelet`, `container runtime`, etc.
 It is not meant to reserve resources for system daemons that are run as pods.
-`kube-reserved` is typically a function of `pod density` on the nodes.
+`kubeReserved` is typically a function of `pod density` on the nodes.
 
 In addition to `cpu`, `memory`, and `ephemeral-storage`, `pid` may be
 specified to reserve the specified number of process IDs for
 kubernetes system daemons.
 
-To optionally enforce `kube-reserved` on kubernetes system daemons, specify the parent
-control group for kube daemons as the value for `--kube-reserved-cgroup` kubelet
-flag.
+To optionally enforce `kubeReserved` on kubernetes system daemons, specify the parent
+control group for kube daemons as the value for `kubeReservedCgroup` setting,
+and [add `kube-reserved` to `enforceNodeAllocatable`](#enforcing-node-allocatable).
 
 It is recommended that the kubernetes system daemons are placed under a top
 level control group (`runtime.slice` on systemd machines for example). Each
 system daemon should ideally run within its own child control group. Refer to
 [the design proposal](https://git.k8s.io/design-proposals-archive/node/node-allocatable.md#recommended-cgroups-setup)
 for more details on recommended control group hierarchy.
 
-Note that Kubelet **does not** create `--kube-reserved-cgroup` if it doesn't
+Note that Kubelet **does not** create `kubeReservedCgroup` if it doesn't
 exist. The kubelet will fail to start if an invalid cgroup is specified. With `systemd`
 cgroup driver, you should follow a specific pattern for the name of the cgroup you
-define: the name should be the value you set for `--kube-reserved-cgroup`,
+define: the name should be the value you set for `kubeReservedCgroup`,
 with `.slice` appended.
 
 ### System Reserved
 
-- **Kubelet Flag**: `--system-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi][,][pid=1000]`
-- **Kubelet Flag**: `--system-reserved-cgroup=`
+- **KubeletConfiguration Setting**: `systemReserved: {}`. Example value `{cpu: 100m, memory: 100Mi, ephemeral-storage: 1Gi, pid=1000}`
+- **KubeletConfiguration Setting**: `systemReservedCgroup: ""`
 
-`system-reserved` is meant to capture resource reservation for OS system daemons
-like `sshd`, `udev`, etc. `system-reserved` should reserve `memory` for the
+`systemReserved` is meant to capture resource reservation for OS system daemons
+like `sshd`, `udev`, etc. `systemReserved` should reserve `memory` for the
 `kernel` too since `kernel` memory is not accounted to pods in Kubernetes at this time.
 Reserving resources for user login sessions is also recommended (`user.slice` in
 systemd world).
@@ -116,33 +114,32 @@ In addition to `cpu`, `memory`, and `ephemeral-storage`, `pid` may be
 specified to reserve the specified number of process IDs for OS system
 daemons.
 
-To optionally enforce `system-reserved` on system daemons, specify the parent
-control group for OS system daemons as the value for `--system-reserved-cgroup`
-kubelet flag.
+To optionally enforce `systemReserved` on system daemons, specify the parent
+control group for OS system daemons as the value for `systemReservedCgroup` setting,
+and [add `system-reserved` to `enforceNodeAllocatable`](#enforcing-node-allocatable).
 
 It is recommended that the OS system daemons are placed under a top level
 control group (`system.slice` on systemd machines for example).
 
-Note that `kubelet` **does not** create `--system-reserved-cgroup` if it doesn't
+Note that `kubelet` **does not** create `systemReservedCgroup` if it doesn't
 exist. `kubelet` will fail if an invalid cgroup is specified.  With `systemd`
 cgroup driver, you should follow a specific pattern for the name of the cgroup you
-define: the name should be the value you set for `--system-reserved-cgroup`,
+define: the name should be the value you set for `systemReservedCgroup`,
 with `.slice` appended.
 
 ### Explicitly Reserved CPU List
 
 {{< feature-state for_k8s_version="v1.17" state="stable" >}}
 
-**Kubelet Flag**: `--reserved-cpus=0-3`
-**KubeletConfiguration Flag**: `reservedSystemCPUs: 0-3`
+**KubeletConfiguration Setting**: `reservedSystemCPUs:`. Example value `0-3`
 
-`reserved-cpus` is meant to define an explicit CPU set for OS system daemons and
-kubernetes system daemons. `reserved-cpus` is for systems that do not intend to
+`reservedSystemCPUs` is meant to define an explicit CPU set for OS system daemons and
+kubernetes system daemons. `reservedSystemCPUs` is for systems that do not intend to
 define separate top level cgroups for OS system daemons and kubernetes system daemons
 with regard to cpuset resource.
-If the Kubelet **does not** have `--system-reserved-cgroup` and `--kube-reserved-cgroup`,
-the explicit cpuset provided by `reserved-cpus` will take precedence over the CPUs
-defined by `--kube-reserved` and `--system-reserved` options.
+If the Kubelet **does not** have `kubeReservedCgroup` and `systemReservedCgroup`,
+the explicit cpuset provided by `reservedSystemCPUs` will take precedence over the CPUs
+defined by `kubeReservedCgroup` and `systemReservedCgroup` options.
 
 This option is specifically designed for Telco/NFV use cases where uncontrolled
 interrupts/timers may impact the workload performance. you can use this option
@@ -155,23 +152,23 @@ For example: in Centos, you can do this using the tuned toolset.
 
 ### Eviction Thresholds
 
-**Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]`
+**KubeletConfiguration Setting**: `evictionHard: {memory.available: "100Mi", nodefs.available: "10%", nodefs.inodesFree: "5%", imagefs.available: "15%"}`. Example value: `{memory.available: "<500Mi"}`
 
 Memory pressure at the node level leads to System OOMs which affects the entire
 node and all pods running on it. Nodes can go offline temporarily until memory
 has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet
 provides [out of resource](/docs/concepts/scheduling-eviction/node-pressure-eviction/)
 management. Evictions are
 supported for `memory` and `ephemeral-storage` only. By reserving some memory via
-`--eviction-hard` flag, the `kubelet` attempts to evict pods whenever memory
+`evictionHard` setting, the `kubelet` attempts to evict pods whenever memory
 availability on the node drops below the reserved value. Hypothetically, if
 system daemons did not exist on a node, pods cannot use more than `capacity -
 eviction-hard`. For this reason, resources reserved for evictions are not
 available for pods.
 
 ### Enforcing Node Allocatable
 
-**Kubelet Flag**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]`
+**KubeletConfiguration setting**: `enforceNodeAllocatable: [pods]`. Example value: `[pods,system-reserved,kube-reserved]`
 
 The scheduler treats 'Allocatable' as the available `capacity` for pods.
 
@@ -180,35 +177,35 @@ by evicting pods whenever the overall usage across all pods exceeds
 'Allocatable'. More details on eviction policy can be found
 on the [node pressure eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/)
 page. This enforcement is controlled by
-specifying `pods` value to the kubelet flag `--enforce-node-allocatable`.
+specifying `pods` value to the KubeletConfiguration setting `enforceNodeAllocatable`.
 
-Optionally, `kubelet` can be made to enforce `kube-reserved` and
-`system-reserved` by specifying `kube-reserved` & `system-reserved` values in
-the same flag. Note that to enforce `kube-reserved` or `system-reserved`,
-`--kube-reserved-cgroup` or `--system-reserved-cgroup` needs to be specified
+Optionally, `kubelet` can be made to enforce `kubeReserved` and
+`systemReserved` by specifying `kube-reserved` & `system-reserved` values in
+the same setting. Note that to enforce `kubeReserved` or `systemReserved`,
+`kubeReservedCgroup` or `systemReservedCgroup` needs to be specified
 respectively.
 
 ## General Guidelines
 
-System daemons are expected to be treated similar to 
-[Guaranteed pods](/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed). 
+System daemons are expected to be treated similar to
+[Guaranteed pods](/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed).
 System daemons can burst within their bounding control groups and this behavior needs
 to be managed as part of kubernetes deployments. For example, `kubelet` should
-have its own control group and share `kube-reserved` resources with the
+have its own control group and share `kubeReserved` resources with the
 container runtime. However, Kubelet cannot burst and use up all available Node
-resources if `kube-reserved` is enforced.
+resources if `kubeReserved` is enforced.
 
-Be extra careful while enforcing `system-reserved` reservation since it can lead
+Be extra careful while enforcing `systemReserved` reservation since it can lead
 to critical system services being CPU starved, OOM killed, or unable
 to fork on the node. The
-recommendation is to enforce `system-reserved` only if a user has profiled their
+recommendation is to enforce `systemReserved` only if a user has profiled their
 nodes exhaustively to come up with precise estimates and is confident in their
 ability to recover if any process in that group is oom-killed.
 
 * To begin with enforce 'Allocatable' on `pods`.
 * Once adequate monitoring and alerting is in place to track kube system
-  daemons, attempt to enforce `kube-reserved` based on usage heuristics.
-* If absolutely necessary, enforce `system-reserved` over time.
+  daemons, attempt to enforce `kubeReserved` based on usage heuristics.
+* If absolutely necessary, enforce `systemReserved` over time.
 
 The resource requirements of kube system daemons may grow over time as more and
 more features are added. Over time, kubernetes project will attempt to bring
@@ -222,9 +219,9 @@ So expect a drop in `Allocatable` capacity in future releases.
 Here is an example to illustrate Node Allocatable computation:
 
 * Node has `32Gi` of `memory`, `16 CPUs` and `100Gi` of `Storage`
-* `--kube-reserved` is set to `cpu=1,memory=2Gi,ephemeral-storage=1Gi`
-* `--system-reserved` is set to `cpu=500m,memory=1Gi,ephemeral-storage=1Gi`
-* `--eviction-hard` is set to `memory.available<500Mi,nodefs.available<10%`
+* `kubeReserved` is set to `{cpu: 1000m, memory: 2Gi, ephemeral-storage: 1Gi}`
+* `systemReserved` is set to `{cpu: 500m, memory: 1Gi, ephemeral-storage: 1Gi}`
+* `evictionHard` is set to `{memory.available: "<500Mi", nodefs.available: "<10%"}`
 
 Under this scenario, 'Allocatable' will be 14.5 CPUs, 28.5Gi of memory and
 `88Gi` of local storage.
@@ -234,7 +231,7 @@ Kubelet evicts pods whenever the overall memory usage across pods exceeds 28.5Gi
 or if overall disk usage exceeds 88Gi. If all processes on the node consume as
 much CPU as they can, pods together cannot consume more than 14.5 CPUs.
 
-If `kube-reserved` and/or `system-reserved` is not enforced and system daemons
+If `kubeReserved` and/or `systemReserved` is not enforced and system daemons
 exceed their reservation, `kubelet` evicts pods whenever the overall node memory
 usage is higher than 31.5Gi or `storage` is greater than 90Gi.