Skip to content

Commit 87b1df4

Browse files
bscott-rhbscott-rh
authored
Merge pull request #69352 from bhardesty/osdocs-8615-rosa-kubeletconfig
OSDOCS-8615: Configuring PID limits
2 parents 174e023 + c24f6f7 commit 87b1df4

File tree

8 files changed

+177
-2
lines changed

8 files changed

+177
-2
lines changed

‎_topic_maps/_topic_map_rosa.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -543,6 +543,8 @@ Topics:
543543
File: rosa-nodes-about-autoscaling-nodes
544544
- Name: Configuring cluster memory to meet container memory and risk requirements
545545
File: nodes-cluster-resource-configure
546+
- Name: Configuring PID limits
547+
File: rosa-configuring-pid-limits
546548
---
547549
Name: Security and compliance
548550
Dir: security

‎modules/risks-setting-higher-process-id-limits.adoc

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * rosa_cluster_admin/rosa-configuring-pid-limits.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="risks-setting-higher-process-id-limits_{context}"]
7+
= Risks of setting higher process ID limits for {product-title} pods
8+
9+
The `podPidsLimit` parameter for a pod controls the maximum number of processes and threads that can run simultaneously in that pod.
10+
11+
You can increase the value for `podPidsLimit` from the default of 4,096 to a maximum of 16,384. Changing this value might incur downtime for applications, because changing the `podPidsLimit` requires rebooting the affected node.
12+
13+
If you are running a large number of pods per node, and you have a high `podPidsLimit` value on your nodes, you risk exceeding the PID maximum for the node.
14+
15+
To find the maximum number of pods that you can run simultaneously on a single node without exceeding the PID maximum for the node, divide 3,650,000 by your `podPidsLimit` value. For example, if your `podPidsLimit` value is 16,384, and you expect the pods to use close to that number of process IDs, you can safely run 222 pods on a single node.
16+
17+
[NOTE]
18+
====
19+
Memory, CPU, and available storage can also limit the maximum number of pods that can run simultaneously, even when the `podPidsLimit` value is set appropriately. For more information, see "Planning your environment" and "Limits and scalability".
20+
====

‎modules/sd-understanding-process-id-limits.adoc

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * rosa_cluster_admin/rosa-configuring-pid-limits.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="understanding-process-id-limits_{context}"]
7+
= Understanding process ID limits
8+
9+
In {product-title}, consider these two supported limits for process ID (PID) usage before you schedule work on your cluster:
10+
11+
* Maximum number of PIDs per pod.
12+
+
13+
The default value is 4,096 in {product-title} 4.11 and later. This value is controlled by the `podPidsLimit` parameter set on the node.
14+
15+
* Maximum number of PIDs per node.
16+
+
17+
The default value depends on link:https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/nodes/index#nodes-nodes-resources-configuring[node resources]. In {product-title}, this value is controlled by the link:https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#system-reserved[`--system-reserved`] parameter, which reserves PIDs on each node based on the total resources of the node.
18+
19+
When a pod exceeds the allowed maximum number of PIDs per pod, the pod might stop functioning correctly and might be evicted from the node. See link:https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#eviction-signals-and-thresholds[the Kubernetes documentation for eviction signals and thresholds] for more information.
20+
21+
When a node exceeds the allowed maximum number of PIDs per node, the node can become unstable because new processes cannot have PIDs assigned. If existing processes cannot complete without creating additional processes, the entire node can become unusable and require reboot. This situation can result in data loss, depending on the processes and applications being run. Customer administrators and Red Hat Site Reliability Engineering are notified when this threshold is reached, and a `Worker node is experiencing PIDPressure` warning will appear in the cluster logs.

‎modules/setting-higher-pid-limit-on-existing-cluster.adoc

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * rosa_cluster_admin/rosa-configuring-pid-limits.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="setting-higher-pid-limit-on-existing-cluster_{context}"]
7+
= Setting a higher PID limit on an existing {product-title} cluster
8+
9+
You can set a higher `podPidsLimit` on an existing {product-title} cluster by creating or editing a `KubeletConfig` object that changes the `--pod-pids-limit` parameter.
10+
11+
[IMPORTANT]
12+
====
13+
Changing the `podPidsLimit` on an existing cluster will trigger non-control plane nodes in the cluster to reboot one at a time. Make this change outside of peak usage hours for your cluster and avoid upgrading or hibernating your cluster until all nodes have rebooted.
14+
====
15+
16+
.Prerequisites
17+
18+
* You have installed the OpenShift CLI (`oc`).
19+
* You have logged in to your Red Hat account by using the ROSA CLI.
20+
21+
.Procedure
22+
23+
. Create or edit the `KubeletConfig` object to change the PID limit.
24+
+
25+
--
26+
** If this is the first time you are changing the default PID limit, create the `KubeletConfig` object and set the `--pod-pids-limit` value by running the following command:
27+
+
28+
[source,terminal]
29+
----
30+
$ rosa create kubeletconfig -c <cluster_name> --pod-pids-limit=<value>
31+
----
32+
+
33+
For example, the following command sets a maximum of 16,384 PIDs per pod for cluster `my-cluster`:
34+
+
35+
[source,terminal]
36+
----
37+
$ rosa create kubeletconfig -c my-cluster --pod-pids-limit=16384
38+
----
39+
40+
** If you previously created a `KubeletConfig` object, edit the existing `KubeletConfig` object and set the `--pod-pids-limit` value by running the following command:
41+
+
42+
[source,terminal]
43+
----
44+
$ rosa edit kubeletconfig -c <cluster_name> --pod-pids-limit=<value>
45+
----
46+
--
47+
+
48+
A cluster-wide rolling reboot of worker nodes is triggered.
49+
50+
. Verify that all of the worker nodes rebooted by running the following command:
51+
+
52+
[source,terminal]
53+
----
54+
$ oc get machineconfigpool
55+
----
56+
+
57+
.Example output
58+
[source,terminal]
59+
----
60+
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
61+
master rendered-master-06c9c4… True False False 3 3 3 0 4h42m
62+
worker rendered-worker-f4b64… True False False 4 4 4 0 4h42m
63+
----
64+
65+
.Verification
66+
67+
When each node in the cluster has rebooted, you can verify that the new setting is in place.
68+
69+
* Check the Pod Pids limit in the `KubeletConfig` object:
70+
+
71+
[source,terminal]
72+
----
73+
$ rosa describe kubeletconfig --cluster=<cluster_name>
74+
----
75+
+
76+
The new PIDs limit appears in the output, as shown in the following example:
77+
+
78+
.Example output
79+
[source,terminal]
80+
----
81+
Pod Pids Limit: 16384
82+
----

‎rosa_architecture/rosa_policy_service_definition/rosa-service-definition.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ include::snippets/pid-limits.adoc[]
2222
.Additional Resources
2323

2424
* xref:../../rosa_architecture/rosa_policy_service_definition/rosa-service-definition.adoc#rosa-sdpolicy-red-hat-operator_rosa-service-definition[Red Hat Operator Support]
25+
* xref:../../rosa_cluster_admin/rosa-configuring-pid-limits.adoc#rosa-configuring-pid-limits[Configuring PID limits]
2526

2627
include::modules/rosa-sdpolicy-am-aws-compute-types.adoc[leveloffset=+2]
2728

‎rosa_cluster_admin/rosa-configuring-pid-limits.adoc

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
// This assembly covers how to configure cluster-wide PID limits by configuring
2+
// the kubeletconfig. Eventually, additional cluster-wide parameters should be
3+
// available for configuration. As additional parameters are added, it might be
4+
// better to change this assembly to be "Configuring cluster parameters" or
5+
// something similar.
6+
//
7+
// Also, this content theoretically applies to OSD as well, but the PID limits
8+
// can only be configured through the ROSA CLI. When other methods are added for
9+
// configuring PID limits, this assembly and its modules can be evaluated for
10+
// inclusion in the OSD docs.
11+
12+
:_mod-docs-content-type: ASSEMBLY
13+
[id="rosa-configuring-pid-limits"]
14+
= Configuring PID limits
15+
include::_attributes/attributes-openshift-dedicated.adoc[]
16+
:context: rosa-configuring-pid-limits
17+
18+
toc::[]
19+
20+
A process identifier (PID) is a unique identifier assigned by the Linux kernel to each process or thread currently running on a system. The number of processes that can run simultaneously on a system is limited to 4,194,304 by the Linux kernel. This number might also be affected by limited access to other system resources such as memory, CPU, and disk space.
21+
22+
In {product-title} 4.11 and later, by default, a pod can have a maximum of 4,096 PIDs. If your workload requires more than that, you can increase the allowed maximum number of PIDs.
23+
24+
// Understanding process ID limits
25+
include::modules/sd-understanding-process-id-limits.adoc[leveloffset=+1]
26+
27+
// Risks of setting higher process ID limits
28+
include::modules/risks-setting-higher-process-id-limits.adoc[leveloffset=+1]
29+
30+
[role="_additional-resources"]
31+
.Additional resources
32+
33+
* xref:../rosa_architecture/rosa_policy_service_definition/rosa-service-definition.adoc#rosa-sdpolicy-instance-types_rosa-service-definition[Instance types]
34+
35+
* xref:../rosa_planning/rosa-planning-environment.adoc#rosa-planning-environment[Planning your environment]
36+
37+
* xref:../rosa_planning/rosa-limits-scalability.adoc#rosa-limits-scalability[Limits and scalability]
38+
39+
// Setting a higher pid limit on an existing cluster
40+
include::modules/setting-higher-pid-limit-on-existing-cluster.adoc[leveloffset=+1]

‎rosa_cluster_admin/rosa_nodes/rosa-nodes-machinepools-about.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,3 +63,4 @@ ifdef::openshift-rosa[]
6363
* xref:../../rosa_cluster_admin/rosa_nodes/rosa-managing-worker-nodes.adoc#rosa-managing-worker-nodes[Managing compute nodes]
6464
endif::[]
6565
* xref:../../rosa_cluster_admin/rosa_nodes/rosa-nodes-about-autoscaling-nodes.adoc#rosa-nodes-about-autoscaling-nodes[About autoscaling]
66+
* xref:../../rosa_cluster_admin/rosa-configuring-pid-limits.adoc#rosa-configuring-pid-limits[Configuring PID limits]

‎snippets/pid-limits.adoc

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,17 @@
1515

1616
:_mod-docs-content-type: SNIPPET
1717

18+
// The PID limits are only configurable through the ROSA CLI. Once they are configurable through OCM, the conditionalization can be removed.
19+
1820
[IMPORTANT]
1921
====
20-
As of the {product-title} versions 4.8.35, 4.9.26, 4.10.6, the {product-title} default per-pod pid limit is `4096`. If you want to enable this PID limit, you must upgrade your {product-title} clusters to these versions or later. {product-title} clusters with prior versions use a default PID limit of `1024`.
22+
As of {product-title} 4.11, the default per-pod PID limit is `4096`. If you want to enable this PID limit, you must upgrade your {product-title} clusters to this version or later. {product-title} clusters running on earlier versions use a default PID limit of `1024`.
23+
24+
ifdef::openshift-rosa[]
25+
You can configure the per-pod PID limit on a {product-title} cluster by using the ROSA CLI. For more information, see "Configuring PID limits".
26+
endif::openshift-rosa[]
2127
28+
ifdef::openshift-dedicated[]
2229
You cannot configure the per-pod PID limit on any {product-title} cluster.
23-
====
30+
endif::openshift-dedicated[]
31+
====

0 commit comments

Comments
 (0)