Skip to content

Commit caa05b9

Browse files
authored
Merge pull request #25245 from SergeyKanzhelev/pid-available
add `pid.available` to the eviction signals list
2 parents 7f1bff9 + d1dc73c commit caa05b9

File tree

2 files changed

+23
-4
lines changed

2 files changed

+23
-4
lines changed

content/en/docs/concepts/policy/pid-limiting.md

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,16 +33,16 @@ On certain Linux installations, the operating system sets the PIDs limit to a lo
3333
such as `32768`. Consider raising the value of `/proc/sys/kernel/pid_max`.
3434
{{< /note >}}
3535

36-
You can configure a kubelet to limit the number of PIDs a given pod can consume.
36+
You can configure a kubelet to limit the number of PIDs a given Pod can consume.
3737
For example, if your node's host OS is set to use a maximum of `262144` PIDs and
38-
expect to host less than `250` pods, one can give each pod a budget of `1000`
38+
expect to host less than `250` Pods, one can give each Pod a budget of `1000`
3939
PIDs to prevent using up that node's overall number of available PIDs. If the
4040
admin wants to overcommit PIDs similar to CPU or memory, they may do so as well
41-
with some additional risks. Either way, a single pod will not be able to bring
41+
with some additional risks. Either way, a single Pod will not be able to bring
4242
the whole machine down. This kind of resource limiting helps to prevent simple
4343
fork bombs from affecting operation of an entire cluster.
4444

45-
Per-pod PID limiting allows administrators to protect one pod from another, but
45+
Per-Pod PID limiting allows administrators to protect one Pod from another, but
4646
does not ensure that all Pods scheduled onto that host are unable to impact the node overall.
4747
Per-Pod limiting also does not protect the node agents themselves from PID exhaustion.
4848

@@ -92,8 +92,26 @@ the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
9292
`SupportPodPidsLimit` to work.
9393
{{< /note >}}
9494

95+
## PID based eviction
96+
97+
You can configure kubelet to start terminating a Pod when it is misbehaving and consuming abnormal amount of resources.
98+
This feature is called eviction. You can [Configure Out of Resource Handling](/docs/tasks/administer-cluster/out-of-resource) for various eviction signals.
99+
Use `pid.available` eviction signal to configure the threshold for number of PIDs used by Pod.
100+
You can set soft and hard eviction policies. However, even with the hard eviction policy, if the number of PIDs growing very fast,
101+
node can still get into unstable state by hitting the node PIDs limit.
102+
Eviction signal value is calculated periodically and does NOT enforce the limit.
103+
104+
PID limiting - per Pod and per Node sets the hard limit.
105+
Once the limit is hit, workload will start experiencing failures when trying to get a new PID.
106+
It may or may not lead to rescheduling of a Pod,
107+
depending on how workload reacts on these failures and how liveleness and readiness
108+
probes are configured for the Pod. However, if limits were set correctly,
109+
you can guarantee that other Pods workload and system processes will not run out of PIDs
110+
when one Pod is misbehaving.
111+
95112
## {{% heading "whatsnext" %}}
96113

97114
- Refer to the [PID Limiting enhancement document](https://github.com/kubernetes/enhancements/blob/097b4d8276bc9564e56adf72505d43ce9bc5e9e8/keps/sig-node/20190129-pid-limiting.md) for more information.
98115
- For historical context, read [Process ID Limiting for Stability Improvements in Kubernetes 1.14](/blog/2019/04/15/process-id-limiting-for-stability-improvements-in-kubernetes-1.14/).
99116
- Read [Managing Resources for Containers](/docs/concepts/configuration/manage-resources-containers/).
117+
- Learn how to [Configure Out of Resource Handling](/docs/tasks/administer-cluster/out-of-resource).

content/en/docs/tasks/administer-cluster/out-of-resource.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ the `kubelet` summary API.
3131
| `nodefs.inodesFree` | `nodefs.inodesFree` := `node.stats.fs.inodesFree` |
3232
| `imagefs.available` | `imagefs.available` := `node.stats.runtime.imagefs.available` |
3333
| `imagefs.inodesFree` | `imagefs.inodesFree` := `node.stats.runtime.imagefs.inodesFree` |
34+
| `pid.available` | `pid.available` := `node.stats.rlimit.maxpid` - `node.stats.rlimit.curproc` |
3435

3536
Each of the above signals supports either a literal or percentage based value.
3637
The percentage based value is calculated relative to the total capacity

0 commit comments

Comments
 (0)