Skip to content

Commit 7fb8441

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #47425 from SergeyKanzhelev/kubelet-files
outline files kubelet uses
2 parents 4585cc1 + 598a418 commit 7fb8441

File tree

2 files changed

+174
-0
lines changed

2 files changed

+174
-0
lines changed

‎content/en/docs/reference/node/_index.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@ This section contains the following reference topics about nodes:
1313

1414
* [Node Labels Populated By The Kubelet](/docs/reference/node/node-labels)
1515

16+
* [Local Files And Paths Used By The Kubelet](/docs/reference/node/kubelet-files)
17+
1618
* [Node `.status` information](/docs/reference/node/node-status/)
1719

1820
* [Seccomp information](/docs/reference/node/seccomp/)

‎content/en/docs/reference/node/kubelet-files.md

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
---
2+
content_type: "reference"
3+
title: Local Files And Paths Used By The Kubelet
4+
weight: 42
5+
---
6+
7+
The {{< glossary_tooltip text="kubelet" term_id="kubelet" >}} is mostly a stateless
8+
process running on a Kubernetes {{< glossary_tooltip text="node" term_id="node" >}}.
9+
This document outlines files that kubelet reads and writes.
10+
11+
{{< note >}}
12+
13+
This document is for informational purpose and not describing any guaranteed behaviors or APIs.
14+
It lists resources used by the kubelet, which is an implementation detail and a subject to change at any release.
15+
16+
{{< /note >}}
17+
18+
The kubelet typically uses the {{< glossary_tooltip text="control plane" term_id="control-plane" >}} as
19+
the source of truth on what needs to run on the Node, and the
20+
{{<glossary_tooltip text="container runtime" term_id="container-runtime">}} to retrieve
21+
the current state of containers. So long as you provide a _kubeconfig_ (API client configuration)
22+
to the kubelet, the kubelet does connect to your control plane; otherwise the node operates in
23+
_standalone mode_.
24+
25+
On Linux nodes, the kubelet also relies on reading cgroups and various system files to collect metrics.
26+
27+
On Windows nodes, the kubelet collects metrics via a different mechanism that does not rely on
28+
paths.
29+
30+
There are also a few other files that are used by the kubelet as well as kubelet communicates using local Unix-domain sockets. Some are sockets that the
31+
kubelet listens on, and for other sockets the kubelet discovers them and then connects
32+
as a client.
33+
34+
{{< note >}}
35+
36+
This page lists paths as Linux paths, which map to the Windows paths by adding a root disk
37+
`C:\` in place of `/` (unless specified otherwise). For example, `/var/lib/kubelet/device-plugins` maps to `C:\var\lib\kubelet\device-plugins`.
38+
39+
{{< /note >}}
40+
41+
## Configuration
42+
43+
### Kubelet configuration files
44+
45+
The path to the kubelet configuration file can be configured
46+
using the command line argument `--config`. The kubelet also supports
47+
[drop-in configuration files](/docs/tasks/administer-cluster/kubelet-config-file/#kubelet-conf-d)
48+
to enhance configuration.
49+
50+
### Certificates
51+
52+
Certificates and private keys are typically located at `/var/lib/kubelet/pki`,
53+
but can be configured using the `--cert-dir` kubelet command line argument.
54+
Names of certificate files are also configurable.
55+
56+
### Manifests
57+
58+
Manifests for static pods are typically located in `/etc/kubernetes/manifests`.
59+
Location can be configured using the `staticPodPath` kubelet configuration option.
60+
61+
### Systemd unit settings
62+
63+
When kubelet is running as a systemd unit, some kubelet configuration may be declared
64+
in systemd unit settings file. Typically it includes:
65+
66+
- command line arguments to [run kubelet](/docs/reference/command-line-tools-reference/kubelet/)
67+
- environment variables, used by kubelet or [configuring golang runtime](https://pkg.go.dev/runtime#hdr-Environment_Variables)
68+
69+
## State
70+
71+
### Checkpoint files for resource managers {#resource-managers-state}
72+
73+
All resource managers keep the mapping of Pods to allocated resources in state files.
74+
State files are located in the kubelet's base directory, also termed the _root directory_
75+
(but not the same as `/`, the node root directory). You can configure the base directory
76+
for the kubelet
77+
using the kubelet command line argument `--root-dir`.
78+
79+
Names of files:
80+
81+
- `memory_manager_state` for the [Memory Manager](/docs/tasks/administer-cluster/memory-manager/)
82+
- `cpu_manager_state` for the [CPU Manager](/docs/tasks/administer-cluster/cpu-management-policies/)
83+
- `dra_manager_state` for [DRA](/docs/concepts/scheduling-eviction/dynamic-resource-allocation/)
84+
85+
### Checkpoint file for device manager {#device-manager-state}
86+
87+
Device manager creates checkpoints in the same directory with socket files: `/var/lib/kubelet/device-plugins/`.
88+
The name of a checkpoint file is `kubelet_internal_checkpoint` for [Device Manager](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-integration-with-the-topology-manager)
89+
90+
### Pod status checkpoint storage {#pod-status-manager-state}
91+
92+
{{< feature-state feature_gate_name="InPlacePodVerticalScaling" >}}
93+
94+
If your cluster has
95+
[in-place Pod vertical scaling](/docs/concepts/workloads/autoscaling/#in-place-resizing)
96+
enabled ([feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
97+
name `InPlacePodVerticalScaling`), then the kubelet stores a local record of Pod status.
98+
99+
The file name is `pod_status_manager_state` within the kubelet base directory
100+
(`/var/lib/kubelet` by default on Linux; configurable using `--root-dir`).
101+
102+
### Container runtime
103+
104+
Kubelet communicates with the container runtime using socket configured via the
105+
configuration parameters:
106+
107+
- `containerRuntimeEndpoint` for runtime operations
108+
- `imageServiceEndpoint` for image management operations
109+
110+
The actual values of those endpoints depend on the container runtime being used.
111+
112+
### Device plugins
113+
114+
The kubelet exposes a socket at the path `/var/lib/kubelet/device-plugins/kubelet.sock` for
115+
various [Device Plugins to register](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-implementation).
116+
117+
When a device plugin registers itself, it provides its socket path for the kubelet to connect.
118+
119+
The device plugin socket should be in the directory `device-plugins` within the kubelet base
120+
directory. On a typical Linux node, this means `/var/lib/kubelet/device-plugins`.
121+
122+
### Pod resources API
123+
124+
[Pod Resources API](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#monitoring-device-plugin-resources)
125+
will be exposed at the path `/var/lib/kubelet/pod-resources`.
126+
127+
### DRA, CSI, and Device plugins
128+
129+
The kubelet looks for socket files created by device plugins managed via [DRA](/docs/concepts/scheduling-eviction/dynamic-resource-allocation/),
130+
device manager, or storage plugins, and then attempts to connect
131+
to these sockets. The directory that the kubelet looks in is `plugins_registry` within the kubelet base
132+
directory, so on a typical Linux node this means `/var/lib/kubelet/plugins_registry`.
133+
134+
Note, for the device plugins there are two alternative registration mechanisms. Only one should be used for a given plugin.
135+
136+
The types of plugins that can place socket files into that directory are:
137+
138+
- CSI plugins
139+
- DRA plugins
140+
- Device Manager plugins
141+
142+
(typically `/var/lib/kubelet/plugins_registry`).
143+
144+
## Security profiles & configuration
145+
146+
### Seccomp
147+
148+
Seccomp profile files referenced from Pods should be placed in `/var/lib/kubelet/seccomp`.
149+
See the [seccomp reference](/docs/reference/node/seccomp/) for details.
150+
151+
### AppArmor
152+
153+
The kubelet does not load or refer to AppArmor profiles by a Kubernetes-specific path.
154+
AppArmor profiles are loaded via the node operating system rather then referenced by their path.
155+
156+
## Locking
157+
158+
{{< feature-state state="alpha" for_k8s_version="v1.2" >}}
159+
160+
161+
A lock file for the kubelet; typically `/var/run/kubelet.lock`. The kubelet uses this to ensure
162+
that two different kubelets don't try to run in conflict with each other.
163+
You can configure the path to the lock file using the the `--lock-file` kubelet command line argument.
164+
165+
If two kubelets on the same node use a different value for the lock file path, they will not be able to
166+
detect a conflict when both are running.
167+
168+
169+
## {{% heading "whatsnext" %}}
170+
171+
- Learn about the kubelet [command line arguments](/docs/reference/command-line-tools-reference/kubelet/).
172+
- Review the [Kubelet Configuration (v1beta1) reference](/docs/reference/config-api/kubelet-config.v1beta1/)

0 commit comments

Comments
 (0)