Skip to content

Commit 3b19dbf

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #32764 from neolit123/1.24-add-steps-for-reconf
kubeadm: add task page for cluster reconf
2 parents f764ec7 + 931581e commit 3b19dbf

File tree

4 files changed

+295
-3
lines changed

4 files changed

+295
-3
lines changed

‎content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,12 @@ you can skip the default CoreDNS deployment and deploy your own variant.
2424
For more details on that see [Using init phases with kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-phases).
2525
{{< /note >}}
2626

27-
<!-- body -->
27+
{{< note >}}
28+
To reconfigure a cluster that has already been created see
29+
[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure).
30+
{{< /note >}}
2831

29-
{{< feature-state for_k8s_version="v1.12" state="stable" >}}
32+
<!-- body -->
3033

3134
## Customizing the control plane with flags in `ClusterConfiguration`
3235

‎content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,9 @@ To customize control plane components, including optional IPv6 assignment to liv
162162
for control plane components and etcd server, provide extra arguments to each component as documented in
163163
[custom arguments](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/).
164164

165+
To reconfigure a cluster that has already been created see
166+
[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure).
167+
165168
To run `kubeadm init` again, you must first [tear down the cluster](#tear-down).
166169

167170
If you join a node with a different architecture to your cluster, make sure that your deployed DaemonSets

‎content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure.md

Lines changed: 281 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,281 @@
1+
---
2+
reviewers:
3+
- sig-cluster-lifecycle
4+
title: Reconfiguring a kubeadm cluster
5+
content_type: task
6+
weight: 10
7+
---
8+
9+
<!-- overview -->
10+
11+
kubeadm does not support automated ways of reconfiguring components that
12+
were deployed on managed nodes. One way of automating this would be
13+
by using a custom [operator](/docs/concepts/extend-kubernetes/operator/).
14+
15+
To modify the components configuration you must manually edit associated cluster
16+
objects and files on disk.
17+
18+
This guide shows the correct sequence of steps that need to be performed
19+
to achieve kubeadm cluster reconfiguration.
20+
21+
## {{% heading "prerequisites" %}}
22+
23+
- You need a cluster that was deployed using kubeadm
24+
- Have administrator credentials (`/etc/kubernetes/admin.conf`) and network connectivity
25+
to a running kube-apiserver in the cluster from a host that has kubectl installed
26+
- Have a text editor installed on all hosts
27+
28+
<!-- steps -->
29+
30+
## Reconfiguring the cluster
31+
32+
kubeadm writes a set of cluster wide component configuration options in
33+
ConfigMaps and other objects. These objects must be manually edited. The command `kubectl edit`
34+
can be used for that.
35+
36+
The `kubectl edit` command will open a text editor where you can edit and save the object directly.
37+
38+
You can use the environment variables `KUBECONFIG` and `KUBE_EDITOR` to specify the location of
39+
the kubectl consumed kubeconfig file and preferred text editor.
40+
41+
For example:
42+
```
43+
KUBECONFIG=/etc/kubernetes/admin.conf KUBE_EDITOR=nano kubectl edit <parameters>
44+
```
45+
46+
{{< note >}}
47+
Upon saving any changes to these cluster objects, components running on nodes may not be
48+
automatically updated. The steps below instruct you on how to perform that manually.
49+
{{< /note >}}
50+
51+
{{< warning >}}
52+
Component configuration in ConfigMaps is stored as unstructured data (YAML string).
53+
This means that validation will not be performed upon updating the contents of a ConfigMap.
54+
You have to be careful to follow the documented API format for a particular
55+
component configuration and avoid introducing typos and YAML indentation mistakes.
56+
{{< /warning >}}
57+
58+
### Applying cluster configuration changes
59+
60+
#### Updating the `ClusterConfiguration`
61+
62+
During cluster creation and upgrade, kubeadm writes its
63+
[`ClusterConfiguration`](/docs/reference/config-api/kubeadm-config.v1beta3/)
64+
in a ConfigMap called `kubeadm-config` in the `kube-system` namespace.
65+
66+
To change a particular option in the `ClusterConfiguration` you can edit the ConfigMap with this command:
67+
68+
```shell
69+
kubectl edit cm -n kube-system kubeadm-config
70+
```
71+
72+
The configuration is located under the `data.ClusterConfiguration` key.
73+
74+
{{< note >}}
75+
The `ClusterConfiguration` includes a variety of options that affect the configuration of individual
76+
components such as kube-apiserver, kube-scheduler, kube-controller-manager, CoreDNS, etcd and kube-proxy.
77+
Changes to the configuration must be reflected on node components manually.
78+
{{< /note >}}
79+
80+
#### Reflecting `ClusterConfiguration` changes on control plane nodes
81+
82+
kubeadm manages the control plane components as static Pod manifests located in
83+
the directory `/etc/kubernetes/manifests`.
84+
Any changes to the `ClusterConfiguration` under the `apiServer`, `controllerManager`, `scheduler` or `etcd`
85+
keys must be reflected in the associated files in the manifests directory on a control plane node.
86+
87+
Such changes may include:
88+
- `extraArgs` - requires updating the list of flags passed to a component container
89+
- `extraMounts` - requires updated the volume mounts for a component container
90+
- `*SANs` - requires writing new certificates with updated Subject Alternative Names.
91+
92+
Before proceeding with these changes, make sure you have backed up the directory `/etc/kubernetes/`.
93+
94+
To write new certificates you can use:
95+
```shell
96+
kubeadm init phase certs <component-name> --config <config-file>
97+
```
98+
99+
To write new manifest files in `/etc/kubernetes/manifests` you can use:
100+
101+
```shell
102+
kubeadm init phase control-plane <component-name> --config <config-file>
103+
```
104+
105+
The `<config-file>` contents must match the updated `ClusterConfiguration`.
106+
The `<component-name>` value must be the name of the component.
107+
108+
{{< note >}}
109+
Updating a file in `/etc/kubernetes/manifests` will tell the kubelet to restart the static Pod for the corresponding component.
110+
Try doing these changes one node at a time to leave the cluster without downtime.
111+
{{< /note >}}
112+
113+
### Applying kubelet configuration changes
114+
115+
#### Updating the `KubeletConfiguration`
116+
117+
During cluster creation and upgrade, kubeadm writes its
118+
[`KubeletConfiguration`](/docs/reference/config-api/kubelet-config.v1beta1/)
119+
in a ConfigMap called `kubelet-config` in the `kube-system` namespace.
120+
121+
You can edit the ConfigMap with this command:
122+
123+
```shell
124+
kubectl edit cm -n kube-system kubelet-config
125+
```
126+
127+
The configuration is located under the `data.kubelet` key.
128+
129+
#### Reflecting the kubelet changes
130+
131+
To reflect the change on kubeadm nodes you must do the following:
132+
- Log in to a kubeadm node
133+
- Run `kubeadm upgrade node phase kubelet-config` to download the latest `kubelet-config`
134+
ConfigMap contents into the local file `/var/lib/kubelet/config.conf`
135+
- Edit the file `/var/lib/kubelet/kubeadm-flags.env` to apply additional configuration with
136+
flags
137+
- Restart the kubelet service with `systemctl restart kubelet`
138+
139+
{{< note >}}
140+
Do these changes one node at a time to allow workloads to be rescheduled properly.
141+
{{< /note >}}
142+
143+
{{< note >}}
144+
During `kubeadm upgrade`, kubeadm downloads the `KubeletConfiguration` from the
145+
`kubelet-config` ConfigMap and overwrite the contents of `/var/lib/kubelet/config.conf`.
146+
This means that node local configuration must be applied either by flags in
147+
`/var/lib/kubelet/kubeadm-flags.env` or by manually updating the contents of
148+
`/var/lib/kubelet/config.conf` after `kubeadm upgrade`, and then restarting the kubelet.
149+
{{< /note >}}
150+
151+
### Applying kube-proxy configuration changes
152+
153+
#### Updating the `KubeProxyConfiguration`
154+
155+
During cluster creation and upgrade, kubeadm writes its
156+
[`KubeProxyConfiguration`](/docs/reference/config-api/kube-proxy-config.v1alpha1/)
157+
in a ConfigMap in the `kube-system` namespace called `kube-proxy`.
158+
159+
This ConfigMap is used by the `kube-proxy` DaemonSet in the `kube-system` namespace.
160+
161+
To change a particular option in the `KubeProxyConfiguration`, you can edit the ConfigMap with this command:
162+
163+
```shell
164+
kubectl edit cm -n kube-system kube-proxy
165+
```
166+
167+
The configuration is located under the `data.config.conf` key.
168+
169+
#### Reflecting the kube-proxy changes
170+
171+
Once the `kube-proxy` ConfigMap is updated, you can restart all kube-proxy Pods:
172+
173+
Obtain the Pod names:
174+
175+
```shell
176+
kubectl get po -n kube-system | grep kube-proxy
177+
```
178+
179+
Delete a Pod with:
180+
181+
```shell
182+
kubectl delete po -n kube-system <pod-name>
183+
```
184+
185+
New Pods that use the updated ConfigMap will be created.
186+
187+
{{< note >}}
188+
Because kubeadm deploys kube-proxy as a DaemonSet, node specific configuration is unsupported.
189+
{{< /note >}}
190+
191+
### Applying CoreDNS configuration changes
192+
193+
#### Updating the CoreDNS Deployment and Service
194+
195+
kubeadm deploys CoreDNS as a Deployment called `coredns` and with a Service `kube-dns`,
196+
both in the `kube-system` namespace.
197+
198+
To update any of the CoreDNS settings, you can edit the Deployment and
199+
Service objects:
200+
201+
```shell
202+
kubectl edit deployment -n kube-system coredns
203+
kubectl edit service -n kube-system kube-dns
204+
```
205+
206+
#### Reflecting the CoreDNS changes
207+
208+
Once the CoreDNS changes are applied you can delete the CoreDNS Pods:
209+
210+
Obtain the Pod names:
211+
212+
```shell
213+
kubectl get po -n kube-system | grep coredns
214+
```
215+
216+
Delete a Pod with:
217+
218+
```shell
219+
kubectl delete po -n kube-system <pod-name>
220+
```
221+
222+
New Pods with the updated CoreDNS configuration will be created.
223+
224+
{{< note >}}
225+
kubeadm does not allow CoreDNS configuration during cluster creation and upgrade.
226+
This means that if you execute `kubeadm upgrade apply`, your changes to the CoreDNS
227+
objects will be lost and must be reapplied.
228+
{{< /note >}}
229+
230+
## Persisting the reconfiguration
231+
232+
During the execution of `kubeadm upgrade` on a managed node, kubeadm might overwrite configuration
233+
that was applied after the cluster was created (reconfiguration).
234+
235+
### Persisting Node object reconfiguration
236+
237+
kubeadm writes Labels, Taints, CRI socket and other information on the Node object for a particular
238+
Kubernetes node. To change any of the contents of this Node object you can use:
239+
240+
```shell
241+
kubectl edit no <node-name>
242+
```
243+
244+
During `kubeadm upgrade` the contents of such a Node might get overwritten.
245+
If you would like to persist your modifications to the Node object after upgrade,
246+
you can prepare a [kubectl patch](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/)
247+
and apply it to the Node object:
248+
249+
```shell
250+
kubectl patch no <node-name> --patch-file <patch-file>
251+
```
252+
253+
#### Persisting control plane component reconfiguration
254+
255+
The main source of control plane configuration is the `ClusterConfiguration`
256+
object stored in the cluster. To extend the static Pod manifests configuration,
257+
[patches](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/#patches) can be used.
258+
259+
These patch files must remain as files on the control plane nodes to ensure that
260+
they can be used by the `kubeadm upgrade ... --patches <directory>`.
261+
262+
If reconfiguration is done to the `ClusterConfiguration` and static Pod manifests on disk,
263+
the set of node specific patches must be updated accordingly.
264+
265+
#### Persisting kubelet reconfiguration
266+
267+
Any changes to the `KubeletConfiguration` stored in `/var/lib/kubelet/config.conf` will be overwritten on
268+
`kubeadm upgrade` by downloading the contents of the cluster wide `kubelet-config` ConfigMap.
269+
To persist kubelet node specific configuration either the file `/var/lib/kubelet/config.conf`
270+
has to be updated manually post-upgrade or the file `/var/lib/kubelet/kubeadm-flags.env` can include flags.
271+
The kubelet flags override the associated `KubeletConfiguration` options, but note that
272+
some of the flags are deprecated.
273+
274+
A kubelet restart will be required after changing `/var/lib/kubelet/config.conf` or
275+
`/var/lib/kubelet/kubeadm-flags.env`.
276+
277+
{{% heading "whatsnext" %}}
278+
279+
- [Upgrading kubeadm clusters](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade)
280+
- [Customizing components with the kubeadm API](/docs/setup/production-environment/tools/kubeadm/control-plane-flags)
281+
- [Certificate management with kubeadm](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs)

‎content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,12 @@ first drain the node (or nodes) that you are upgrading. In the case of control p
4343
they could be running CoreDNS Pods or other critical workloads. For more information see
4444
[Draining nodes](/docs/tasks/administer-cluster/safely-drain-node/).
4545
- All containers are restarted after upgrade, because the container spec hash value is changed.
46-
- To verify that the kubelet service has successfully restarted after the kubelet has been upgraded, you can execute `systemctl status kubelet` or view the service logs with `journalctl -xeu kubelet`.
46+
- To verify that the kubelet service has successfully restarted after the kubelet has been upgraded,
47+
you can execute `systemctl status kubelet` or view the service logs with `journalctl -xeu kubelet`.
48+
- Usage of the `--config` flag of `kubeadm upgrade` with
49+
[kubeadm configuration API types](/docs/reference/config-api/kubeadm-config.v1beta3)
50+
with the purpose of reconfiguring the cluster is not recommended and can have unexpected results. Follow the steps in
51+
[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure) instead.
4752

4853
<!-- steps -->
4954

0 commit comments

Comments
 (0)