Document kubelet client certificate recovery

break long lines

simplify wording (suggested by neolit123)

phrasing error/typo

don't mention creating a new kuebadm config file, not really relevant here

remove unnecessary stop from kubelet cert troubleshoot

minor phrasing improvement: *the* kublet

make the steps for fixing kubelet certs separate from other troubleshooting suggestions

move kubelet cert troubleshooting to a better place

improve note wording and add link

add missing dot

Co-authored-by: Lubomir I. Ivanov <[email protected]>

add missing dot

Co-authored-by: Lubomir I. Ivanov <[email protected]>

remove steps not related to the issue

Make instructions work on worker nodes too

Co-authored-by: Lubomir I. Ivanov <[email protected]>

Author: antoinep92

content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md

@@ -220,6 +220,25 @@ Unable to connect to the server: x509: certificate signed by unknown authority (
   sudo chown $(id -u):$(id -g) $HOME/.kube/config
   ```
 
+## Kubelet client certificate rotation fails {#kubelet-client-cert}
+
+By default, kubeadm configures a kubelet with automatic rotation of client certificates by using the `/var/lib/kubelet/pki/kubelet-client-current.pem` symlink specified in `/etc/kubernetes/kubelet.conf`.
+If this rotation process fails you might see errors such as `x509: certificate has expired or is not yet valid`
+in kube-apserver logs. To fix the issue you must follow these steps:
+
+1. Backup and delete `/etc/kubernetes/kubelet.conf` and `/var/lib/kubelet/pki/kubelet-client*` from the failed node.
+1. From a working control plane node in the cluster that has `/etc/kubernetes/pki/ca.key` execute
+`kubeadm kubeconfig user --org system:nodes --client-name system:node:$NODE > kubelet.conf`.
+`$NODE` must be set to the name of the existing failed node in the cluster.
+Modify the resulted `kubelet.conf` manually to adjust the cluster name and server endpoint,
+or pass `kubeconfig user --config` (it accepts `InitConfiguration`). If your cluster does not have
+the `ca.key` you must sign the embedded certificates in the `kubelet.conf` externally.
+1. Copy this resulted `kubelet.conf` to `/etc/kubernetes/kubelet.conf` on the failed node.
+1. Restart the kubelet (`systemctl restart kubelet`) on the failed node and wait for
+`/var/lib/kubelet/pki/kubelet-client-current.pem` to be recreated.
+1. Run `kubeadm init phase kubelet-finalize all` on the failed node. This will make the new
+`kubelet.conf` file use `/var/lib/kubelet/pki/kubelet-client-current.pem` and will restart the kubelet.
+1. Make sure the node becomes `Ready`.
 ## Default NIC When using flannel as the pod network in Vagrant
 
 The following error might indicate that something was wrong in the pod network:

content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md

@@ -85,7 +85,11 @@ Additionally, kubeadm informs the user if the certificate is externally managed;
 {{< /warning >}}
 
 {{< note >}}
-`kubelet.conf` is not included in the list above because kubeadm configures kubelet for automatic certificate renewal.
+`kubelet.conf` is not included in the list above because kubeadm configures kubelet
+for [automatic certificate renewal](/docs/tasks/tls/certificate-rotation/)
+with rotatable certificates under `/var/lib/kubelet/pki`.
+To repair an expired kubelet client certificate see
+[Kubelet client certificate rotation fails](/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert).
 {{< /note >}}
 
 {{< warning >}}