Merge pull request #27071 from neolit123/1.21-add-kubelet-serving-cert-guide

kubeadm: add guide for kubelet serving certs and metrics-server

Author: k8s-ci-robot

content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md

@@ -408,7 +408,7 @@ be advised that this is modifying a design principle of the Linux distribution.
 
 ## `kubeadm upgrade plan` prints out `context deadline exceeded` error message
 
-This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with `kubeadm upgrade apply ...`. 
+This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with `kubeadm upgrade apply ...`.
 
 This issue is fixed as of version 1.19.
 
@@ -420,3 +420,20 @@ To workaround the issue, re-mount the `/var/lib/kubelet` directory after perform
 
 This is a regression introduced in kubeadm 1.15. The issue is fixed in 1.20.
 
+## Cannot use the metrics-server securely in a kubeadm cluster
+
+In a kubeadm cluster, the [metrics-server](https://github.com/kubernetes-sigs/metrics-server)
+can be used insecurely by passing the `--kubelet-insecure-tls` to it. This is not recommended for production clusters.
+
+If you want to use TLS between the metrics-server and the kubelet there is a problem,
+since kubeadm deploys a self-signed serving certificate for the kubelet. This can cause the following errors
+on the side of the metrics-server:
+```
+x509: certificate signed by unknown authority
+x509: certificate is valid for IP-foo not IP-bar
+```
+
+See [Enabling signed kubelet serving certificates](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs)
+to understand how to configure the kubelets in a kubeadm cluster to have properly signed serving certificates.
+
+Also see [How to run the metrics-server securely](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md#how-to-run-metrics-server-securely).

content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md

@@ -207,3 +207,71 @@ After a certificate is signed using your preferred method, the certificate and t
 Kubeadm does not support rotation or replacement of CA certificates out of the box.
 
 For more information about manual rotation or replacement of CA, see [manual rotation of CA certificates](/docs/tasks/tls/manual-rotation-of-ca-certificates/).
+
+## Enabling signed kubelet serving certificates {#kubelet-serving-certs}
+
+By default the kubelet serving certificate deployed by kubeadm is self-signed.
+This means a connection from external services like the
+[metrics-server](https://github.com/kubernetes-sigs/metrics-server) to a
+kubelet cannot be secured with TLS.
+
+To configure the kubelets in a new kubeadm cluster to obtain properly signed serving
+certificates you must pass the following minimal configuration to `kubeadm init`:
+
+```yaml
+apiVersion: kubeadm.k8s.io/v1beta2
+kind: ClusterConfiguration
+---
+apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+serverTLSBootstrap: true
+```
+
+If you have already created the cluster you must adapt it by doing the following:
+ - Find and edit the `kubelet-config-{{< skew latestVersion >}}` ConfigMap in the `kube-system` namespace.
+In that ConfigMap, the `config` key has a
+[KubeletConfiguration](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
+document as its value. Edit the KubeletConfiguration document to set `serverTLSBootstrap: true`.
+- On each node, add the `serverTLSBootstrap: true` field in `/var/lib/kubelet/config.yaml`
+and restart the kubelet with `systemctl restart kubelet`
+
+The field `serverTLSBootstrap: true` will enable the bootstrap of kubelet serving
+certificates by requesting them from the `certificates.k8s.io` API. One known limitation
+is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically
+approved by the default signer in the kube-controller-manager -
+[`kubernetes.io/kubelet-serving`](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers).
+This will require action from the user or a third party controller.
+
+These CSRs can be viewed using:
+
+```shell
+kubectl get csr
+NAME        AGE     SIGNERNAME                        REQUESTOR                      CONDITION
+csr-9wvgt   112s    kubernetes.io/kubelet-serving     system:node:worker-1           Pending
+csr-lz97v   1m58s   kubernetes.io/kubelet-serving     system:node:control-plane-1    Pending
+```
+
+To approve them you can do the following:
+```shell
+kubectl certificate approve <CSR-name>
+```
+
+By default, these serving certificate will expire after one year. Kubeadm sets the
+`KubeletConfiguration` field `rotateCertificates` to `true`, which means that close
+to expiration a new set of CSRs for the serving certificates will be created and must
+be approved to complete the rotation. To understand more see
+[Certificate Rotation](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#certificate-rotation).
+
+If you are looking for a solution for automatic approval of these CSRs it is recommended
+that you contact your cloud provider and ask if they have a CSR signer that verifies
+the node identity with an out of band mechanism.
+
+{{% thirdparty-content %}}
+
+Third party custom controllers can be used:
+- [kubelet-rubber-stamp](https://github.com/kontena/kubelet-rubber-stamp)
+
+Such a controller is not a secure mechanism unless it not only verifies the CommonName
+in the CSR but also verifies the requested IPs and domain names. This would prevent
+a malicious actor that has access to a kubelet client certificate to create
+CSRs requesting serving certificates for any IP or domain name.