Merge pull request #44105 from windsonsea/shooty

Clean up troubleshooting-kubeadm.md

Author: k8s-ci-robot

content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md

@@ -73,7 +73,8 @@ If you see the following warnings while running `kubeadm init`
 [preflight] WARNING: ethtool not found in system path
 ```
 
-Then you may be missing `ebtables`, `ethtool` or a similar executable on your node. You can install them with the following commands:
+Then you may be missing `ebtables`, `ethtool` or a similar executable on your node.
+You can install them with the following commands:
 
 - For Ubuntu/Debian users, run `apt install ebtables ethtool`.
 - For CentOS/Fedora users, run `yum install ebtables ethtool`.
@@ -90,9 +91,9 @@ This may be caused by a number of problems. The most common are:
 
 - network connection problems. Check that your machine has full network connectivity before continuing.
 - the cgroup driver of the container runtime differs from that of the kubelet. To understand how to
-  configure it properly see [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/).
+  configure it properly, see [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/).
 - control plane containers are crashlooping or hanging. You can check this by running `docker ps`
-  and investigating each container by running `docker logs`. For other container runtime see
+  and investigating each container by running `docker logs`. For other container runtime, see
   [Debugging Kubernetes nodes with crictl](/docs/tasks/debug/debug-cluster/crictl/).
 
 ## kubeadm blocks when removing managed containers
@@ -144,10 +145,12 @@ provider. Please contact the author of the Pod Network add-on to find out whethe
 
 Calico, Canal, and Flannel CNI providers are verified to support HostPort.
 
-For more information, see the [CNI portmap documentation](https://github.com/containernetworking/plugins/blob/master/plugins/meta/portmap/README.md).
+For more information, see the
+[CNI portmap documentation](https://github.com/containernetworking/plugins/blob/master/plugins/meta/portmap/README.md).
 
-If your network provider does not support the portmap CNI plugin, you may need to use the [NodePort feature of
-services](/docs/concepts/services-networking/service/#type-nodeport) or use `HostNetwork=true`.
+If your network provider does not support the portmap CNI plugin, you may need to use the
+[NodePort feature of services](/docs/concepts/services-networking/service/#type-nodeport)
+or use `HostNetwork=true`.
 
 ## Pods are not accessible via their Service IP
 
@@ -157,9 +160,10 @@ services](/docs/concepts/services-networking/service/#type-nodeport) or use `Hos
   add-on provider to get the latest status of their support for hairpin mode.
 
 - If you are using VirtualBox (directly or via Vagrant), you will need to
-  ensure that `hostname -i` returns a routable IP address. By default the first
+  ensure that `hostname -i` returns a routable IP address. By default, the first
   interface is connected to a non-routable host-only network. A work around
-  is to modify `/etc/hosts`, see this [Vagrantfile](https://github.com/errordeveloper/k8s-playground/blob/22dd39dfc06111235620e6c4404a96ae146f26fd/Vagrantfile#L11)
+  is to modify `/etc/hosts`, see this
+  [Vagrantfile](https://github.com/errordeveloper/k8s-playground/blob/22dd39dfc06111235620e6c4404a96ae146f26fd/Vagrantfile#L11)
   for an example.
 
 ## TLS certificate errors
@@ -175,6 +179,7 @@ Unable to connect to the server: x509: certificate signed by unknown authority (
   regenerate a certificate if necessary. The certificates in a kubeconfig file
   are base64 encoded. The `base64 --decode` command can be used to decode the certificate
   and `openssl x509 -text -noout` can be used for viewing the certificate information.
+
 - Unset the `KUBECONFIG` environment variable using:
 
   ```sh
@@ -190,15 +195,16 @@ Unable to connect to the server: x509: certificate signed by unknown authority (
 - Another workaround is to overwrite the existing `kubeconfig` for the "admin" user:
 
   ```sh
-  mv  $HOME/.kube $HOME/.kube.bak
+  mv $HOME/.kube $HOME/.kube.bak
   mkdir $HOME/.kube
   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   sudo chown $(id -u):$(id -g) $HOME/.kube/config
   ```
 
 ## Kubelet client certificate rotation fails {#kubelet-client-cert}
 
-By default, kubeadm configures a kubelet with automatic rotation of client certificates by using the `/var/lib/kubelet/pki/kubelet-client-current.pem` symlink specified in `/etc/kubernetes/kubelet.conf`.
+By default, kubeadm configures a kubelet with automatic rotation of client certificates by using the
+`/var/lib/kubelet/pki/kubelet-client-current.pem` symlink specified in `/etc/kubernetes/kubelet.conf`.
 If this rotation process fails you might see errors such as `x509: certificate has expired or is not yet valid`
 in kube-apiserver logs. To fix the issue you must follow these steps:
 
@@ -231,24 +237,34 @@ The following error might indicate that something was wrong in the pod network:
 Error from server (NotFound): the server could not find the requested resource
 ```
 
-- If you're using flannel as the pod network inside Vagrant, then you will have to specify the default interface name for flannel.
+- If you're using flannel as the pod network inside Vagrant, then you will have to
+  specify the default interface name for flannel.
 
-  Vagrant typically assigns two interfaces to all VMs. The first, for which all hosts are assigned the IP address `10.0.2.15`, is for external traffic that gets NATed.
+  Vagrant typically assigns two interfaces to all VMs. The first, for which all hosts
+  are assigned the IP address `10.0.2.15`, is for external traffic that gets NATed.
 
-  This may lead to problems with flannel, which defaults to the first interface on a host. This leads to all hosts thinking they have the same public IP address. To prevent this, pass the `--iface eth1` flag to flannel so that the second interface is chosen.
+  This may lead to problems with flannel, which defaults to the first interface on a host.
+  This leads to all hosts thinking they have the same public IP address. To prevent this,
+  pass the `--iface eth1` flag to flannel so that the second interface is chosen.
 
 ## Non-public IP used for containers
 
-In some situations `kubectl logs` and `kubectl run` commands may return with the following errors in an otherwise functional cluster:
+In some situations `kubectl logs` and `kubectl run` commands may return with the
+following errors in an otherwise functional cluster:
 
 ```console
 Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc65b868-glc5m/mysql: dial tcp 10.19.0.41:10250: getsockopt: no route to host
 ```
 
-- This may be due to Kubernetes using an IP that can not communicate with other IPs on the seemingly same subnet, possibly by policy of the machine provider.
-- DigitalOcean assigns a public IP to `eth0` as well as a private one to be used internally as anchor for their floating IP feature, yet `kubelet` will pick the latter as the node's `InternalIP` instead of the public one.
+- This may be due to Kubernetes using an IP that can not communicate with other IPs on
+  the seemingly same subnet, possibly by policy of the machine provider.
+- DigitalOcean assigns a public IP to `eth0` as well as a private one to be used internally
+  as anchor for their floating IP feature, yet `kubelet` will pick the latter as the node's
+  `InternalIP` instead of the public one.
 
-  Use `ip addr show` to check for this scenario instead of `ifconfig` because `ifconfig` will not display the offending alias IP address. Alternatively an API endpoint specific to DigitalOcean allows to query for the anchor IP from the droplet:
+  Use `ip addr show` to check for this scenario instead of `ifconfig` because `ifconfig` will
+  not display the offending alias IP address. Alternatively an API endpoint specific to
+  DigitalOcean allows to query for the anchor IP from the droplet:
 
   ```sh
   curl http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address
@@ -270,12 +286,13 @@ Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc6
 
 ## `coredns` pods have `CrashLoopBackOff` or `Error` state
 
-If you have nodes that are running SELinux with an older version of Docker you might experience a scenario
-where the `coredns` pods are not starting. To solve that you can try one of the following options:
+If you have nodes that are running SELinux with an older version of Docker, you might experience a scenario
+where the `coredns` pods are not starting. To solve that, you can try one of the following options:
 
 - Upgrade to a [newer version of Docker](/docs/setup/production-environment/container-runtimes/#docker).
 
 - [Disable SELinux](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security-enhanced_linux/sect-security-enhanced_linux-enabling_and_disabling_selinux-disabling_selinux).
+
 - Modify the `coredns` deployment to set `allowPrivilegeEscalation` to `true`:
 
 ```bash
@@ -284,7 +301,8 @@ kubectl -n kube-system get deployment coredns -o yaml | \
   kubectl apply -f -
 ```
 
-Another cause for CoreDNS to have `CrashLoopBackOff` is when a CoreDNS Pod deployed in Kubernetes detects a loop. [A number of workarounds](https://github.com/coredns/coredns/tree/master/plugin/loop#troubleshooting-loops-in-kubernetes-clusters)
+Another cause for CoreDNS to have `CrashLoopBackOff` is when a CoreDNS Pod deployed in Kubernetes detects a loop.
+[A number of workarounds](https://github.com/coredns/coredns/tree/master/plugin/loop#troubleshooting-loops-in-kubernetes-clusters)
 are available to avoid Kubernetes trying to restart the CoreDNS Pod every time CoreDNS detects the loop and exits.
 
 {{< warning >}}
@@ -300,7 +318,7 @@ If you encounter the following error:
 rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""
 ```
 
-this issue appears if you run CentOS 7 with Docker 1.13.1.84.
+This issue appears if you run CentOS 7 with Docker 1.13.1.84.
 This version of Docker can prevent the kubelet from executing into the etcd container.
 
 To work around the issue, choose one of these options:
@@ -344,6 +362,7 @@ to pick up the node's IP address properly and has knock-on effects to the proxy
 load balancers.
 
 The following error can be seen in kube-proxy Pods:
+
 ```
 server.go:610] Failed to retrieve node IP: host IP unknown; known addresses: []
 proxier.go:340] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
@@ -352,8 +371,26 @@ proxier.go:340] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
 A known solution is to patch the kube-proxy DaemonSet to allow scheduling it on control-plane
 nodes regardless of their conditions, keeping it off of other nodes until their initial guarding
 conditions abate:
+
 ```
-kubectl -n kube-system patch ds kube-proxy -p='{ "spec": { "template": { "spec": { "tolerations": [ { "key": "CriticalAddonsOnly", "operator": "Exists" }, { "effect": "NoSchedule", "key": "node-role.kubernetes.io/control-plane" } ] } } } }'
+kubectl -n kube-system patch ds kube-proxy -p='{
+  "spec": {
+    "template": {
+      "spec": {
+        "tolerations": [
+          {
+            "key": "CriticalAddonsOnly",
+            "operator": "Exists"
+          },
+          {
+            "effect": "NoSchedule",
+            "key": "node-role.kubernetes.io/control-plane"
+          }
+        ]
+      }
+    }
+  }
+}'
 ```
 
 The tracking issue for this problem is [here](https://github.com/kubernetes/kubeadm/issues/1027).
@@ -365,12 +402,15 @@ For [flex-volume support](https://github.com/kubernetes/community/blob/ab55d85/c
 Kubernetes components like the kubelet and kube-controller-manager use the default path of
 `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/`, yet the flex-volume directory _must be writeable_
 for the feature to work.
-(**Note**: FlexVolume was deprecated in the Kubernetes v1.23 release)
 
-To workaround this issue you can configure the flex-volume directory using the kubeadm
+{{< note >}}
+FlexVolume was deprecated in the Kubernetes v1.23 release.
+{{< /note >}}
+
+To workaround this issue, you can configure the flex-volume directory using the kubeadm
 [configuration file](/docs/reference/config-api/kubeadm-config.v1beta3/).
 
-On the primary control-plane Node (created using `kubeadm init`) pass the following
+On the primary control-plane Node (created using `kubeadm init`), pass the following
 file using `--config`:
 
 ```yaml
@@ -402,7 +442,10 @@ be advised that this is modifying a design principle of the Linux distribution.
 
 ## `kubeadm upgrade plan` prints out `context deadline exceeded` error message
 
-This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with `kubeadm upgrade apply ...`.
+This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in
+the case of running an external etcd. This is not a critical bug and happens because
+older versions of kubeadm perform a version check on the external etcd cluster.
+You can proceed with `kubeadm upgrade apply ...`.
 
 This issue is fixed as of version 1.19.
 
@@ -422,6 +465,7 @@ can be used insecurely by passing the `--kubelet-insecure-tls` to it. This is no
 If you want to use TLS between the metrics-server and the kubelet there is a problem,
 since kubeadm deploys a self-signed serving certificate for the kubelet. This can cause the following errors
 on the side of the metrics-server:
+
 ```
 x509: certificate signed by unknown authority
 x509: certificate is valid for IP-foo not IP-bar
@@ -438,6 +482,7 @@ Only applicable to upgrading a control plane node with a kubeadm binary v1.28.3
 where the node is currently managed by kubeadm versions v1.28.0, v1.28.1 or v1.28.2.
 
 Here is the error message you may encounter:
+
 ```
 [upgrade/etcd] Failed to upgrade etcd: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: static Pod hash for component etcd on Node kinder-upgrade-control-plane-1 did not change after 5m0s: timed out waiting for the condition
 [upgrade/etcd] Waiting for previous etcd to become available
@@ -454,16 +499,19 @@ k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.performEtcdStaticPodUpgrade
 ...
 ```
 
-The reason for this failure is that the affected versions generate an etcd manifest file with unwanted defaults in the PodSpec.
-This will result in a diff from the manifest comparison, and kubeadm will expect a change in the Pod hash, but the kubelet will never update the hash.
+The reason for this failure is that the affected versions generate an etcd manifest file with
+unwanted defaults in the PodSpec. This will result in a diff from the manifest comparison,
+and kubeadm will expect a change in the Pod hash, but the kubelet will never update the hash.
 
 There are two way to workaround this issue if you see it in your cluster:
+
 - The etcd upgrade can be skipped between the affected versions and v1.28.3 (or later) by using:
-```shell
-kubeadm upgrade {apply|node} [version] --etcd-upgrade=false
-```
 
-This is not recommended in case a new etcd version was introduced by a later v1.28 patch version.
+  ```shell
+  kubeadm upgrade {apply|node} [version] --etcd-upgrade=false
+  ```
+
+  This is not recommended in case a new etcd version was introduced by a later v1.28 patch version.
 
 - Before upgrade, patch the manifest for the etcd static pod, to remove the problematic defaulted attributes:
 
@@ -509,4 +557,5 @@ This is not recommended in case a new etcd version was introduced by a later v1.
         path: /etc/kubernetes/pki/etcd
   ```
 
-More information can be found in the [tracking issue](https://github.com/kubernetes/kubeadm/issues/2927) for this bug.
+More information can be found in the
+[tracking issue](https://github.com/kubernetes/kubeadm/issues/2927) for this bug.