Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=metrics)

[venudevops1458]

Hi Team,

one of Kafka node becomes unresponsive on Strimzi Kafka cluster which running on kubernetes cluster with below error on that specific node.

kubelet_node_status.go:402] Error updating node status, will retry: error getting node "node11.dc02.its.myorgcorp.net": an error on the server ("") has prevented the request from succeeding (get nodes node11.dc02.its.myorgcorp.net)
kubelet_node_status.go:389] Unable to update node status: update node status exceeds retry count
webhook.go:199] Failed to make webhook authorizer request: Post https://dev-k8s-api-server.its.myorgcorp.net:8443/apis/authorization.k8s.io/v1/subjectaccessreviews: write tcp 10.84.73.59:41992->10.84.120.106:8443: use of closed network connection
server.go:269] Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=metrics)%!(EXTRA *url.Error=Post https://dev-k8s-api-server.its.myorgcorp.net:8443/apis/authorization.k8s.io/v1/subjectaccessreviews: wr
webhook.go:199] Failed to make webhook authorizer request: Post https://dev-k8s-api-server.its.myorgcorp.net:8443/apis/authorization.k8s.io/v1/subjectaccessreviews: write tcp 10.84.73.59:41992->10.84.120.106:8443: use of closed network connection
server.go:269] Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=metrics)%!(EXTRA *url.Error=Post https://dev-k8s-api-server.its.myorgcorp.net:8443/apis/authorization.k8s.io/v1/subjectaccessreviews: wr
container_manager_linux.go:470] [ContainerManager]: Discovered runtime cgroups name: /systemd/system.slice
reflector.go:178] object-"kube-system"/"kube-proxy-token-f7mmc": Failed to list *v1.Secret: an error on the server ("") has prevented the request from succeeding (get secrets)
reflector.go:178] object-"kafka"/"dev-demoapp-cluster-ca-cert": Failed to list *v1.Secret: an error on the server ("") has prevented the request from succeeding (get secrets)
reflector.go:178] object-"kafka"/"dev-demoapp-kafka-token-25vzc": Failed to list *v1.Secret: an error on the server ("") has prevented the request from succeeding (get secrets)
webhook.go:199] Failed to make webhook authorizer request: Post https://dev-k8s-api-server.its.myorgcorp.net:8443/apis/authorization.k8s.io/v1/subjectaccessreviews: write tcp 10.84.73.59:41992->10.84.120.106:8443: use of closed network connection

We have 3zookeeper nodes(pods/kubernetes master nodes) and 8broker nodes(pods/kubernetes worker nodes). issue came on only one node.

It got fixed after reboot, but we would like to know root cause for this, so it can be prevented instead of fixing after happening.

Please let me know if anyone knows anything about this error.

[scholzj]

Sorry, but how is this related to Strimzi? This seems like some platform issue.