linkerd-viz check failing

[samueleydaniel]

Environment: AWS EKS with Cilium CNI
Problem: linkerd viz check failing. All the pods are running. Below are some outputs from the cluster. The pods in the linkerd-viz namespaces are not injected.
Version: 2.11.1

kgp -n linkerd

$ kgp -n linkerd-viz

$ linkerd viz check

linkerd-viz

√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
‼ tap API service is running
FailedDiscoveryCheck: failing or missing response from https://10.0.1.123:8089/apis/tap.linkerd.io/v1alpha1: Get "https://10.0.1.123:8089/apis/tap.linkerd.io/v1alpha1": Address is not allowed
see https://linkerd.io/2.11/checks/#l5d-tap-api for hints
‼ linkerd-viz pods are injected
could not find proxy container for grafana-fd54d9bdd-dh7l5 pod
see https://linkerd.io/2.11/checks/#l5d-viz-pods-injection for hints
√ viz extension pods are running
× viz extension proxies are healthy
No "linkerd-proxy" containers found in the "linkerd" namespace
see https://linkerd.io/2.11/checks/#l5d-viz-proxy-healthy for hints

Status check results are ×

I also see these kind of errors in all the pods in the linkerd namespace
$kl linkerd-proxy-injector-54d699c44d-hc7j8 -n linkerd -c linkerd-proxy
[ 0.001389s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime [ 0.002036s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191 [ 0.002049s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143 [ 0.002052s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140 [ 0.002055s] INFO ThreadId(01) linkerd2_proxy: Tap DISABLED [ 0.002058s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local [ 0.002075s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local) [ 0.002078s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local) [ 0.006889s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.0.4.35:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) [ 0.114907s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.0.4.35:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) [ 0.333759s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.0.4.35:8080}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111) [ 1.005874s] WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.0.1.159:8090}: linkerd_reconnect: Failed to connect error=connect timed out after 1s [ 1.244498s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local [ 1.262704s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=non-HTTP connection from 127.0.0.1:33334 [ 2.113184s] WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.0.1.159:8090}: linkerd_reconnect: Failed to connect error=connect timed out after 1s [ 2.468805s] WARN ThreadId(01) outbound:server{orig_dst=172.20.0.1:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.0.1.159:8086}: linkerd_reconnect: Failed to connect error=connect timed out after 1s