Connection refused errors for linkerd-proxy (Linkerd+Calico)

[PriyaSharma9]

Hi Team,
We are facing some issues while installing linkerd on our cluster (CNI - Calico) .
There are no deny policies, only a set of allow policies applied by Calico.
We have added the pod and service CIDRs to the linkerd deployments. (as we were seeing "Control Plane" related errors earlier)
These are the errors that we see in linkerd-proxy sidecar

kubectl logs -f linkerd-proxy-injector-85b46644cc-jgvjh -n linkerd linkerd-proxy
[     0.000907s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.002606s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.002675s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.002755s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.002793s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.002847s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.002904s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.002925s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.035196s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.646601s]  INFO ThreadId(01) inbound:server{port=9995}:rescue{client.addr=10.94.64.208:58621}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[    15.551782s]  WARN ThreadId(01) policy:watch{port=8443}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.30.75.90:8090}: linkerd_reconnect: Service failed error=channel closed
[    15.558348s]  WARN ThreadId(01) outbound:server{orig_dst=172.21.0.1:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=172.30.75.90:8086}: linkerd_reconnect: Service failed error=channel closed
[    15.663158s]  WARN ThreadId(01) outbound:server{orig_dst=172.21.0.1:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=172.30.75.90:8086}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    15.663209s]  WARN ThreadId(01) policy:watch{port=8443}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.30.75.90:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    15.875097s]  WARN ThreadId(01) policy:watch{port=8443}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.30.75.90:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    15.875295s]  WARN ThreadId(01) outbound:server{orig_dst=172.21.0.1:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=172.30.75.90:8086}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    16.291797s]  WARN ThreadId(01) outbound:server{orig_dst=172.21.0.1:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=172.30.75.90:8086}: 

We created a dnsutils pod as described here and the dns resolution seems to be working as expected for all the linkerd hostnames.

Could someone please help as we have been stuck for a while, thankyou!

[adleong]

Based on these logs, it looks like the proxy-injector is not able to connect to the destination controller pod. Is the destination controller up and healthy? Does linkerd check show any errors? Are there any errors in the logs of the destination controller?