Linkerd failing to proxy outgoing gRPC traffic to a headless service

[vijaygos]

• 'Linkerd check' looks great
• All linkerd pods are running fine with no errors
• My application pod is running fine
• No issues with 'kubectl api-resources'
• K8s version 1.14 and Linkerd version 2.7.1

[ 3200.246474462s] INFO outbound:accept{peer.addr=10.240.5.148:48840}:source{target.addr=10.240.5.95:8001}:logical{addr=10-240-5-95.my-service.my-namespace.svc.cluster.local:8001}:profile:balance{addr=10-240-5-95.my-service.my-namespace.svc.cluster.local:8001}: linkerd2_proxy_api_resolve::resolve: No endpoints
[ 3203.241936032s] WARN outbound:accept{peer.addr=10.240.5.148:48840}:source{target.addr=10.240.5.95:8001}: linkerd2_app_core::errors: Failed to proxy request: request timed out
[ 3203.298673231s] WARN outbound:accept{peer.addr=10.240.5.148:48840}:source{target.addr=10.240.5.95:8001}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast

Any suggestions on how to debug this further?

This is my setup without linkerd:

gRPC client -> Custom Loadbalancer client sidecar ----> Custom Loadbalancer server sidecar-> backend gRPC pods

My setup with linkerd:

[ gRPC client -> Custom Loadbalancer client sidecar -> Linkerd proxy ] ----> [ Linkerd proxy -> Custom Loadbalancer server sidecar -> backend gRPC pods ]

where [ ............................ ] represents a single pod.

The logs above are from the Linkerd proxy running with the gRPC client. If I take this proxy out, my connections seem fine.

[grampelberg]

Did you add the headers to your ingress? Why is the source and dst the same?

[vijaygos]

I realized I copy pasted the wrong logs. Please find the correct ones above.
I am not sure I understand the need for headers to my ingress here. I am not using any custom headers here and I don't really see why ingress is a problem.

[grampelberg]

See using ingress.

[vijaygos]

@grampelberg , I realized that I had described my scenario incorrectly. Sorry for dragging you down the wrong path with my ineptness at explaining things better. I fixed the scenario above to indicate that there is absolutely NO ingress involved. The linkerd proxies are meant to talk to one another directly. Any idea why the connection wouldn't go through?

gRPC client -> Custom Loadbalancer client sidecar -> Linkerd proxy ----> Linkerd proxy -> Custom Loadbalancer server sidecar -> backend gRPC pods