Linkerd v2.11.1: Intermittent 502 error

[sumitrindhe]

I am running Linkerd v2.11.1 in a Kubernetes v1.18 cluster. I have Traefik service which receives incoming requests and forwards the requests to Service A. Both Traefik and Service A are part of the service mesh. For most requests to Traefik I receive 200 response but I also see small number of 502 response intermittently. For the 502 error I see the following in Traefik logs:

origin_L5d-Proxy-Error "Connection refused (os error 111)"
downstream_L5d-Proxy-Error "Connection refused (os error 111)"

I also see the following error in linkerd-proxy sitting next to Traefik.

[ 8064.142632s] INFO ThreadId(01) outbound:server{orig_dst=172.20.34.60:80}:rescue{client.addr=100.64.110.91:57376}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111) [ 8064.142669s] INFO ThreadId(01) outbound:server{orig_dst=172.20.34.60:80}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set

I do not see any error on Service A logs pr linkerd-proxy sitting next to Service A. Seems the request from Traefik does not reach Service A. Linkerd check command returns "Status check passed".

I saw some 502 issues posted on github but I doubt they are related to this issue.

[adleong]

Hi @sumitrindhe. It appears that the server running at 172.20.34.60:80 is intermittently refusing connections from Linkerd. This causes Linkerd to return a 502 in that case. I'd recommend trying to diagnose why the server running at 172.20.34.60:80 is refusing connections.

[sumitrindhe]

Thanks @adleong for your reply.

Actually I found the issue. Linkerd was running fine, issue was with my service configuration.

Traefik was forwarding the requests to Service A K8s LB. Behind the Service A K8s LB there were Service A pods running(as expected) and also Service B pods running(not expected) because of incorrect tagging on Service B pod configuration. So when requests were hitting the Service A pods then 200 response was seen and when requests were hitting the Service B pods then we got 502 response.