Linkerd connection issues

[prasus]

Greetings,

We have started using linkerd (version stable-2.9.0, deployed using helm) a few weeks back on our self hosted k8s cluster (v1.18.8) and have a few of our deployments injected with linkerd proxy. I recently noticed intermittent connection leaks/failures in our apps and while investigating we found the following log entries in the linkerd-proxy container of the affected services.

"Dec 2, 2020 @ 06:45:51.861","[    22.103150s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.33.169:49658 target.addr=10.244.59.53:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"
"Dec 2, 2020 @ 06:45:51.861","[    22.103162s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.33.169:49874 target.addr=10.244.59.53:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"
"Dec 2, 2020 @ 06:45:51.860","[    22.103116s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.33.169:49654 target.addr=10.244.59.53:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"
"Dec 2, 2020 @ 06:45:51.860","[    22.103127s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.33.169:49656 target.addr=10.244.59.53:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"
"Dec 2, 2020 @ 06:45:51.859","[    22.103104s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.33.169:49652 target.addr=10.244.59.53:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"
"Dec 2, 2020 @ 06:45:51.859","[    22.103089s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.33.169:49740 target.addr=10.244.59.53:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"
"Dec 2, 2020 @ 06:45:51.858","[    10.069981s]  WARN ThreadId(01) inbound:accept{peer.addr=10.244.33.105:37338 target.addr=10.244.33.169:80}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)"
"Dec 2, 2020 @ 06:45:51.858","[    11.203328s]  WARN ThreadId(01) inbound:accept{peer.addr=10.244.33.1:46326 target.addr=10.244.33.169:80}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)"
"Dec 2, 2020 @ 06:45:51.858","[    22.103038s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.33.169:49660 target.addr=10.244.59.53:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"
"Dec 2, 2020 @ 06:45:51.857","[     0.894634s]  INFO ThreadId(02) daemon:identity: linkerd2_app: Certified identity: default.apps-ns.serviceaccount.identity.linkerd.cluster.local"
"Dec 2, 2020 @ 06:45:51.857","[     1.202936s]  WARN ThreadId(01) inbound:accept{peer.addr=10.244.33.1:45558 target.addr=10.244.33.169:80}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)"
"Dec 2, 2020 @ 06:45:51.856","[     0.002700s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)"
"Dec 2, 2020 @ 06:45:51.856","[     0.003025s]  INFO ThreadId(01) outbound: linkerd2_app: listen.addr=127.0.0.1:4140 ingress_mode=false"
"Dec 2, 2020 @ 06:45:51.856","[     0.003094s]  INFO ThreadId(01) inbound: linkerd2_app: listen.addr=0.0.0.0:4143"
"Dec 2, 2020 @ 06:45:51.855","[     0.002687s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.apps-ns.serviceaccount.identity.linkerd.cluster.local"
"Dec 2, 2020 @ 06:45:51.855","[     0.002696s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)"
"Dec 2, 2020 @ 06:45:51.854","[     0.002678s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143"
"Dec 2, 2020 @ 06:45:51.854","[     0.002682s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140"
"Dec 2, 2020 @ 06:45:51.854","[     0.002685s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190"
"Dec 2, 2020 @ 06:45:51.853","[     0.001602s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime"
"Dec 2, 2020 @ 06:45:51.853","[     0.002665s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191"
"Dec 2, 2020 @ 06:45:51.851","time=""2020-12-02T06:45:22Z"" level=info msg=""running version stable-2.9.0"""
"Dec 2, 2020 @ 06:45:47.043","[ 80905.816738s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.90.101:39120 target.addr=10.244.57.132:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"
"Dec 2, 2020 @ 06:45:47.042","[ 80905.241029s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.90.101:34980 target.addr=10.244.54.131:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"
"Dec 2, 2020 @ 06:45:47.042","[ 80905.314951s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.90.101:38930 target.addr=10.244.57.132:9092}: linkerd2_app_core::serve: Connection closed error=Service in fail-fast"

Also there is os error 104/107 lines in linkerd-proxy logs for most of the injected services as seen below

[  4784.452349s]  INFO ThreadId(01) inbound:accept{peer.addr=10.244.44.107:44614 target.addr=10.244.44.106:80}: linkerd2_app_core::serve: Connection closed error=connection error: Transport endpoint is not connected (os error 107)
[  5089.706325s]  INFO ThreadId(01) inbound:accept{peer.addr=10.244.44.107:44636 target.addr=10.244.44.106:80}: linkerd2_app_core::serve: Connection closed error=connection error: Transport endpoint is not connected (os error 107)
[  5124.234815s]  INFO ThreadId(01) inbound:accept{peer.addr=10.244.44.107:46202 target.addr=10.244.44.106:80}: linkerd2_app_core::serve: Connection closed error=connection error: Transport endpoint is not connected (os error 107)
[  5318.467170s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:60420 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5329.525274s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:59506 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5331.837249s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:58566 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5334.995788s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:35018 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5340.005670s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:48375 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5345.637845s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:36610 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5351.125524s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:34292 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5351.451279s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:48372 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5364.015738s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:39484 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5365.568555s]  INFO ThreadId(01) outbound:accept{peer.addr=10.244.44.106:37052 target.addr=10.97.182.190:5432}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
[  5410.235235s]  INFO ThreadId(01) inbound:accept{peer.addr=10.244.44.107:35196 target.addr=10.244.44.106:80}: linkerd2_app_core::serve: Connection closed error=connection error: Transport endpoint is not connected (os error 107)
[  5633.826560s]  INFO ThreadId(01) inbound:accept{peer.addr=10.244.44.107:45720 target.addr=10.244.44.106:80}: linkerd2_app_core::serve: Connection closed error=connection error: Transport endpoint is not connected (os error 107)
[  5700.137574s]  INFO ThreadId(01) inbound:accept{peer.addr=10.244.44.107:52420 target.addr=10.244.44.106:80}: linkerd2_app_core::serve: Connection closed error=connection error: Transport endpoint is not connected (os error 107)

I am not sure what is causing this, so I would really appreciate if someone could share details on fixing this. Thanks!

[adleong]

Hey @prasus! Are your pods sending traffic to a headless service? If so, this sounds like it could be #5209 which was a bug in stable-2.9.0. This has since been fixed and you can try the latest edge release to see if it resolves the problem for you. This fix will be included in the stable-2.9.1 patch release.

[prasus]

Thanks @adleong! We seems to have these messages for non-headless service targets as well.
I have updated #5331 with details, We'll continue with that. Thanks 🙏🏿

[adleong]

Great! I'll look for updates on #5331 once you have a chance to test stable-2.9.1