Linkerd-proxy reporting HTTP logical service in fail-fast

[asheeshv]

Hello folks,

I've integrated linkerd in our microservices deployment, and am running into HTTP fail-fast errors, which only occur when I increase a certain load to my system My server application uses gRPC for communicating with the external clients and internal services.
When I start with 50 clients, it runs fine, but when I increase the number of connections to 50, linkerd-proxy starts reporting http logical in failfast.
However, I don't see any errors in my application service code, so I'm really puzzled as to what's going on.
Here are the linkerd-proxy logs from my api-gateway service, which shows Http logical in failfast even though the rest of the pods are still up and running.
I'm using v2.11.4 of linkerd. Really appreciate any inputs or pointers as we are closing in our development timeline and need to wrap up load-testing. It'd be unfortunate to not use linkerd if this turns out to be a bigger issue, so would appreciate any guidance here.

[   169.588015s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}: linkerd_detect: DetectResult protocol=Some(H2) elapsed=12.901µs
[   169.588040s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}: linkerd_proxy_http::server: Creating HTTP service version=H2
[   169.588061s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}: linkerd_proxy_http::server: Handling as HTTP version=H2
[   169.588246s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.588297s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.588334s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.588369s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.588389s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.588399s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.588423s]  INFO ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:rescue{client.addr=10.1.21.171:42514}: linkerd_app_core::errors::respond: Request failed error=HTTP Logical service in fail-fast
[   169.588436s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_core::errors::respond: Handling error on gRPC connection code=The service is currently unavailable
[   169.588463s]  INFO ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:rescue{client.addr=10.1.21.171:42514}: linkerd_app_core::errors::respond: Request failed error=HTTP Logical service in fail-fast
[   169.588475s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_core::errors::respond: Handling error on gRPC connection code=The service is currently unavailable
[   169.588492s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}: linkerd_proxy_http::server: The stack is tearing down the connection
[   169.588771s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.588848s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.588889s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.588929s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.588986s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.589024s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.589060s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.589087s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.21.175:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.21.171:42514
[   169.589116s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.589142s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.589162s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.589173s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.589186s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.589207s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.589219s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.589234s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80
[   169.589262s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.589276s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.589280s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.589284s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.589289s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.589293s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.589298s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.589303s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   169.589323s]  INFO ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:rescue{client.addr=10.1.21.171:42514}: linkerd_app_core::errors::respond: Request failed error=HTTP Logical service in fail-fast
[   169.589329s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_core::errors::respond: Handling error on gRPC connection code=The service is currently unavailable
[   169.589347s]  INFO ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}:rescue{client.addr=10.1.21.171:42514}: linkerd_app_core::errors::respond: Request failed error=HTTP Logical service in fail-fast
[   169.589351s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.21.171:42514}:server{port=80}:http{v=h2}: linkerd_app_core::errors::respond: Handling error on gRPC connection code=The service is currently unavailable

I also took logs out of the linkerd-proxy-injector and saw these

linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
linkerd-proxy-injector-748c6b6898-zn775 linkerd-proxy [    60.844509s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
linkerd-proxy-injector-748c6b6898-zn775 linkerd-proxy [    61.350751s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
linkerd-proxy-injector-748c6b6898-zn775 linkerd-proxy [    61.857294s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: INme="2022-07-28T15:22:35Z" level=info msg="injection patch generated for: pod/gameserver-app-wabc-registration-65869f864b-"

I see these failures for failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN.

What does this mean? Is this some sort of a configuration failure, where the proxy fails to come up.

[adleong]

Hi @asheeshv! To dig into the details of exactly why the proxy is doing failfast here, I'd recommend turning up the log level to trace https://linkerd.io/2.11/tasks/modifying-proxy-log-level/ and looking at the logs around when the proxy first starts reporting failfast. This should give some clue as to why failfast is getting triggered.

[asheeshv]

Thanks for the response, @adleong. Much appreciated! I turned up the log levels to debug and there's a lot logging statements. I will turn it down to trace then. Mulitple of my proxies reported fail-fast, but I'm guessing that since they are dependent, if one of them failed, the other ones fail as well becaue of a ripple effect. I'll post some data after turning the logs down to trace, and see if I can extract more info.

[asheeshv]

Hello @adleong, I turned the trace logs on one of the service proxies, and here's the data as follows:
[ 1613.847175s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: service request=Request { method: POST, uri: http://upstream_balancer/aristocrat_server_api_gateway.ClientApi/RequestGamePlay, version: HTTP/2.0, headers: {"x-request-id": "c3766451048f9267a7381de89ed138bc", "x-real-ip": "10.1.110.244", "x-forwarded-for": "10.1.110.244", "x-forwarded-host": "minik8-temp06", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "te": "trailers", "user-agent": "grpc-dotnet/2.38.0.0", "grpc-accept-encoding": "identity,gzip", "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlZ20iOiI0NCIsIm5iZiI6MTY1OTEyNTA4MCwiZXhwIjoyMTQ3NDcyMDAwLCJpYXQiOjE2NTkxMjUwODB9.1vY5K4-yMTBZ5Ts1hf6O7Dmu1K0xWk63STtmhAvjLwE", "content-type": "application/grpc", "l5d-dst-canonical": "gameserver-app-api-gateway.ati.svc.cluster.local:80", "l5d-client-id": "nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"}, body: BoxBody } [ 1613.847198s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: drop [ 1613.847210s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80 [ 1613.847215s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: new [ 1613.847220s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_cache: Using cached service [ 1613.847225s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: service ready=true ok=true [ 1613.847228s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: service request=Request { method: POST, uri: http://upstream_balancer/aristocrat_server_api_gateway.ClientApi/ReportGameOutcome, version: HTTP/2.0, headers: {"x-request-id": "cdb09520d73f1ced9a03fb34c98655b7", "x-real-ip": "10.1.110.244", "x-forwarded-for": "10.1.110.244", "x-forwarded-host": "minik8-temp06", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "te": "trailers", "user-agent": "grpc-dotnet/2.38.0.0", "grpc-accept-encoding": "identity,gzip", "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlZ20iOiIzMiIsIm5iZiI6MTY1OTEyNTA1NiwiZXhwIjoyMTQ3NDcyMDAwLCJpYXQiOjE2NTkxMjUwNTZ9.XgZWnOfRykQlvShVTz4gG5hUtDl3PeRq4ktaAgXsZ7k", "content-type": "application/grpc", "l5d-dst-canonical": "gameserver-app-api-gateway.ati.svc.cluster.local:80", "l5d-client-id": "nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"}, body: BoxBody } [ 1613.847244s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: drop [ 1613.847250s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80 [ 1613.847255s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: new [ 1613.847259s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_cache: Using cached service [ 1613.847266s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: service ready=true ok=true [ 1613.847270s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: service request=Request { method: POST, uri: http://upstream_balancer/aristocrat_server_api_gateway.ClientApi/ReportGameOutcome, version: HTTP/2.0, headers: {"x-request-id": "6010d79b59ba25968b781494895f3b33", "x-real-ip": "10.1.110.244", "x-forwarded-for": "10.1.110.244", "x-forwarded-host": "minik8-temp06", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "te": "trailers", "user-agent": "grpc-dotnet/2.38.0.0", "grpc-accept-encoding": "identity,gzip", "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlZ20iOiI0NCIsIm5iZiI6MTY1OTEyNTA4MCwiZXhwIjoyMTQ3NDcyMDAwLCJpYXQiOjE2NTkxMjUwODB9.1vY5K4-yMTBZ5Ts1hf6O7Dmu1K0xWk63STtmhAvjLwE", "content-type": "application/grpc", "l5d-dst-canonical": "gameserver-app-api-gateway.ati.svc.cluster.local:80", "l5d-client-id": "nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"}, body: BoxBody } [ 1613.847288s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: drop [ 1613.847296s] TRACE ThreadId(01) evict{target=Logical { logical: Some(NameAddr { name: Name("gameserver-app-api-gateway.ati.svc.cluster.local"), port: 80 }), addr: Remote(ServerAddr(10.1.110.228:80)), http: H2, tls: Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }), permit: Permit { dst: OrigDstAddr(10.1.110.228:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } }, labels: {"saz_name": "default:all-unauthenticated", "srv_name": "default:all-unauthenticated"} }}: linkerd_cache: Reset [ 1613.847301s] TRACE ThreadId(01) evict{target=Logical { logical: Some(NameAddr { name: Name("gameserver-app-api-gateway.ati.svc.cluster.local"), port: 80 }), addr: Remote(ServerAddr(10.1.110.228:80)), http: H2, tls: Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }), permit: Permit { dst: OrigDstAddr(10.1.110.228:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } }, labels: {"saz_name": "default:all-unauthenticated", "srv_name": "default:all-unauthenticated"} }}: linkerd_cache: Reset [ 1613.853594s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_stack_tracing: service ready=true ok=true [ 1613.853706s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_stack_tracing: service ready=true ok=true [ 1613.853759s] TRACE ThreadId(01) evict{target=Logical { logical: Some(NameAddr { name: Name("gameserver-app-api-gateway.ati.svc.cluster.local"), port: 80 }), addr: Remote(ServerAddr(10.1.110.228:80)), http: H2, tls: Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }), permit: Permit { dst: OrigDstAddr(10.1.110.228:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } }, labels: {"saz_name": "default:all-unauthenticated", "srv_name": "default:all-unauthenticated"} }}: linkerd_cache: Reset [ 1613.889446s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_stack_tracing: service ready=true ok=true [ 1613.893745s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_stack_tracing: service ready=true ok=true [ 1613.893871s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_stack_tracing: service request=Request { method: POST, uri: http://upstream_balancer/aristocrat_server_api_gateway.ClientApi/RequestGamePlay, version: HTTP/2.0, headers: {"x-request-id": "c27b504697c5d56b7adc052515bc5f26", "x-real-ip": "10.1.110.244", "x-forwarded-for": "10.1.110.244", "x-forwarded-host": "minik8-temp06", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "te": "trailers", "user-agent": "grpc-dotnet/2.38.0.0", "grpc-accept-encoding": "identity,gzip", "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlZ20iOiIzMiIsIm5iZiI6MTY1OTEyNTA1NiwiZXhwIjoyMTQ3NDcyMDAwLCJpYXQiOjE2NTkxMjUwNTZ9.XgZWnOfRykQlvShVTz4gG5hUtDl3PeRq4ktaAgXsZ7k", "content-type": "application/grpc", "l5d-dst-canonical": "gameserver-app-api-gateway.ati.svc.cluster.local:80"}, body: UpgradeBody { body: Body(Streaming), upgrade: None } } [ 1613.893905s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::set_identity_header: Setting identity header header=l5d-client-id id="nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local" [ 1613.893914s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Authorizing request policy=AllowPolicy { dst: OrigDstAddr(10.1.110.228:80), server: Receiver { shared: Shared { value: RwLock(PhantomData, RwLock { data: ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" } }), state: AtomicState(2), ref_count_rx: 11, notify_rx: Notify { state: 4, waiters: Mutex(PhantomData, Mutex { data: LinkedList { head: None, tail: None } }) }, notify_tx: Notify { state: 1, waiters: Mutex(PhantomData, Mutex { data: LinkedList { head: Some(0x7f1cc82b8d78), tail: Some(0x7f1cc82b8d78) } }) } }, version: Version(0) } } [ 1613.893944s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.1.110.228:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }) client=10.1.110.244:42172 [ 1613.893980s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_stack_tracing: service ready=true ok=true [ 1613.893997s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}: linkerd_app_inbound::http::router: using l5d-dst-canonical addr=gameserver-app-api-gateway.ati.svc.cluster.local:80 [ 1613.894007s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: new [ 1613.894018s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_cache: Using cached service [ 1613.894032s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: service ready=true ok=true [ 1613.894037s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: service request=Request { method: POST, uri: http://upstream_balancer/aristocrat_server_api_gateway.ClientApi/RequestGamePlay, version: HTTP/2.0, headers: {"x-request-id": "c27b504697c5d56b7adc052515bc5f26", "x-real-ip": "10.1.110.244", "x-forwarded-for": "10.1.110.244", "x-forwarded-host": "minik8-temp06", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "te": "trailers", "user-agent": "grpc-dotnet/2.38.0.0", "grpc-accept-encoding": "identity,gzip", "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlZ20iOiIzMiIsIm5iZiI6MTY1OTEyNTA1NiwiZXhwIjoyMTQ3NDcyMDAwLCJpYXQiOjE2NTkxMjUwNTZ9.XgZWnOfRykQlvShVTz4gG5hUtDl3PeRq4ktaAgXsZ7k", "content-type": "application/grpc", "l5d-dst-canonical": "gameserver-app-api-gateway.ati.svc.cluster.local:80", "l5d-client-id": "nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"}, body: BoxBody } [ 1613.894062s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack_tracing: drop [ 1613.894075s] TRACE ThreadId(01) evict{target=Logical { logical: Some(NameAddr { name: Name("gameserver-app-api-gateway.ati.svc.cluster.local"), port: 80 }), addr: Remote(ServerAddr(10.1.110.228:80)), http: H2, tls: Some(Established { client_id: Some(ClientId(Name("nginx-ingress-microk8s-serviceaccount.ingress.serviceaccount.identity.linkerd.cluster.local"))), negotiated_protocol: None }), permit: Permit { dst: OrigDstAddr(10.1.110.228:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } }, labels: {"saz_name": "default:all-unauthenticated", "srv_name": "default:all-unauthenticated"} }}: linkerd_cache: Reset [ 1613.989449s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989499s] WARN ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical entering failfast after 1s [ 1613.989506s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989531s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989537s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989545s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989550s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989557s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989562s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989569s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989574s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989582s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989587s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989594s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989599s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989606s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989613s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989620s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989631s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989639s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989644s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast [ 1613.989651s] TRACE ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::switch_ready: SwitchReady::poll state=Secondary [ 1613.989655s] DEBUG ThreadId(01) inbound:accept{client.addr=10.1.110.244:42172}:server{port=80}:http{v=h2}:http2{name=gameserver-app-api-gateway.ati.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical in failfast
Do let me know if you need more info.
Many thanks for looking into this.
Asheesh

[asheeshv]

I do see these errors in my service container now:
gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation System.Net.Http.HttpRequestException: An error occurred while sending the request. gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation ---> System.IO.IOException: The request was aborted. gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation ---> System.Net.Http.Http2StreamException: The HTTP/2 server reset the stream. HTTP/2 error code 'NO_ERROR' (0x0). gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation --- End of inner exception stack trace --- gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation --- End of inner exception stack trace --- gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) gameserver-app-evaluation-68f45d9b9b-n9h7x evaluation at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)

[asheeshv]

Hello @adleong, Any inputs on this. I'd appreciate your help!

[asheeshv]

@adleong Here are some of the logs I see after turning debug logs in my proxy:

[   408.645883s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60676}:server{port=80}: linkerd_detect: DetectResult protocol=Some(H2) elapsed=11.7┬╡s
[   408.645913s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60676}:server{port=80}: linkerd_proxy_http::server: Creating HTTP service version=H2
[   408.645976s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60676}:server{port=80}: linkerd_proxy_http::server: Handling as HTTP version=H2
[   408.657066s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60676}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.244.0.86:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=None(NoClientHello) client=10.244.0.41:60676
[   408.680980s] DEBUG ThreadId(01) logical{dst=gameserver-app-management.ati.svc.cluster.local:80}:concrete{addr=gameserver-app-management.ati.svc.cluster.local:80}:endpoint{server.addr=10.244.0.95:80}:h2: linkerd_proxy_http::client: method=POST uri=http://gameserver-app-management/aristocrat_server_api_gateway.InternalApi/UpdateMeters version=HTTP/2.0
[   408.686357s] DEBUG ThreadId(01) logical{dst=gameserver-app-management.ati.svc.cluster.local:80}:concrete{addr=gameserver-app-management.ati.svc.cluster.local:80}:endpoint{server.addr=10.244.0.95:80}:h2: linkerd_proxy_http::client: headers={"user-agent": "grpc-dotnet/2.44.0 (.NET 6.0.7; CLR 6.0.7; net6.0; linux; x64)", "te": "trailers", "grpc-accept-encoding": "identity,gzip,deflate", "traceparent": "00-9b621ffed5b7410c96ddcdc8b21c3bca-d26cef33d029b5d5-01", "content-type": "application/grpc", "l5d-dst-canonical": "gameserver-app-management.ati.svc.cluster.local:80"}
[   408.703127s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60684}:server{port=80}: linkerd_app_inbound::accept: Accepted policy=AllowPolicy { dst: OrigDstAddr(10.244.0.86:80), server: Receiver { shared: Shared { value: RwLock(PhantomData, RwLock { data: ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" } }), state: AtomicState(2), ref_count_rx: 3952, notify_rx: Notify { state: 4, waiters: Mutex(PhantomData, Mutex { data: LinkedList { head: None, tail: None } }) }, notify_tx: Notify { state: 1, waiters: Mutex(PhantomData, Mutex { data: LinkedList { head: Some(0x7fb4628b8d78), tail: Some(0x7fb4628b8d78) } }) } }, version: Version(0) } }
[   408.703253s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60684}:server{port=80}: linkerd_tls::server: Peeked bytes from TCP stream sz=512
[   408.703278s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60684}:server{port=80}: linkerd_detect: DetectResult protocol=Some(H2) elapsed=10.6┬╡s
[   408.703288s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60684}:server{port=80}: linkerd_proxy_http::server: Creating HTTP service version=H2
[   408.703304s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60684}:server{port=80}: linkerd_proxy_http::server: Handling as HTTP version=H2
[   408.703514s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:60684}:server{port=80}:http{v=h2}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(10.244.0.86:80), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=None(NoClientHello) client=10.244.0.41:60684
[   408.901961s]  WARN ThreadId(01) accept{client.addr=10.244.0.41:59574}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical entering failfast after 1s
[   408.902158s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:59574}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.902350s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:58062}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.902555s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:57778}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.902796s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:58992}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.902980s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:59614}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.903141s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:59180}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.903737s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:59142}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.903880s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:56882}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.903989s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:59368}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.904748s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:58682}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.904910s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:58978}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.905126s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:56784}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.905244s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:58674}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast
[   408.905422s] DEBUG ThreadId(01) accept{client.addr=10.244.0.41:59656}:server{port=80}:http{v=h2}:http2{name=upstream_balancer:80}: linkerd_stack::failfast: HTTP Logical in failfast

My question is that is there a way I can determine what is the proxy that's getting disconnected and which service container is it associated with? I don't see any fail-fast errors in any of the other proxies my api-gateway is connected to. Shouldn't I be seeing errors there as well?