Does response_total double count?

[alexklibisz]

I'm testing out some linkerd metrics for a gRPC service. I noticed that when I intentionally run a large number of failing requests (and no successful requests), the response_total metric shows the same number of requests with grpc_status="0" as grpc_status!="0".

Specifically, if I submit a large number of failing requests, the following two prometheus metrics evaluate to the same value.

sum without(pod, instance) (increase(response_total{direction="inbound", authority="my.service.svc.cluster.local", grpc_status="0"}[10m]))

sum without(pod, instance) (increase(response_total{direction="inbound", authority="my.service.svc.cluster.local", grpc_status!="0", grpc_status!=""}[10m]))

It seems to work as expected if I use the route_response_total metric. I can do that for this case, because I have a service profile, but not all of our services have service profiles.

Is that intentional/by design? It seems like it might be counting some sort of hand-shake request for each of the failing requests?

[olix0r]

@alexklibisz I'm not aware of any reason that would cause responses to be double-counted...

You can spot-check a deploy with:

:; linkerd metrics -n $ns deploy/$name | grep -e ^response_total -e ^route_response_total

Do the values diverge? Do you see anything else unexpected?

[alexklibisz]

Here are the metrics when the service has been totally idle for a while:

response_total{direction="inbound",tls="no_identity",no_tls_reason="no_tls_from_remote",status_code="200",classification="success"} 989
route_response_total{direction="inbound",dst="dst1:7654",status_code="200",classification="success"} 950
route_response_total{direction="inbound",dst="dst1:9095",status_code="200",classification="success"} 39
response_total{direction="inbound",tls="no_identity",no_tls_reason="no_tls_from_remote",classification="failure",error="unclassified"} 1
response_total{direction="inbound",tls="no_identity",no_tls_reason="no_tls_from_remote",status_code="200",classification="success"} 1700
route_response_total{direction="inbound",dst="dst2:7654",classification="failure",error="unclassified"} 1
route_response_total{direction="inbound",dst="dst2:7654",status_code="200",classification="success"} 1632
route_response_total{direction="inbound",dst="dst2:9095",status_code="200",classification="success"} 68

Here it is immediately after submitting 1000 bad requests:

response_total{direction="inbound",tls="no_identity",no_tls_reason="no_tls_from_remote",status_code="200",classification="success"} 1075
response_total{authority="mygrpcsvc:50051",direction="inbound",tls="true",client_id="default.other-namespace.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success",grpc_status="0"} 388
response_total{authority="mygrpcsvc:50051",direction="inbound",tls="true",client_id="default.other-namespace.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success",grpc_status="3"} 422
route_response_total{direction="inbound",dst="IP1:7654",status_code="200",classification="success"} 1032
route_response_total{direction="inbound",dst="IP1:9095",status_code="200",classification="success"} 43
route_response_total{direction="inbound",dst="mygrpcsvc:50051",status_code="200",classification="success",grpc_status="0"} 388
route_response_total{direction="inbound",dst="mygrpcsvc:50051",rt_route="MyGrpcRoute",status_code="200",classification="success",grpc_status="3"} 422
response_total{direction="inbound",tls="no_identity",no_tls_reason="no_tls_from_remote",classification="failure",error="unclassified"} 1
response_total{direction="inbound",tls="no_identity",no_tls_reason="no_tls_from_remote",status_code="200",classification="success"} 1787
response_total{authority="mygrpcsvc:50051",direction="inbound",tls="true",client_id="default.other-namespace.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success",grpc_status="3"} 579
response_total{authority="mygrpcsvc:50051",direction="inbound",tls="true",client_id="default.other-namespace.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success",grpc_status="0"} 613
route_response_total{direction="inbound",dst="IP2:7654",classification="failure",error="unclassified"} 1
route_response_total{direction="inbound",dst="IP2:7654",status_code="200",classification="success"} 1715
route_response_total{direction="inbound",dst="IP2:9095",status_code="200",classification="success"} 72
route_response_total{direction="inbound",dst="mygrpcsvc:50051",status_code="200",classification="success",grpc_status="0"} 613
route_response_total{direction="inbound",dst="mygrpcsvc:50051",rt_route="MyGrpcRoute",status_code="200",classification="success",grpc_status="3"} 579

I anonymized the svc name, IPs, and routes.

[olix0r]

That looks coherent to me:

response_total{authority="mygrpcsvc:50051",direction="inbound",tls="true",client_id="default.other-namespace.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success",grpc_status="3"} 422
route_response_total{direction="inbound",dst="mygrpcsvc:50051",rt_route="MyGrpcRoute",status_code="200",classification="success",grpc_status="3"} 422

response_total{authority="mygrpcsvc:50051",direction="inbound",tls="true",client_id="default.other-namespace.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success",grpc_status="3"} 579
route_response_total{direction="inbound",dst="mygrpcsvc:50051",rt_route="MyGrpcRoute",status_code="200",classification="success",grpc_status="3"} 579

The response totals match the route response totals...

Perhaps your prometheus query is aggregating in some unintentional way?

[alexklibisz]

That's possible. I'll try to reproduce on monday and post back with the exact prom queries.