Inconsistently-collected metrics

[zack-littke-smith-ai]

Hi there! Newish Linkerd user here, so forgive me if these questions are common knowledge. When investigating the collected values of some core metrics, I became somewhat suspicious of them (I don't want to be suspicious of my metrics!). Let me be more specific:

• The tags collected are strange at times - I believe response_total and route_response_total collect status_code (which is strange because this is a gRPC service only) but only sometimes collect grpc_status_code, even though they seem to collect grpc_status
• Why does response_total collect 5XX connection errors while route_response_total does not?
• Tags are incorrect at times - why do some metrics report status_code: 200 when they also report classification: failure? Which of these should I trust?
• The naming of some tags is strange / arbitrary (route_actual_response_total vs route_response_total?)
• There is very little documentation describing these metrics:
  • This is all the documentation that exists for route_response_total
  • Googling for route_actual_response_total hilariously only has three results across the entire internet. What does "Total count of actual route HTTP responses." mean??

These issues are with a bread-and-butter metric that I'd really prefer to trust, and in light of these issues I am considering instrumenting our services instead of using linkerd's metrics. Am I being too rash?