How do I see TCP load on routes?

[GabriFila]

Hello! 👋🏻
I'm working on a little project and I would like to view TCP load metrics (amount of data exchanged) between 2 microservices in a k8s cluster. For testing, I'm using the google-demo-app. I wanted to use Linkerd to find the data I need but I could not make it work.

Here's what I've tried/seen so far with the Linkerd dashboard:

• In the Pods panel I can see the amount of TCP data sent and received by a microservice. This doesn't help my use case because I need to know the data between 2 and not the data to/from 1
• In Routes panel I can see data between 2 microservices, but when I look into a route between 2 pods I can't see any TCP metrics, only HTTP.

Is there a way to see TCP metrics on a route? That would be perfect for my use case.

Thank you!

[mateiidavid]

Hey 👋

Think you can make use of the cli viz extension to do this. If you're running latest (2.10) you need to install linkerd and then linkerd-viz (linkerd viz install | kubectl (..)).

With viz, you can get tcp stats using the cli. Normally, running the command gives you inbound stats for the resource you select; if you use --from or --to you can get outbound stats on it too. For TCP load, you can use the -o wide flag to get bytes written/read.

Putting it all together: if we have a microservice foo that talks to bar we can:

• Get inbound stats for foo: linkerd viz stat deploy/bar -o wide
• Get outbound stats for foo: linkerd viz stat deploy/foo -o wide (note: works with namespaces, deploys, pods, etc).

ref:
https://linkerd.io/2.10/reference/cli/viz/
https://linkerd.io/2.10/reference/proxy-metrics/

[GabriFila]

Thank you very much for the fast reply!
Your answer is what I was looking for, thank you.

The command for my use-case is linkerd viz stat pod/frontend-86c5bd866b-nn58k --namespace test-google --to pod/adservice-f4b9d578f-rlvz6 --to-namespace test-google -o wide

I've seen that the linkerd-proxy /metrics endpoint provides inbound and outbound TCP bytes, but I can't figure out if can get the same filter (to & from) on the Prometheus metrics as the CLI approach.

Do you know if this is feasible or I need to do it myself with custom scripts?

Thank you!

[mateiidavid]

Yeah you should be able to. The CLI approach queries prometheus so you can use the same queries directly if you want.

If you set the log-level on the metrics api deployment (or pod) to debug you can see exactly what queries are being sent. If you send a couple of stat requests you'll see all of the queries in your logs so you can copy the template and then change it as you like and (I assume) query prometheus directly or make your own grafana dashboard. I haven't personally tried it but no reason why it shouldn't work :)

For example, something like this would get you inbound read bytes for the namespace linkerd:

sum(increase(tcp_read_bytes_total{direction=\"inbound\", namespace=\"linkerd\", peer=\"src\"}[1m])) by (namespace, deployment)"

[GabriFila]

Thank you very much!

I will look into it