Missing TCP metrics for opaque ports

[errb]

I've recently set up a test cluster where I wanted to see if Linkerd can proxy mysql connections and provide both the metrics and mTLS for that traffic. However, thus far I've not been able to capture the metrics nor verify that the traffic is encrypted.

The docs clearly state that this can be done with mysql as long as the port is recognized as opaque:

Opaque ports are generally preferred as they allow Linkerd to provide mTLS, TCP-level metrics, policy, etc. Skip ports circumvent Linkerd’s ability to provide security features.

Within my test cluster, I have the following deployed:

• replicated mysql (using default 3306 port)
• a testservice that queries mysql primary at a 200ms interval
• linkerd-viz for observability

Both mysql and testservice are meshed. I have verbose logging enabled on linkerd-proxy in testservice pod and I can see that outbound tcp traffic going to mysql instances is correctly identified as opaque:

[  4706.834543s] DEBUG ThreadId(01) outbound:accept{client.addr=10.96.1.45:46920}:server{orig_dst=10.100.13.36:3306}:profile:tcp:endpoint{server.addr=10.96.0.56:3306 server.id=testapp-userdata-db-mysql.testapp.serviceaccount.identity.linkerd.cluster.local}: linkerd_app_outbound::tcp::opaque_transport: Using opaque transport target_port=3306 opaque_port=4143
[  4706.834575s] DEBUG ThreadId(01) outbound:accept{client.addr=10.96.1.45:46920}:server{orig_dst=10.100.13.36:3306}:profile:tcp:endpoint{server.addr=10.96.0.56:3306 server.id=testapp-userdata-db-mysql.testapp.serviceaccount.identity.linkerd.cluster.local}: linkerd_tls::client: Initiating TLS connection server.id=testapp-userdata-db-mysql.testapp.serviceaccount.identity.linkerd.cluster.local
[  4706.834581s] DEBUG ThreadId(01) outbound:accept{client.addr=10.96.1.45:46920}:server{orig_dst=10.100.13.36:3306}:profile:tcp:endpoint{server.addr=10.96.0.56:3306 server.id=testapp-userdata-db-mysql.testapp.serviceaccount.identity.linkerd.cluster.local}: linkerd_proxy_transport::connect: Connecting server.addr=10.96.0.56:4143
[  4706.834936s] DEBUG ThreadId(01) outbound:accept{client.addr=10.96.1.45:46920}:server{orig_dst=10.100.13.36:3306}:profile:tcp:endpoint{server.addr=10.96.0.56:3306 server.id=testapp-userdata-db-mysql.testapp.serviceaccount.identity.linkerd.cluster.local}: linkerd_proxy_transport::connect: Connected local.addr=10.96.1.45:41364 keepalive=Some(10s)
[  4706.836976s] DEBUG ThreadId(01) outbound:accept{client.addr=10.96.1.45:46920}:server{orig_dst=10.100.13.36:3306}:profile:tcp:endpoint{server.addr=10.96.0.56:3306 server.id=testapp-userdata-db-mysql.testapp.serviceaccount.identity.linkerd.cluster.local}: linkerd_tls::client: alpn=Ok("transport.l5d.io/v1")
[  4706.837051s] DEBUG ThreadId(01) outbound:accept{client.addr=10.96.1.45:46920}:server{orig_dst=10.100.13.36:3306}:profile:tcp:endpoint{server.addr=10.96.0.56:3306 server.id=testapp-userdata-db-mysql.testapp.serviceaccount.identity.linkerd.cluster.local}: linkerd_app_outbound::tcp::opaque_transport: Wrote transport header sz=30
[  4706.837305s] DEBUG ThreadId(01) outbound:accept{client.addr=10.96.1.45:46920}:server{orig_dst=10.100.13.36:3306}:profile:tcp:endpoint{server.addr=10.96.0.56:3306 server.id=testapp-userdata-db-mysql.testapp.serviceaccount.identity.linkerd.cluster.local}: linkerd_transport_metrics::client: client connection open
[  4706.848876s] DEBUG ThreadId(01) outbound:accept{client.addr=10.96.1.45:46920}: linkerd_app_core::serve: Connection closed

However, Linkerd dashboard doesn't show this traffic anywhere. Neither the linkerd-viz itself nor grafana dashboards indicate that testservice even connects to mysql pods. If I pause/resume queries, dashboards don't show any observable change in traffic on either end.

Note: the two inbound pods are tap and prometheus

What could I be missing here? Is it normal for opaque port traffic to go through the proxy and not be picked up by metric collectors?

Using version 2.11

[olix0r]

I'd probably start by using linkerd diagnostics proxy-metrics ... | grep ^tcp to see what metrics are being exported by the proxy. It's possible that it's just a misconfiguration in the Grafana dashboard.