protocol detection timeout despite opaque port annotation

[dwilliams782]

Hi all,

We've been working through this on Slack with @mateiidavid here but also raising a discussion for more input.

I'm seeing instances of linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s despite having the opaque port annotation on both the service and pod.

Linkerd: edge-22.6.2
GKE: v1.21.11-gke.1100

We have Grafana talking to Thanos Query Frontend, via a NodePort service. The NodePort port isn't actually used, instead it uses port 9090. The target service and target deployment are both annotated with config.linkerd.io/opaque-ports: "9090":

99.99% of the traffic works fine, but we are seeing very sporadic timeouts from the Grafana linkerd-proxy container:

Where the IP 10.224.44.90 is the the ClusterIP of the thanos-query-frontend service. We run a single replica of each service, and the timeout logs do not correlate with application/pod start up times etc. They are, as far as I can tell, "random".

Each time one of these logs is emitted, we get a corresponding log on the inbound proxy that suggests the requests are opaque:

however the logs on the outbound (where the timestamps also correspond to the protocol detection log timestamps) look like the service:port is not opaque, with opaque_protocol: false:

It looks like the service is not being detected as opaque, however we've tested this using the destination script and get back an opaque port:

It might not be relevant, but in almost all cases of this protocol detection timeout log being emitted, 10 seconds prior, we see logs around the service becoming unavailable:

[419972.169692s] DEBUG ThreadId(01) outbound:accept{client.addr=10.225.20.32:35952}:proxy{addr=10.224.44.90:9090}:http{v=1.x}:logical{dst=thanos-query-frontend.monitoring.svc.cluster.local:9090}:concrete{addr=thanos-query-frontend.monitoring.svc.cluster.local:9090}: linkerd_stack::failfast: HTTP Balancer service has become unavailable

[419972.169703s] DEBUG ThreadId(01) outbound:accept{client.addr=10.225.20.32:35952}:proxy{addr=10.224.44.90:9090}:http{v=1.x}:logical{dst=thanos-query-frontend.monitoring.svc.cluster.local:9090}: linkerd_stack::failfast: HTTP Logical service has become unavailable

Note: I also see this HTTP Balancer service has become unavailable far more frequently than we are seeing protocol timeout logs, and I'm unsure why.

There is a gist here which contains the manifests for:

• the injected grafana pod
• the injected thanos-query-frontend pod
• the thanos-query-frontend NodePort service

If there's any other information I can provide, please let me know. I have debug logs enabled on the outbound and inbound proxies so I can provide more logs as required.