tap not working in public dashboard exposed through nginix

[kkcmadhu]

hi i am having linkerd dashboard, the tap related section of the dashboard are not getting rendered.
I have ensured that the enforce-host of web is configured correctly, i also ensured that the origin and host names manage that of whats configured in enforce-host of web..

But unfortunately the Tap details do not get render. , neither it works with linkerd cli.

i see error "HTTP error, status Code [503] (unexpected API response: service unavailable"

in the browser i see response with body

Connection Closed: 1011 HTTP error, status Code [503] (unexpected API response: service unavailable and Http Response code 101

Not sure what happening.

[adleong]

Hi @kkcmadhu. I'd recommend running linkerd viz check and linkerd viz check --proxy to ensure that the viz extension is installed and configured properly. If those commands don't show any errors, can you provide the specific output from the linkerd viz tap command?

[kkcmadhu]

Thanks @adleong for taking a look.. here is my results

linkerd viz check
linkerd-viz

linkerd-viz

√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
‼ tap API service is running
FailedDiscoveryCheck: failing or missing response from https://10.1.144.120:8089/apis/tap.linkerd.io/v1alpha1: Get "https://10.1.144.120:8089/apis/tap.linkerd.io/v1alpha1": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
see https://linkerd.io/2.11/checks/#l5d-tap-api for hints
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
√ viz extension proxies are up-to-date
√ viz extension proxies and cli versions match
√ prometheus is installed and configured correctly
√ can initialize the client
√ viz extension self-check

.........

linkerd viz check --proxy

linkerd-viz

√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
‼ tap API service is running
FailedDiscoveryCheck: failing or missing response from https://10.1.144.120:8089/apis/tap.linkerd.io/v1alpha1: Get "https://10.1.144.120:8089/apis/tap.linkerd.io/v1alpha1": context deadline exceeded
see https://linkerd.io/2.11/checks/#l5d-tap-api for hints
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
√ viz extension proxies are up-to-date
√ viz extension proxies and cli versions match
√ prometheus is installed and configured correctly
√ can initialize the client
√ viz extension self-check

linkerd-viz-data-plane

√ data plane namespace exists
‼ data plane proxy metrics are present in Prometheus
Data plane metrics not found for idm/kc-13-pf1-aw-us-0, idm/kc-13-pf2-aw-us-1, idm/kc-13-pf2-aw-us-0, idm/kc-13-pf1-aw-us-1.
see https://linkerd.io/2.11/checks/#l5d-data-plane-prom for hints

Status check results are √

[kkcmadhu]

@adleong , can you recommend/suggest how i should proceed.

[kkcmadhu]

well i thought its related to ingress.. but seems this is not working in localhost too
and even through the cli "linkerd top pod/vote-bot-6d7677bb68-zkq9v --namespace emojivoto"
I the metrics logs i see this line TAP DISABLED.. what does it mean?

ogs -f metrics-api-6dc9cc7db-96lr9  linkerd-proxy -n linkerd-viz
[     0.000579s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.000967s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.000979s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.000981s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.000983s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.000985s]  INFO ThreadId(01) linkerd2_proxy: Local identity is metrics-api.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
[     0.000989s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.000991s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.013954s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: metrics-api.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
[  7235.908536s]  INFO ThreadId(01) outbound:server{orig_dst=172.20.80.146:9090}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[  7235.908581s]  INFO ThreadId(01) outbound:server{orig_dst=172.20.80.146:9090}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[  7253.370196s]  INFO ThreadId(01) outbound:server{orig_dst=172.20.80.146:9090}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[  7253.370251s]  INFO ThreadId(01) outbound:server{orig_dst=172.20.80.146:9090}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[  8407.900638s]  INFO ThreadId(01) outbound:server{orig_dst=172.20.80.146:9090}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[  8407.900699s]  INFO ThreadId(01) outbound:server{orig_dst=172.20.80.146:9090}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s`

web and web's linkerd-proxy i see

 kubectl logs -f web-7fc56d5655-24ch7 web -n linkerd-viz
time="2022-08-23T13:58:36Z" level=info msg="running version stable-2.11.4"
time="2022-08-23T13:58:36Z" level=info msg="starting admin server on :9994"
time="2022-08-23T13:58:36Z" level=info msg="starting HTTP server on :8084"
time="2022-08-23T15:58:22Z" level=error msg="Unexpected close error: websocket: close 1011 (internal server error): HTTP error, status Code [503] (unexpected API response: service unavailable\n)"
time="2022-08-23T15:58:27Z" level=error msg="Unexpected close error: websocket: close 1011 (internal server error): HTTP error, status Code [503] (unexpected API response: service unavailable\n)"

kubectl logs -f web-7fc56d5655-24ch7  linkerd-proxy -n linkerd-viz
[     0.000610s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.001020s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.001034s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.001036s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.001039s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.001042s]  INFO ThreadId(01) linkerd2_proxy: Local identity is web.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
[     0.001049s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.001052s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.009367s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: web.linkerd-viz.serviceaccount.identity.linkerd.cluster.local