Install viz components on linkerd namespace (2.10.0) helm

[shahriak]

Hello,

I am trying to install linkerd 2.10.0 using Helm but I want all viz components installed under the same namespace (linkerd) (similar to what we had prior to 2.10.0 version) instead of the new namespace introduced (linkerd-viz). I was able to update the values file to point to linkerd namespace, however, some of the viz components are showing 0/1. I noticed linkerd inject is disabled on linkerd namespace by default and enabled on linkerd-viz.

grafana-7868878968-c2t9x                             1/1     Running   0          2m37s
linkerd-controller-7fd9d448d6-4lds4                  2/2     Running   0          2m31s
linkerd-destination-8c7f5bb8d-8fb7c                  2/2     Running   0          2m32s
linkerd-identity-7957678545-fsddp                    2/2     Running   0          2m30s
linkerd-proxy-injector-8bbbfcb9f-mq8hp               2/2     Running   0          2m35s
linkerd-sp-validator-75cfd46dfd-q972f                2/2     Running   0          2m32s
linkerd2-ingress-nginx-controller-6d887d4845-nw572   1/1     Running   0          2m28s
metrics-api-5dc96564c6-n5748                         0/1     Running   2          2m37s
prometheus-96c789bf9-djnql                           1/1     Running   0          2m37s
tap-5d6bc465f8-v58q2                                 0/1     Running   2          2m37s
tap-injector-5db94dfb7c-8z4fv                        0/1     Running   2          2m37s
web-5bdc4f8798-gjqnq                                 0/1     Running   2          2m31s

When i describe the tap pod, I see readiness probe failed error:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  7m4s                   default-scheduler  Successfully assigned linkerd/tap-5d6bc465f8-v58q2 to aks-l-32106900-vmss000000
  Normal   Pulling    7m2s                   kubelet            Pulling image "cr.l5d.io/linkerd/tap:stable-2.10.0"
  Normal   Pulled     6m55s                  kubelet            Successfully pulled image "cr.l5d.io/linkerd/tap:stable-2.10.0"
  Normal   Created    5m52s (x2 over 6m55s)  kubelet            Created container tap
  Normal   Started    5m52s (x2 over 6m54s)  kubelet            Started container tap
  Normal   Pulled     5m52s                  kubelet            Container image "cr.l5d.io/linkerd/tap:stable-2.10.0" already present on machine
  Warning  Unhealthy  5m13s (x6 over 6m43s)  kubelet            Liveness probe failed: Get http://10.240.0.7:9998/ping: dial tcp 10.240.0.7:9998: connect: connection refused
  Normal   Killing    5m13s (x2 over 6m23s)  kubelet            Container tap failed liveness probe, will be restarted
  Warning  Unhealthy  2m1s (x30 over 6m51s)  kubelet            Readiness probe failed: Get http://10.240.0.7:9998/ready: dial tcp 10.240.0.7:9998: connect: connection refused

[shahriak]

restarting the linkerd-viz namespaced components resolved the issue!

[mateiidavid]

Hey @shahriak,

I'm glad you got it sorted in the end!

noticed linkerd inject is disabled on linkerd namespace by default and enabled on linkerd-viz

Hm yeah, that's expected behaviour in a way. To support linkerd inject, we use admission webhooks. Those webhooks use the proxy-injector component of the control plane as a server [1]; for this reason, when we install linkerd we can't rely on automatic proxy injection (since part of the control plane it wouldn't be there to inject the rest of our control plane components). When we install viz the control plane is already live so we can use automatic injection.

I'll go ahead and mark your answer as correct since you seem to have solved it. If this comes back down the line then let us know and perhaps open an issue with some reproduction steps, we might be able to fix it! :)

[pcantea]

I'm running into the same issue today, linkerd-viz components aren't coming up correctly when installed in the linkerd namespace. We tried to restart all the viz components with no luck

NAME                                      READY   STATUS    RESTARTS   AGE
linkerd-controller-5d7d7dffc4-4m2kt       2/2     Running   0          4h4m
linkerd-controller-5d7d7dffc4-fr8pn       2/2     Running   0          4h4m
linkerd-controller-5d7d7dffc4-qfk6b       2/2     Running   0          4h4m
linkerd-destination-74cfd654c8-7d8kj      2/2     Running   0          4h4m
linkerd-destination-74cfd654c8-92gqh      2/2     Running   0          4h4m
linkerd-destination-74cfd654c8-znpdw      2/2     Running   0          4h4m
linkerd-identity-74c6bdf594-msxpt         2/2     Running   0          4h4m
linkerd-identity-74c6bdf594-wc896         2/2     Running   0          4h4m
linkerd-identity-74c6bdf594-zfxwf         2/2     Running   0          4h4m
linkerd-proxy-injector-8655cb956b-hc98h   2/2     Running   0          4h4m
linkerd-proxy-injector-8655cb956b-wxkv2   2/2     Running   0          4h4m
linkerd-proxy-injector-8655cb956b-x6tl9   2/2     Running   0          4h4m
linkerd-sp-validator-5bc6f667c-dqvrt      2/2     Running   0          4h4m
linkerd-sp-validator-5bc6f667c-gs58j      2/2     Running   0          4h4m
linkerd-sp-validator-5bc6f667c-jd9b8      2/2     Running   0          4h4m
metrics-api-94554866f-6f7zs               0/1     Running   1          117s
tap-6bd45884ff-lx2vv                      0/1     Running   1          117s
tap-injector-596867475b-t5lzv             0/1     Running   1          117s
web-d7f9f4bbf-8j8bh                       0/1     Running   1          117s

Events from the tap pod:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  2m26s                 default-scheduler  Successfully assigned linkerd/tap-6bd45884ff-lx2vv to ip-10-140-10-93.us-west-2.compute.internal
  Normal   Pulled     83s (x2 over 2m25s)   kubelet            Container image "cr.l5d.io/linkerd/tap:stable-2.10.2" already present on machine
  Normal   Created    83s (x2 over 2m25s)   kubelet            Created container tap
  Normal   Started    83s (x2 over 2m25s)   kubelet            Started container tap
  Warning  Unhealthy  44s (x6 over 2m14s)   kubelet            Liveness probe failed: Get "http://10.7.25.58:9998/ping": dial tcp 10.7.25.58:9998: connect: connection refused
  Normal   Killing    44s (x2 over 114s)    kubelet            Container tap failed liveness probe, will be restarted
  Warning  Unhealthy  37s (x11 over 2m17s)  kubelet            Readiness probe failed: Get "http://10.7.25.58:9998/ready": dial tcp 10.7.25.58:9998: connect: connection refused

[kleimkuhler]

@pcantea by default the linkerd namespace has the linkerd.io/inject: disabled annotation which will prevent workloads from being injected by the proxy injector. If the namespace still has this annotation then this is the reason that the workloads are not being injected.

It's possible that this annotation was removed if you installed linkerd-viz with linkerd viz install --set namespace=linkerd which would overwrite the namespace and change the annotation value to linkerd.io/inject: enabled. If this is the case, there is likely something else going on so we can go from there if that is the case.

[dboshardy]

I'm also encountering a variant of this issue. I'm following the Getting Started documentation and trying to install these to their default namespaces (linkerd-viz, linkerd-jaeger). Both jaegar and viz extensions' readiness and liveness probes fail with connection refused, specifically:

• metrics-api
• web
• tap
• tap-injector
• jaeger-injector

Logs from the proxy-injector for me show the following:

time="2021-07-12T22:27:47Z" level=info msg="running version stable-2.10.2"
time="2021-07-12T22:27:47Z" level=info msg="waiting for caches to sync"
time="2021-07-12T22:27:47Z" level=info msg="caches synced"
time="2021-07-12T22:27:47Z" level=info msg="starting admin server on :9995"
time="2021-07-12T22:27:47Z" level=info msg="listening at :8443"

For me, the linkerd namespace does not have the inject annotation enabled, only the linkerd-viz and linkerd-jaeger ns do.

FWIW, restarting the deployments did not fix the issue.

[dboshardy]

The logs for each look identical to what I pasted above. There have been no new logs since just after I installedlinkerd yesterday.

[dboshardy]

I'll be honest, the troubleshooting page in the documentation isn't very helpful for when linkerd doesn't work in the first place. Googling this is also not really turning up anything helpful.

[dboshardy]

This is a GKE private cluster, is this behavior indicative of a potential firewall issue?

[kleimkuhler]

@dboshardy sorry that the docs have not been too helpful with this. It is a potential firewall issue; if you are part of the linkerd slack you can see that user ran into this recently so it may be a potential fix for you: https://linkerd.slack.com/archives/C89RTCWJF/p1624290630036000?thread_ts=1623772655.290400&cid=C89RTCWJF

Without needing to follow-through that link, the gist is that they had a private GKE cluster and the firewall from their CIDR range was not whitelisted on the linkerd webhook service port. Let me know if that helps.

[dboshardy]

@kleimkuhler thanks for your help with this. I'll have to wait for help with that unfortunately. Thank you again for taking the time to look into this!