Can't setup linkerd-viz due zero log

[rumanzo]

Hello!
I try to setup linkerd (several weeks, aclually). I successfully setup main part - linkerd2, but I can't install linkerd-viz.
My cluster has calico, psp rules, some taints and I get certificates with cert-manager + vault issuer.
I deployed linkerd2 via helm with custom calico templates. It works.
But when I try to setup linkerd-viz via helm, pods not pass health checks, and setup fail. The worst is no logs at all, all pods has empty logs. I don't know what I miss, fw rules, or something else.

$ linkerd2 check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2021-07-21T01:42:04Z
    see https://linkerd.io/2.10/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days

linkerd-api
-----------
√ control plane pods are ready
√ can initialize the client
√ can query the control plane API

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

Status check results are √

$ kubectl describe deploy -n linkerd-viz metrics-api
Message:
Name:                   metrics-api
Namespace:              linkerd-viz
CreationTimestamp:      Mon, 19 Jul 2021 14:49:08 +0300
Labels:                 app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=metrics-api
                        app.kubernetes.io/part-of=Linkerd
                        app.kubernetes.io/version=stable-2.10.2
                        component=metrics-api
                        linkerd.io/extension=viz
Annotations:            deployment.kubernetes.io/revision: 1
                        linkerd.io/created-by: linkerd/helm stable-2.10.2
                        meta.helm.sh/release-name: linkerd-viz
                        meta.helm.sh/release-namespace: linkerd-viz
Selector:               component=metrics-api,linkerd.io/extension=viz
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           component=metrics-api
                    linkerd.io/extension=viz
  Annotations:      checksum/config: 0d5b035f4d141dc2c13e1f89046de78fe0fb1208075734c3977400b866f2db51
                    linkerd.io/created-by: linkerd/helm stable-2.10.2
  Service Account:  metrics-api
  Containers:
   metrics-api:
    Image:       registry.domain/linkerd/metrics-api:stable-2.10.2
    Ports:       8085/TCP, 9995/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      -controller-namespace=linkerd
      -log-level=debug
      -cluster-domain=cluster.local
      -prometheus-url=https://prometheus.domain
    Limits:
      memory:  250Mi
    Requests:
      cpu:        100m
      memory:     50Mi
    Liveness:     http-get http://:9995/ping delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:9995/ready delay=0s timeout=1s period=10s #success=1 #failure=7
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  <none>
NewReplicaSet:   metrics-api-847f977b8f (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  86s   deployment-controller  Scaled up replica set metrics-api-847f977b8f to 1

$ kubectl describe po -n linkerd-viz metrics-api-847f977b8f-nw4pk
Message:
Name:         metrics-api-847f977b8f-nw4pk
Namespace:    linkerd-viz
Priority:     0
Node:         kube-master2.domain/10.0.0.2
Start Time:   Mon, 19 Jul 2021 14:49:18 +0300
Labels:       component=metrics-api
              linkerd.io/extension=viz
              pod-template-hash=847f977b8f
Annotations:  checksum/config: 0d5b035f4d141dc2c13e1f89046de78fe0fb1208075734c3977400b866f2db51
              cni.projectcalico.org/podIP: 10.0.1.225/32
              cni.projectcalico.org/podIPs: 10.0.1.225/32
              kubernetes.io/psp: z-default
              linkerd.io/created-by: linkerd/helm stable-2.10.2
Status:       Running
IP:           10.0.1.225
IPs:
  IP:           10.0.1.225
Controlled By:  ReplicaSet/metrics-api-847f977b8f
Containers:
  metrics-api:
    Container ID:  containerd://d4c394a65b820c19989ed970c7b8dfa58096f6f6e5473f03d136a975ad966ad3
    Image:        registry.domain/linkerd/metrics-api:stable-2.10.2
    Image ID:      registry.domain/linkerd/metrics-api@sha256:2f7566f85a15f67fb78a3182e7762f4a0587b8eda073cf3306fda1e46a28bc8a
    Ports:         8085/TCP, 9995/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      -controller-namespace=linkerd
      -log-level=debug
      -cluster-domain=cluster.local
      -prometheus-url=https://prometheus.domain
    State:          Running
      Started:      Mon, 19 Jul 2021 14:50:30 +0300
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 19 Jul 2021 14:49:21 +0300
      Finished:     Mon, 19 Jul 2021 14:50:28 +0300
    Ready:          False
    Restart Count:  1
    Limits:
      memory:  250Mi
    Requests:
      cpu:        100m
      memory:     50Mi
    Liveness:     http-get http://:9995/ping delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:9995/ready delay=0s timeout=1s period=10s #success=1 #failure=7
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dcscb (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-dcscb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              beta.kubernetes.io/os=linux
                             kubernetes.io/role=master
Tolerations:                 :NoSchedule op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  97s                default-scheduler  Successfully assigned linkerd-viz/metrics-api-847f977b8f-nw4pk to kube-master2.domain
  Normal   Pulled     95s                kubelet            Successfully pulled image "registry-registry.domain/linkerd/metrics-api:stable-2.10.2" in 753.660436ms
  Normal   Killing    57s                kubelet            Container metrics-api failed liveness probe, will be restarted
  Normal   Pulling    27s (x2 over 96s)  kubelet            Pulling image "registry.domain/linkerd/metrics-api:stable-2.10.2"
  Normal   Pulled     26s                kubelet            Successfully pulled image "registry.domain/linkerd/metrics-api:stable-2.10.2" in 837.394822ms
  Normal   Created    25s (x2 over 94s)  kubelet            Created container metrics-api
  Normal   Started    25s (x2 over 94s)  kubelet            Started container metrics-api
  Warning  Unhealthy  7s (x11 over 93s)  kubelet            Readiness probe failed: Get "http://10.0.1.225:9995/ready": dial tcp 10.0.1.225:9995: connect: connection refused
  Warning  Unhealthy  7s (x4 over 77s)   kubelet            Liveness probe failed: Get "http://10.0.1.225:9995/ping": dial tcp 10.0.1.225:9995: connect: connection refused


Attention: the link can not be opened more than once, content has already been erased from server.

and I not have any logs

$ kubectl logs -n linkerd-viz metrics-api-847f977b8f-nw4pk
$ kubectl logs -n linkerd-viz tap-6dc9cc7948-bzpnx
$ kubectl logs -n linkerd-viz tap-injector-dd794db8d-nh2bq
$ kubectl logs -n linkerd-viz web-689f8b97c6-7fbtc
$ kubectl logs -n linkerd-viz metrics-api-847f977b8f-nw4pk -p

If I install namespace with helm (--create-namespace -n linkerd-viz), I won't have sidecar pods with proxy. If I use ns with

  annotations:
    linkerd.io/inject: enabled

I can't see any pods, and get this status in deployment describe

status:
  conditions:
  - lastTransitionTime: "2021-07-19T12:15:09Z"
    lastUpdateTime: "2021-07-19T12:15:09Z"
    message: Created new replica set "metrics-api-847f977b8f"
    reason: NewReplicaSetCreated
    status: "True"
    type: Progressing
  - lastTransitionTime: "2021-07-19T12:15:09Z"
    lastUpdateTime: "2021-07-19T12:15:09Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2021-07-19T12:15:09Z"
    lastUpdateTime: "2021-07-19T12:15:09Z"
    message: 'pods "metrics-api-847f977b8f-" is forbidden: PodSecurityPolicy: unable
      to admit pod: [spec.initContainers[0].securityContext.runAsNonRoot: Invalid
      value: false: must be true spec.initContainers[0].securityContext.capabilities.add:
      Invalid value: "NET_ADMIN": capability may not be added spec.initContainers[0].securityContext.capabilities.add:
      Invalid value: "NET_RAW": capability may not be added]'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure
  observedGeneration: 1
  unavailableReplicas: 1

So I have at least 2 problems:
1.) I can't understand what happens when linkerd-viz try to start, and I have zero logs
2.) I have some trouble with after inject proxy container

Can anyone help me?

[dadjeibaah]

@rumanzo, I have a guess that this might be related to this issue #6122, which was fixed in https://github.com/linkerd/linkerd2/releases/tag/edge-21.6.1. Can you confirm what version of linkerd are you using for the control plane and linkerd-viz?

[rumanzo]

Can you confirm what version of linkerd are you using for the control plane and linkerd-viz?

stable-2.10.2 for both of them

I will try setup with last edge

[rumanzo]

@dadjeibaah
I have installed linkerd2 and linkerd-viz edge-21.7.3 . Linkerd-viz started without linkerd-proxy.
With injected linkerd-proxy it stack on pod initialazing, and I can't see any logs except for linkerd-init (nothing interesting).

[rumanzo]

I resolved my problem, thanks @dadjeibaah .
First thing was bug in stable-2.10.2 version connected with psp.
Second thing was incorrect calico rules. Proxy couldn't connect to control plane.