linkerd viz tap not working

[loshsu]

Hello All,
I was testing linkerd in a local private k8s cluster, and trying to tap demo emoji voting app without success.
Do you have any suggestion on how this issue might be fixed?

Command:

sudo ./linkerd viz tap deployment/voting -n emojivoto

Note: I needed to use sudo to operate on non-default namespaces -- current arrangement of the private k8s cluster.

The following error messages in linkerd-tap pod were observed:

time="2021-02-25T07:48:27Z" level=info msg="Tapping 1 pods for target: namespace:"emojivoto" type:"deployment" name:"voting""
time="2021-02-25T07:48:27Z" level=info msg="Establishing tap on 172.100.2.136:4190"
time="2021-02-25T07:48:27Z" level=error msg="[172.100.2.136] encountered an error: rpc error: code = FailedPrecondition desc = request required the identity 'voting.emojivoto.serviceaccount.identity.linkerd.cluster.local' but no identity found"

linkerd version:
Client version: edge-21.2.3
Server version: edge-21.2.3

kubectl version:
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:26:42Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:18:22Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

[kleimkuhler]

@loshsu Hey sorry you're running into an issue with this. What are you using for creating the cluster? Are you on something like k3d/kind, or a cloud option like GKE/EKS?

[kleimkuhler]

@loshsu Could you provide any steps you used for creating the cluster? There is a similar issue in #5860 and it looks like it's permissions related based off the way the cluster is created.

If you want to double check your permissions I would also reference our docs here.

It provides GKE specific debugging, but it may be helpful for whatever cloud you are on.

[loshsu]

@kleimkuhler Thank you for the responses. I am testing on a local on-premise private cluster that was built by another team. Therefore, I have asked them to provide steps that they used to create the cluster. I will post them when I get a response.

In the mean time, I have checked the tap permissions described in your docs, but still no luck so far. I even granted cluster-admin to myself and tried the tap command again, but still got the same "but no identity found" error in linkerd-tap pod. I am wondering if that means this is not permissions related problem or there are other places that I should look at?

[kleimkuhler]

It may be worth looking at the root issue from #5860 which seems to be related to how the cluster was created. Once you find out the details about that, it should be a little easier to continue digging into what the issue is.

[loshsu]

@kleimkuhler Our private cluster was built with this command:

kubeadm init --apiserver-advertise-address=10.77.110.18 --pod-network-cidr=172.100.0.0/16 --service-cidr=10.96.0.0/16 --kubernetes-version=v1.19.1 --log-dir=/data/log/kubernetes  --cri-socket="/var/run/dockershim.sock" --upload-certs

Is there anything suspicious?

[kleimkuhler]

The issue for the user in #5860 was the use of the CIDR. I'm not totally sure why that causes an issue with tap, but they mentioned that it was flaky and seemed to work after a creation or two. It may worth a separate issue into why that occurs.