Error when following multi cluster official guide - https://linkerd.io/2.10/tasks/multicluster/

[dmraj]

Hi,

I'm new to linkerd so please bear with me.

I'm following this guide for multicluster linkerd setup.

https://linkerd.io/2.10/tasks/multicluster/

I have my local MAC (west) and AWS (east)

version( both east and west):

RADM-M-74M0:linker_demo radm$ linkerd version
Client version: stable-2.10.0
Server version: stable-2.10.0

when doing the "Linking the clusters" section im facing error when i dolinkerd multicluster check

RADM-M-74M0:linker_demo radm$ linkerd --context=west multicluster check
linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
	* east
√ remote cluster access credentials are valid
	* east
√ clusters share trust anchors
	* east
√ service mirror controller has required permissions
	* east
√ service mirror controllers are running
	* east
× all gateway mirrors are healthy
        probe-gateway-east.linkerd-multicluster mirrored from cluster [east] has no endpoints
    see https://linkerd.io/checks/#l5d-multicluster-gateways-endpoints for hints

Status check results are ×

I also see this error in the service mirror pods(west):

RADM-M-74M0:linker_demo radm$ kubectl logs pod/linkerd-service-mirror-east-7b8bb648dd-jht5l -n linkerd-multicluster -c service-mirror | tail -3
time="2021-03-29T14:38:58Z" level=info msg="Link watch terminated; restarting watch"
time="2021-03-29T14:38:58Z" level=info msg="Got updated link east: {Name:east Namespace:linkerd-multicluster TargetClusterName:east TargetClusterDomain:cluster.local TargetClusterLinkerdNamespace:linkerd ClusterCredentialsSecret:cluster-credentials-east GatewayAddress:a79d38f5df79b935bc33de3-1109770263.us-west-2.elb.amazonaws.com GatewayPort:4143 GatewayIdentity:linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local ProbeSpec:ProbeSpec: {path: /health, port: 4181, period: 3s} Selector:{MatchLabels:map[] MatchExpressions:[{Key:mirror.linkerd.io/exported Operator:Exists Values:[]}]}}"
time="2021-03-29T14:38:58Z" level=error msg="Unable to parse kube config: invalid configuration: unable to read certificate-authority /Users/radm/linker_demo/fake-ca-file for east due to open /Users/radm/linker_demo/fake-ca-file: no such file or directory"

but the file is present in my mac on same path:

RADM-M-74M0:~ radm$ ls -ltr /Users/radm/linker_demo/fake-ca-file
-rw-r--r--  1 radm  staff  1111 Mar 29 17:31 /Users/radm/linker_demo/fake-ca-file

other info:

RADM-M-74M0:linker_demo radm$ linkerd --context=west multicluster gateways
CLUSTER  ALIVE    NUM_SVC  LATENCY_P50  LATENCY_P95  LATENCY_P99

RADM-M-74M0:linker_demo radm$ kubectl  get all -n linkerd-multicluster
NAME                                   READY   STATUS    RESTARTS   AGE
pod/linkerd-gateway-6f94dc94bb-kdct5   2/2     Running   0          3h55m
NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                         AGE
service/linkerd-gateway   LoadBalancer   10.101.219.229   a79d38f9f79b935bc33de3-1109770263.us-west-2.elb.amazonaws.com   4143:30599/TCP,4181:32431/TCP   3h55m
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/linkerd-gateway   1/1     1            1           3h55m
NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/linkerd-gateway-6f94dc94bb   1         1         1       3h55m
RADM-M-74M0:linker_demo radm$ kubectl config use-context west
Switched to context "west".
RADM-M-74M0:linker_demo radm$ kubectl  get all -n linkerd-multicluster
NAME                                               READY   STATUS    RESTARTS   AGE
pod/linkerd-gateway-6f94dc94bb-vv6jf               2/2     Running   0          3h56m
pod/linkerd-service-mirror-east-7b8bb648dd-jht5l   2/2     Running   0          179m
NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
service/linkerd-gateway      LoadBalancer   10.101.237.152   <pending>     4143:32722/TCP,4181:30654/TCP   3h56m
service/probe-gateway-east   ClusterIP      10.107.152.171   <none>        4181/TCP                        3h50m
NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/linkerd-gateway               1/1     1            1           3h56m
deployment.apps/linkerd-service-mirror-east   1/1     1            1           3h50m
NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/linkerd-gateway-6f94dc94bb               1         1         1       3h56m
replicaset.apps/linkerd-service-mirror-east-6b695fb57b   0         0         0       3h50m
replicaset.apps/linkerd-service-mirror-east-7b8bb648dd   1         1         1       179m
RADM-M-74M0:linker_demo radm$

i see only one in west (local mac) for linkerd gateway for external-ip could that cause this kind of problem ?

curl to east (aws) gateway:

RADM-M-74M0:linker_demo radm$ curl http://a79d38f55e422c9f79b935bc33de3-1109770263.us-west-2.elb.amazonaws.com:4143
curl: (52) Empty reply from server
RADM-M-74M0:linker_demo radm$ telnet a79d38f55e422c9f79b935bc33de3-1109770263.us-west-2.elb.amazonaws.com 4143
Trying 52.40.2.158...
Connected to a79d38f55e422c9f79b935bc33de3-1109770263.us-west-2.elb.amazonaws.com.
Escape character is '^]'.
Connection closed by foreign host.
RADM-M-74M0:linker_demo radm$ curl http://a79d38f55e422c9f79b935bc33de3-1109770263.us-west-2.elb.amazonaws.com:4181
curl: (52) Empty reply from server
RADM-M-74M0:linker_demo radm$

Please let me know if any other details are required.

Thanks

[olix0r]

also see this error in the service mirror pods(west):

RADM-M-74M0:linker_demo radm$ kubectl logs pod/linkerd-service-mirror-east-7b8bb648dd-jht5l -n linkerd-multicluster -c service-mirror | tail -3
time="2021-03-29T14:38:58Z" level=info msg="Link watch terminated; restarting watch"
time="2021-03-29T14:38:58Z" level=info msg="Got updated link east: {Name:east Namespace:linkerd-multicluster TargetClusterName:east TargetClusterDomain:cluster.local TargetClusterLinkerdNamespace:linkerd ClusterCredentialsSecret:cluster-credentials-east GatewayAddress:a79d38f5df79b935bc33de3-1109770263.us-west-2.elb.amazonaws.com GatewayPort:4143 GatewayIdentity:linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local ProbeSpec:ProbeSpec: {path: /health, port: 4181, period: 3s} Selector:{MatchLabels:map[] MatchExpressions:[{Key:mirror.linkerd.io/exported Operator:Exists Values:[]}]}}"
time="2021-03-29T14:38:58Z" level=error msg="Unable to parse kube config: invalid configuration: unable to read certificate-authority /Users/radm/linker_demo/fake-ca-file for east due to open /Users/radm/linker_demo/fake-ca-file: no such file or directory"

but the file is present in my mac on same path:

RADM-M-74M0:~ radm$ ls -ltr /Users/radm/linker_demo/fake-ca-file
-rw-r--r--  1 radm  staff  1111 Mar 29 17:31 /Users/radm/linker_demo/fake-ca-file

That's the problem. The error message is coming from your k8s cluster, but it's referencing a file that you only have in your home directory on your Mac.

Can you share the specific commands you ran to install/link the multi cluster gateways?