Multi-Cluster communication connections must be mutually authenticated

[mbaykara]

Following steps I have performed:

1. I have created 2 AKS clusters east and west.
2. Create certs via

//cert 
step certificate create root.linkerd.cluster.local root.crt root.key --profile root-ca --no-password --insecure
//issuer
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key --profile intermediate-ca --not-after 8760h --no-password --insecure  --ca root.crt --ca-key root.key

3. Installed Linkerd for each cluster

linkerd install \
  --identity-trust-anchors-file root.crt \
  --identity-issuer-certificate-file issuer.crt \
  --identity-issuer-key-file issuer.key \
  | kubectl apply -f -

4. Verified the installation, then installed multi-cluster part for each cluster

linkerd multicluster install |  kubectl apply -f -

5. Then run k -n linkerd-multicluster logs --context west linkerd-gateway-57988667cd-tkt2w -c linkerd-proxy the output

   0.000442s] ERROR ThreadId(01) linkerd_app::env: No inbound ports specified via LINKERD2_PROXY_INBOUND_PORTS
[     0.000611s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.001147s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.001157s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.001159s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.001161s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.001163s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[     0.001167s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.001169s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.047778s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[     4.938778s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[     9.947119s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[    10.949054s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[    15.950121s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[    16.952876s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[    21.959351s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[    22.961091s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[    27.964759s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[    28.967106s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[    33.982771s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[    34.983523s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out

Where do I do wrong? Why are these errors?
environment:
aks cluster
version: 1.22.6
Linkerd version

Client version: stable-2.11.1
Server version: stable-2.11.1

6. Link the west cluster to east

linkerd --context=east multicluster link --set enableHeadlessServices=true --cluster-name east | kubectl --context=west apply -f -

7. Create namespace for cluster east and west

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    linkerd.io/inject: enabled  
  name: monitoring

8. Lastly I installed Prometheus both cluster and Thanos in cluster west
9. Expose it headless svc from east to west

kubectl --context=east label svc prometheus-kube-prometheus-thanos-discovery  mirror.linkerd.io/exported=true

Now I have same service in west cluster with east suffix but the new created mirror svc is not headless, it has an ClusterIP address.
And the Thanos query is not getting the metrics from east cluster.
Both clusters monitoring namespace has linkerd injected. So I do not understand why is not working

[mateiidavid]

@mbaykara are your workloads injected? If you're sending traffic from east to west, the client in east needs to be injected.

[mbaykara]

@mateiidavid Yes I injected, still not getting any metrics from the mirrored service :/ pls see my updates

[yanglan-ly]

hey, bro. have you solved this problem?

[mbaykara]

Nope, I gave a break, let me know if you find it

[manju-rn]

after the injection, did you restart the pods / deployment? To double check if it is meshed or not, you could install the viz extension and check in UI to confirm

[jradams381]

I'm seeing the same thing with 2.12.3. Admittedly, I am a novice user so it is almost certainly a configuration error on my part. This happens before any kind of linking on my part.

All checks pass (e.g. linkerd check). From what I can tell, this appears to be coming from the nodes' IPs, perhaps kubelet probes of some kind?

[jradams381]

Apologies, but I was able to confirm my situation was a simple configuration issue on my part. The egress rules simply failed to include 4143.

[andrew-gropyus]

After experiencing this on AKS too, we tracked it down to the load balancer's health probe.

In short, this is not a problem however the logs are noisy and can hide other issues.

The issue seems to be for the proxy's data plane port (4143 by default), which wants mTLS traffic if directly addressed.

From the load balancer's TCP probe perspective: a TCP connection that is accepted and then closed remotely is considered a success. However from the linkerd-proxy's perspective, a non mTLS connection is bad and so it closes the connection, producing the log.

We had success with using some annotations on the linkerd-gateway service, indicating that the probe for the data plane should use the admin port's /ready endpoint.

In the helm values for the multicluster chart this would look something like (untested):

gateway:
  serviceAnnotations:
    "service.beta.kubernetes.io/port_4191_health-probe_protocol": "Http"
    "service.beta.kubernetes.io/port_4191_health-probe_request-path": "/ready"
    "service.beta.kubernetes.io/port_4143_health-probe_protocol": "Http"
    "service.beta.kubernetes.io/port_4143_health-probe_port": "4191"
    "service.beta.kubernetes.io/port_4143_health-probe_request-path": "/ready"

See https://cloud-provider-azure.sigs.k8s.io/topics/loadbalancer/#custom-load-balancer-health-probe-for-port for docs on what these annotations do.

@mateiidavid We sunk a few hours into this and I suppose other AKS folks are too, would it make sense to make these annotation a default or maybe add this to the docs somewhere? Maybe time to promote this from a discussion to an issue?

[whiskeysierra]

I'd assume the same issue exists for AWS as well: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/annotations/