You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I went through all the steps and was able to get the two example services running, also exported the service on east to west
east being knvc and west being knvc1 in my case
Linkerd check on knvc(east)
kvalliyu@kvalliyu-mac-0 volume-controller % linkerd check
Linkerd core checks
===================
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version
linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks can be verified
√ cluster networks contains all node podCIDRs
linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used
linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor
linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days
linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date
control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match
linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match
Linkerd extensions checks
=========================
linkerd-multicluster
--------------------
√ Link CRD exists
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match
linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
√ viz extension proxies are up-to-date
√ viz extension proxies and cli versions match
√ prometheus is installed and configured correctly
√ can initialize the client
√ viz extension self-check
Status check results are √
Linkerd check on knvc1(west)
Linkerd core checks
===================
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version
linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks can be verified
√ cluster networks contains all node podCIDRs
linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used
linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor
linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days
linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date
control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match
linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match
Linkerd extensions checks
=========================
linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
* kvalliyu-nvc-cluster
√ remote cluster access credentials are valid
* kvalliyu-nvc-cluster
√ clusters share trust anchors
* kvalliyu-nvc-cluster
√ service mirror controller has required permissions
* kvalliyu-nvc-cluster
√ service mirror controllers are running
* kvalliyu-nvc-cluster
√ all gateway mirrors are healthy
* kvalliyu-nvc-cluster
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match
linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
√ viz extension proxies are up-to-date
√ viz extension proxies and cli versions match
√ prometheus is installed and configured correctly
√ can initialize the client
√ viz extension self-check
Status check results are √
Services on knvc1(west)
kubectl get svc -n test --context=knvc1
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
frontend ClusterIP 30.0.14.107 <none> 8080/TCP 13h
podinfo ClusterIP 30.0.10.223 <none> 9898/TCP,9999/TCP 13h
podinfo-kvalliyu-nvc-cluster ClusterIP 30.0.10.245 <none> 9898/TCP,9999/TCP 13h
podinfo-kvalliyu-nvc-cluster is the exported service from the knvc cluster(east)
Now following the guide when I try to curl the knvc service (east) from knvc1 service (west) I get the following error
/ # curl -v http://podinfo-kvalliyu-nvc-cluster:9898
* Trying 30.0.10.245:9898...
* Connected to podinfo-kvalliyu-nvc-cluster (30.0.10.245) port 9898 (#0)
> GET / HTTP/1.1
> Host: podinfo-kvalliyu-nvc-cluster:9898
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< l5d-proxy-error: client: Connection reset by peer (os error 104)
< connection: close
< content-length: 0
< date: Thu, 04 Aug 2022 14:32:24 GMT
<
* Closing connection 0
I confirmed that the both the deployments are meshed, in the knvc(east) gateway, I see this error
kubectl logs linkerd-gateway-d6df7cc4d-b8vf8 -n linkerd-multicluster -f
Defaulted container "linkerd-proxy" out of: linkerd-proxy, pause, linkerd-init (init)
[ 0.001006s] ERROR ThreadId(01) linkerd_app::env: No inbound ports specified via LINKERD2_PROXY_INBOUND_PORTS
[ 0.001287s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[ 0.002011s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.002029s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.002033s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.002037s] INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[ 0.002041s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[ 0.002050s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.002054s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.025716s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[ 40.267047s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated client.addr=29.54.0.28:12969
[ 86.079102s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated client.addr=29.54.129.1:29474
Which seemed to suggested that the TLS isn't working ? I am not sure how to debug this further, I did search for anyone hitting similar issues but their solutions didn't seem to apply to me
Logs of service mirror on knvc1(west)
kubectl logs linkerd-service-mirror-kvalliyu-nvc-cluster-59b7b9f88c-jnvm7 -n linkerd-multicluster --context=knvc1
Defaulted container "linkerd-proxy" out of: linkerd-proxy, service-mirror, linkerd-init (init)
[ 0.000786s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[ 0.001397s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.001416s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.001419s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.001422s] INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[ 0.001424s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-service-mirror-kvalliyu-nvc-cluster.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[ 0.001431s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.001434s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.023176s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-service-mirror-kvalliyu-nvc-cluster.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[ 47804.196074s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}:rescue{client.addr=30.0.133.24:45340}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: connect timed out after 1s
[ 47804.196131s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set
[ 47806.291021s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}:rescue{client.addr=30.0.133.24:45344}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 47806.291117s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set
[ 47809.335278s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}:rescue{client.addr=30.0.133.24:45390}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 47809.335336s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set
[ 47812.355670s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}:rescue{client.addr=30.0.133.24:45458}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 47812.355726s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set
[ 47815.420769s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}:rescue{client.addr=30.0.133.24:45462}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 47815.420849s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set
[ 47818.562294s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}:rescue{client.addr=30.0.133.24:45484}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 47818.562359s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set
[ 47821.781447s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}:rescue{client.addr=30.0.133.24:45572}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 47821.781502s] INFO ThreadId(01) outbound:server{orig_dst=30.0.12.8:4191}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I am following the Multicluster example setup to setup two clusters on GKE https://linkerd.io/2.11/tasks/multicluster/#linking-the-clusters
I went through all the steps and was able to get the two example services running, also exported the service on east to west
east being knvc and west being knvc1 in my case
Linkerd check on knvc(east)
Linkerd check on knvc1(west)
Services on knvc1(west)
podinfo-kvalliyu-nvc-clusteris the exported service from the knvc cluster(east)Now following the guide when I try to curl the knvc service (east) from knvc1 service (west) I get the following error
I confirmed that the both the deployments are meshed, in the knvc(east) gateway, I see this error
Which seemed to suggested that the TLS isn't working ? I am not sure how to debug this further, I did search for anyone hitting similar issues but their solutions didn't seem to apply to me
Logs of service mirror on knvc1(west)
The gateway itself seems to be fine
Any suggestions or pointers on what might be wrong or how to debug this further
ENV: GKE kubernets cluster
Kubernets v1.23
Beta Was this translation helpful? Give feedback.
All reactions