dial tcp: lookup prometheus.linkerd-viz.svc.cluster.local on 10.43.0.10:53: no such host #6601

hongbo-miao · 2021-08-03T23:57:26Z

hongbo-miao
Aug 3, 2021

I am following Linkerd2 Multi-Cluster Demo to install Linkerd for 3 clusters dev, east, west.

https://github.com/olix0r/l2-k3d-multi/blob/4cb5e6994671a5f6f507b7ad281b8c008927c9d0/install.sh#L39-L46

linkerd --context="k3d-$cluster" install \
        --proxy-log-level="linkerd=debug,trust_dns=debug,info" \
        --cluster-domain="$domain" \
        --identity-trust-domain="$domain" \
        --identity-trust-anchors-file="$CA_DIR/ca.crt" \
        --identity-issuer-certificate-file="${crt}" \
        --identity-issuer-key-file="${key}" |
    kubectl --context="k3d-$cluster" apply -f -

Then I installed Linkerd Viz to dev cluster by linkerd viz install --context=k3d-dev | kubectl apply -f -.

➜ kubectl --context=k3d-dev get svc -A
NAMESPACE              NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
default                kubernetes                  ClusterIP      10.43.0.1       <none>        443/TCP                         4h6m
kube-system            kube-dns                    ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP          4h6m
kube-system            metrics-server              ClusterIP      10.43.223.70    <none>        443/TCP                         4h6m
kube-system            traefik                     LoadBalancer   10.43.28.4      172.18.0.2    80:30019/TCP,443:30416/TCP      4h5m
linkerd                linkerd-identity            ClusterIP      10.43.231.143   <none>        8080/TCP                        4h3m
linkerd                linkerd-identity-headless   ClusterIP      None            <none>        8080/TCP                        4h3m
linkerd                linkerd-controller-api      ClusterIP      10.43.166.181   <none>        8085/TCP                        4h3m
linkerd                linkerd-dst                 ClusterIP      10.43.176.204   <none>        8086/TCP                        4h3m
linkerd                linkerd-dst-headless        ClusterIP      None            <none>        8086/TCP                        4h3m
linkerd                linkerd-proxy-injector      ClusterIP      10.43.15.54     <none>        443/TCP                         4h3m
linkerd                linkerd-sp-validator        ClusterIP      10.43.177.76    <none>        443/TCP                         4h3m
linkerd-multicluster   linkerd-gateway             LoadBalancer   10.43.35.159    172.18.0.2    4143:30272/TCP,4181:30850/TCP   4h2m
linkerd-multicluster   probe-gateway-east          ClusterIP      10.43.19.236    <none>        4181/TCP                        3h58m
linkerd-multicluster   probe-gateway-west          ClusterIP      10.43.253.129   <none>        4181/TCP                        3h58m
linkerd-viz            metrics-api                 ClusterIP      10.43.54.137    <none>        8085/TCP                        35m
linkerd-viz            grafana                     ClusterIP      10.43.229.196   <none>        3000/TCP                        35m
linkerd-viz            prometheus                  ClusterIP      10.43.31.32     <none>        9090/TCP                        35m
linkerd-viz            tap                         ClusterIP      10.43.254.106   <none>        8088/TCP,443/TCP                35m
linkerd-viz            tap-injector                ClusterIP      10.43.142.244   <none>        443/TCP                         35m
linkerd-viz            web                         ClusterIP      10.43.165.212   <none>        8084/TCP,9994/TCP               35m

➜ kubectl --context=k3d-dev get deploy -A
NAMESPACE              NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
kube-system            coredns                       1/1     1            1           4h7m
kube-system            metrics-server                1/1     1            1           4h7m
kube-system            local-path-provisioner        1/1     1            1           4h7m
kube-system            traefik                       1/1     1            1           4h6m
linkerd                linkerd-identity              1/1     1            1           4h4m
linkerd                linkerd-controller            1/1     1            1           4h4m
linkerd                linkerd-destination           1/1     1            1           4h4m
linkerd                linkerd-proxy-injector        1/1     1            1           4h4m
linkerd                linkerd-sp-validator          1/1     1            1           4h4m
linkerd-multicluster   linkerd-gateway               1/1     1            1           4h3m
linkerd-multicluster   linkerd-service-mirror-east   1/1     1            1           3h58m
linkerd-multicluster   linkerd-service-mirror-west   1/1     1            1           3h58m
linkerd-viz            metrics-api                   1/1     1            1           36m
linkerd-viz            tap                           1/1     1            1           36m
linkerd-viz            tap-injector                  1/1     1            1           36m
linkerd-viz            grafana                       1/1     1            1           36m
linkerd-viz            web                           1/1     1            1           36m
linkerd-viz            prometheus                    1/1     1            1           36m

However, when I got error

➜ linkerd --context=k3d-dev viz check
linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ prometheus is installed and configured correctly
√ can initialize the client
-× viz extension self-check
    Error calling Prometheus from the control plane: Post "http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query": dial tcp: lookup prometheus.linkerd-viz.svc.cluster.local on 10.43.0.10:53: no such host
    see https://linkerd.io/2.10/checks/#l5d-viz-metrics-api for hints

Status check results are ×

and I am not be able to open the dashboard

➜ linkerd --context=k3d-dev viz dashboard
Waiting for linkerd-viz extension to become available
Waiting for linkerd-viz extension to become available
Waiting for linkerd-viz extension to become available

Here is some the log for deploy/metrics-api:

➜ kubectl --context=k3d-dev -n linkerd-viz logs deploy/metrics-api metrics-api
time="2021-08-03T23:15:51Z" level=info msg="running version dev-undefined"
time="2021-08-03T23:15:51Z" level=info msg="prometheusClient: &api.httpClient{endpoint:(*url.URL)(0xc0003c35f0), client:http.Client{Transport:(*http.Transport)(0x27fd2c0), CheckRedirect:(func(*http.Request, []*http.Request) error)(nil), Jar:http.CookieJar(nil), Timeout:0}}"
time="2021-08-03T23:15:51Z" level=info msg="Using cluster domain: cluster.local"
time="2021-08-03T23:15:51Z" level=info msg="waiting for caches to sync"
time="2021-08-03T23:15:51Z" level=info msg="caches synced"
time="2021-08-03T23:15:51Z" level=info msg="starting admin server on :9995"
time="2021-08-03T23:15:51Z" level=info msg="starting HTTP server on :8085"
time="2021-08-03T23:16:53Z" level=error msg="Query(max(process_start_time_seconds{}) by (pod, namespace)) failed with: Post \"http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus.linkerd-viz.svc.cluster.local on 10.43.0.10:53: no such host"
time="2021-08-03T23:16:58Z" level=error msg="Query(max(process_start_time_seconds{}) by (pod, namespace)) failed with: Post \"http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus.linkerd-viz.svc.cluster.local on 10.43.0.10:53: no such host"
time="2021-08-03T23:18:30Z" level=error msg="Query(max(process_start_time_seconds{}) by (pod, namespace)) failed with: Post \"http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus.linkerd-viz.svc.cluster.local on 10.43.0.10:53: no such host"
time="2021-08-03T23:19:23Z" level=error msg="Query(max(process_start_time_seconds{}) by (pod, namespace)) failed with: Post \"http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus.linkerd-viz.svc.cluster.local on 10.43.0.10:53: no such host"

I port forward web 8084 from linkerd-viz namespace, it will be a blank page.

Any idea? Thanks!

Answered by hongbo-miao

Aug 4, 2021

I found when create cluster without --k3s-server-arg="--cluster-domain=$cluster.${ORG_DOMAIN}" at
https://github.com/olix0r/l2-k3d-multi/blob/4cb5e6994671a5f6f507b7ad281b8c008927c9d0/create.sh#L19

And when install Linkerd without --cluster-domain=${domain} at
https://github.com/olix0r/l2-k3d-multi/blob/4cb5e6994671a5f6f507b7ad281b8c008927c9d0/install.sh#L41

Linkerd Viz dashboard is able to open.

But then my multi cluster starts to have this old "all gateway mirrors are healthy" error again

➜ linkerd --context=k3d-west multicluster check

linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
	* k3d-east
× remote cluster access credentials are valid
        …

View full answer

hongbo-miao · 2021-08-04T01:36:43Z

hongbo-miao
Aug 4, 2021
Author

I found when create cluster without --k3s-server-arg="--cluster-domain=$cluster.${ORG_DOMAIN}" at
https://github.com/olix0r/l2-k3d-multi/blob/4cb5e6994671a5f6f507b7ad281b8c008927c9d0/create.sh#L19

And when install Linkerd without --cluster-domain=${domain} at
https://github.com/olix0r/l2-k3d-multi/blob/4cb5e6994671a5f6f507b7ad281b8c008927c9d0/install.sh#L41

Linkerd Viz dashboard is able to open.

But then my multi cluster starts to have this old "all gateway mirrors are healthy" error again

➜ linkerd --context=k3d-west multicluster check

linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
	* k3d-east
× remote cluster access credentials are valid
            * failed to connect to API for cluster: [k3d-east]: Get "https://192.168.80.2:6443/version?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    see https://linkerd.io/2.10/checks/#l5d-smc-target-clusters-access for hints
W0803 19:06:37.876061   46004 transport.go:260] Unable to cancel request for promhttp.RoundTripperFunc
× clusters share trust anchors
    Problematic clusters:
    * k3d-east: unable to fetch anchors: Get "https://192.168.80.2:6443/api/v1/namespaces/linkerd/configmaps/linkerd-config?timeout=30s": context deadline exceeded
    see https://linkerd.io/2.10/checks/#l5d-multicluster-clusters-share-anchors for hints
√ service mirror controller has required permissions
	* k3d-east
√ service mirror controllers are running
	* k3d-east
× all gateway mirrors are healthy
        probe-gateway-k3d-east.linkerd-multicluster mirrored from cluster [k3d-east] has no endpoints
    see https://linkerd.io/2.10/checks/#l5d-multicluster-gateways-endpoints for hints

with linkerd-service-mirror-k3d-east-xxx pod log:

time="2021-08-04T02:17:13Z" level=info msg="Got updated link k3d-east: {Name:k3d-east Namespace:linkerd-multicluster TargetClusterName:k3d-east TargetClusterDomain:cluster.local TargetClusterLinkerdNamespace:linkerd ClusterCredentialsSecret:cluster-credentials-k3d-east GatewayAddress:192.168.80.2 GatewayPort:4143 GatewayIdentity:linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.east.k3d.hongbomiao.com ProbeSpec:ProbeSpec: {path: /health, port: 4181, period: 3s} Selector:{MatchLabels:map[] MatchExpressions:[{Key:mirror.linkerd.io/exported Operator:Exists Values:[]}]}}"
time="2021-08-04T02:17:34Z" level=error msg="Unable to create cluster watcher: cannot connect to api for target cluster remote: an error on the server (\"\") has prevented the request from succeeding"
time="2021-08-04T02:17:44Z" level=info msg="Link watch terminated; restarting watch"
time="2021-08-04T02:17:44Z" level=info msg="Got updated link k3d-east: {Name:k3d-east Namespace:linkerd-multicluster TargetClusterName:k3d-east TargetClusterDomain:cluster.local TargetClusterLinkerdNamespace:linkerd ClusterCredentialsSecret:cluster-credentials-k3d-east GatewayAddress:192.168.80.2 GatewayPort:4143 GatewayIdentity:linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.east.k3d.hongbomiao.com ProbeSpec:ProbeSpec: {path: /health, port: 4181, period: 3s} Selector:{MatchLabels:map[] MatchExpressions:[{Key:mirror.linkerd.io/exported Operator:Exists Values:[]}]}}"
time="2021-08-04T02:18:05Z" level=error msg="Unable to create cluster watcher: cannot connect to api for target cluster remote: an error on the server (\"\") has prevented the request from succeeding"
time="2021-08-04T02:18:15Z" level=info msg="Link watch terminated; restarting watch"

2 replies

hongbo-miao Aug 4, 2021
Author

Based on early error message. I think we need let prometheus.linkerd-viz.svc.cluster.local run in the k3d-dev's cluster. Not cluster.local.
However, linkerd viz install currently does not have a --cluster-domain parameter.

hongbo-miao Aug 4, 2021
Author

Thanks for @Pothulapati answering at here.

For linkerd viz install, to set cluster domain, --set is being used.

linkerd viz install --set clusterDomain="google.com"

My final working solution to use Linkerd Viz Dashboard in multi clusters is at
https://github.com/Hongbo-Miao/hongbomiao.com/blob/2a60a87705ec7634ba2bc47d0d58e9a7c551dd70/kubernetes/bin/setupK3dMultiCluster.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

dial tcp: lookup prometheus.linkerd-viz.svc.cluster.local on 10.43.0.10:53: no such host #6601

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

dial tcp: lookup prometheus.linkerd-viz.svc.cluster.local on 10.43.0.10:53: no such host #6601

Uh oh!

Uh oh!

hongbo-miao Aug 3, 2021

Replies: 1 comment · 2 replies

Uh oh!

Uh oh!

hongbo-miao Aug 4, 2021 Author

Uh oh!

Uh oh!

hongbo-miao Aug 4, 2021 Author

Uh oh!

Uh oh!

hongbo-miao Aug 4, 2021 Author

hongbo-miao
Aug 3, 2021

Replies: 1 comment 2 replies

hongbo-miao
Aug 4, 2021
Author

hongbo-miao Aug 4, 2021
Author

hongbo-miao Aug 4, 2021
Author