Linkerd controller pods are not coming up

[sathishkumar3009]

What is the issue?

Deployed linkerD using helm charts.
Repository: "https://helm.linkerd.io/stable"
Version: 2.10.0

Pods of linkerd is not coming up due to the below error.

[  3115.012070s]  WARN ThreadId(01) daemon:identity: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN

NAME                                      READY        STATUS       RESTARTS   AGE
linkerd-controller-6568cb69c4-j6j55          0/2             CrashLoopBackOff   17         50m
linkerd-destination-7f4b5bfd-8kztt            0/2             CrashLoopBackOff   17         50m
linkerd-identity-64894647c9-7jkmh           0/2             Init:0/1           0          50m
linkerd-proxy-injector-5d45fb6b6b-zkpsl   0/2            Init:0/1           0          50m
linkerd-sp-validator-7ddb4b68f4-xjztl        0/2            Init:0/1           0          50m

NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
linkerd-controller-api      ClusterIP   10.0.166.165   <none>        8085/TCP   50m
linkerd-dst                 ClusterIP   10.0.251.17    <none>        8086/TCP   50m
linkerd-dst-headless        ClusterIP   None           <none>        8086/TCP   50m
linkerd-identity            ClusterIP   10.0.145.50    <none>        8080/TCP   50m
linkerd-identity-headless   ClusterIP   None           <none>        8080/TCP   50m
linkerd-proxy-injector      ClusterIP   10.0.75.48     <none>        443/TCP    50m
linkerd-sp-validator        ClusterIP   10.0.101.116   <none>        443/TCP    50m

How can it be reproduced?

Deploying helm charts from this repo.https://github.com/Azure-Terraform/terraform-helm-linkerd/blob/main/variables.tf

Logs, error output, etc

[ 3115.012070s] WARN ThreadId(01) daemon:identity: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN

output of linkerd check -o short

Notworking

Environment

Kubernetes version 1.22.0

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

No response

[mateiidavid]

Hey @sathishkumar3009

[ 3115.012070s] WARN ThreadId(01) daemon:identity: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN

Is this the only log warning you're seeing? This appears to be recorded around 50 minutes after your pod's proxy was started (i.e 3115.012070s). In order for the proxy to function correctly, it needs to establish a connection to the identity service. Can you confirm the identity pod for linkerd is up and running? (kubectl get pods -n linkerd).

It would be helpful to see if the services have been created: kubectl get services -n linkerd.

And finally, can you confirm your k8s cluster has successfully set-up a DNS record for the headless identity service? I find that dig SRV <host> works best here, but you could also look-up the hostname using nslookup. The Kubernetes official docs have a guide on how to debug DNS resolution.

Edit: ah, I have just seen that your identity pod is not up. Can you do kubectl describe -n linkerd linkerd-identity-64894647c9-7jkmh and post the output here please?

[sathishkumar3009]

Hey david, thanks for the response,

Now all the pods are up and running but getting different warnings in the logs of Identity and controller. Is this an issue?

It is showing different record name now. linkerd-dst-headless.linkerd.svc.cluster.local.

Previously it is different when pods were not up: linkerd-identity-headless.linkerd.svc.cluster.local

Identity Logs:

 kubectl logs linkerd-identity-74c57b7fdb-pm4zh -n linkerd -c linkerd-proxy
time="2022-02-22T13:47:23Z" level=info msg="running version stable-2.10.1"
[     0.004617s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.005654s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.005681s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.005689s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.005696s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.005702s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.005719s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via localhost:8080
[     0.005727s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.029063s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
[     9.935615s]  WARN ThreadId(01) server{port=8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN
[     9.938528s]  WARN ThreadId(01) server{port=8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN
[    14.937915s]  WARN ThreadId(01) server{port=8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN
[    19.938312s]  WARN ThreadId(01) server{port=8080}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN

Controller Logs:

linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN
[    10.618514s]  WARN ThreadId(01) server{orig_dst=10.0.244.53:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN
[    13.616348s]  WARN ThreadId(01) server{orig_dst=10.0.244.53:8086}: linkerd_timeout::failfast: TCP Server entering failfast after 3s
[    13.616498s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast
[    14.633118s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast
[    15.618967s]  WARN ThreadId(01) server{orig_dst=10.0.244.53:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN
[    16.249476s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast
[    18.944873s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast
[    20.618547s]  WARN ThreadId(01) server{orig_dst=10.0.244.53:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN
[    22.337432s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast

[mateiidavid]

If the pods have all successfully come up but you still get warnings, it means your DNS resolution might not work correctly in the cluster. You can follow the steps I posted above to pinpoint the issue:

It would be helpful to see if the services have been created: kubectl get services -n linkerd.

And finally, can you confirm your k8s cluster has successfully set-up a DNS record for the headless identity service? I find that dig SRV works best here, but you could also look-up the hostname using nslookup. The Kubernetes official docs have a guide on how to debug DNS resolution.

[sathishkumar3009]

Output of kubectl get svc -n linkerd

NAME                                TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
linkerd-controller-api      ClusterIP   10.0.13.217   <none>        8085/TCP   23h
linkerd-dst                       ClusterIP   10.0.244.53   <none>        8086/TCP   23h
linkerd-dst-headless        ClusterIP   None          <none>        8086/TCP   23h
linkerd-identity                ClusterIP   10.0.43.176   <none>        8080/TCP   23h
linkerd-identity-headless ClusterIP   None          <none>        8080/TCP   23h
linkerd-proxy-injector      ClusterIP   10.0.180.79   <none>        443/TCP    23h
linkerd-sp-validator        ClusterIP   10.0.107.59   <none>        443/TCP    23h

[mateiidavid]

Have you managed to run dig or nslookup on the headless host (linkerd-dst-headless.linkerd.svc.cluster.local)?

[ayushiaks]

hi @mateiidavid, I'm facing the same issue, and nslookup is also working fine. Any further suggestions here?

kubectl exec -i -t dnsutils -- nslookup linkerd-identity-headless.linkerd.svc.cluster.local                                                                                  ─╯
Server:         10.0.0.10
Address:        10.0.0.10#53

Name:   linkerd-identity-headless.linkerd.svc.cluster.local
Address: 10.244.4.11

[sathishkumar3009]

I have followed the above mentioned page and got this as output.
kubectl exec -i -t dnsutils -- nslookup linkerd-dst-headless.linkerd.svc.cluster.local
Server: 10.0.0.10
Address: 10.0.0.10#53

Name: linkerd-dst-headless.linkerd.svc.cluster.local
Address: 192.168.2.185

[ayushiaks]

Hi @sathishkumar3009, how did you end up resolving the DNS issue here?
For me, the nslookup itself fails.

Edit: The lookup is working fine, but the control plane components are still failing

[sathishkumar3009]

@mateiidavid Please help me
I have injected linkerd for few of our services like vault and jenkins and there is no issue.

And I tried injected for Jfrog Artifactory, Pods are not at all coming up for services and also linkerd-proxy containers.

Below is the describe of pod and linkerd init container is getting failed.

   Reason:  iptables v1.8.2 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.

  linkerd-init:
    Container ID:  containerd://31192fc213ef1d26e8726fc3facf284d6458b4a19eb826cd115c3d45a40e8358
    Image:         cr.l5d.io/linkerd/proxy-init:v1.3.11
    Image ID:      cr.l5d.io/linkerd/proxy-init@sha256:9a7d17d2bce7274f5abfd611d34e24460a59c55e55ff93440c31ab6b60b542b2
    Port:          <none>
    Host Port:     <none>
    Args:
      --incoming-proxy-port
      4143
      --outgoing-proxy-port
      4140
      --proxy-uid
      2102
      --inbound-ports-to-ignore
      4190,4191
    State:       Waiting
      Reason:    CrashLoopBackOff
    Last State:  Terminated
      Reason:    Error
      Message:   /02/25 08:33:56 Will ignore port [4190 4191] on chain PROXY_INIT_REDIRECT
2022/02/25 08:33:56 Will redirect all INPUT ports to proxy
2022/02/25 08:33:56 Ignoring uid 2102
2022/02/25 08:33:56 Redirecting all OUTPUT to 4140
2022/02/25 08:33:56

2022/02/25 08:33:56 adding rules
------------------------------------------------------------
2022/02/25 08:33:56 :; iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/1645778036
2022/02/25 08:33:56 iptables v1.8.2 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.

2022/02/25 08:33:56 Aborting firewall configuration
Error: exit status 3

[mateiidavid]

Hey @sathishkumar3009, can you try to upgrade Linkerd to a recent edge version and set the following flag? --set "proxyInit.runAsRoot=true". It seems that iptables needs to run as root to be able to set things up.

Also, in the issue form, we typically suggest you include more details about your environment, not just your Kubernetes version. What OS are you using? Is this a managed cluster? We can make use of the information to figure out the problem here.

[sathishkumar3009]

@mateiidavid

I have set that value in helm and it is working now. Thanks. Jfrog application is also working as expected.

Yes we are using AKS managed cluster and linkerd is deployed using helm chart.

But I could see these logs in linkerd-proxy container. Any solution for this? For other applications logs are clear.

Keycloak Application, Linkerd Proxy container Logs:

[    36.863958s]  WARN ThreadId(01) server{port=8080}: linkerd_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) client.addr=10.1.0.98:46898
[    46.861631s]  WARN ThreadId(01) server{port=8080}: linkerd_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) client.addr=10.1.0.98:47180
[    57.863803s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=connection closed before message completed
[    67.862515s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=connection closed before message completed
[ 15431.363476s]  WARN ThreadId(01) server{orig_dst=10.0.0.1:443}: linkerd_reconnect::service: Service failed error=channel closed
[ 37000.127089s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=request timed out

Artifactory application, linkerd-proxy container logs:

[ 39493.116590s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=Unexpected TLS connection to linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local from 10.1.0.54:54306

Jenkins application, linkerd-proxy container logs:
[     0.065972s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: jenkins.jenkins.serviceaccount.identity.linkerd.cluster.local
[     1.049199s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=non-HTTP connection from 127.0.0.1:46700
[  7106.933937s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s

[sathishkumar3009]

@mateiidavid Can you help me please.

1. Similar issue was raised but no solution has been provided . Linkerd proxy container showing "Failed to proxy request: error trying to connect: Connection refused (os error 111)" #6248
2. Connection closed error=TCP Server service in fail-fast #6200

[adleong]

Hi @sathishkumar3009. Other than the log lines, is the application behaving correctly now?

[sathishkumar3009]

@adleong

 Some applications are working as expected but some are not.

 There is an application called Jfrog and which has xray capabilities.
  
 Jfrog application is working as expected but the xray pods are not get initialized due to below errors. Bot the application are in same namespace and communicates each other.

 Logs of linkerd-proxy container in XRAY pod.   

[     0.002380s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.002385s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.065909s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: xray.artifactory.serviceaccount.identity.linkerd.cluster.local
[     1.060204s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=non-HTTP connection from 127.0.0.1:38980
[    19.888190s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connection closed before message completedbilities as well.

       Logs of Xray container where linkerd is injected:

2022-03-07T09:57:51.202Z ←[33m[jfxr ]←[0m ←[1m←[31m[ERROR]←[0m [                ] [bin_mgr_http_service:139      ] [main                ] failed to get token for binMgr, err: failed to create token, failed to execute gRPC call CreateOrRefreshToken. The error is: rpc error: code = Internal desc = server closed the stream without sending trailers
2022-03-07T09:57:51.202Z ←[33m[jfxr ]←[0m ←[1m←[31m[ERROR]←[0m [                ] [bin_mgr_http_service:107      ] [main                ] failed to set token for xray, err: %!v(MISSING)
2022-03-07T09:57:51.202Z ←[33m[jfxr ]←[0m ←[1m←[31m[ERROR]←[0m [fa94fd4b0f66f76 ] [secure_manager_with_router:121] [main                ] Failed to set token for binary manager, err: failed to create token, failed to execute gRPC call CreateOrRefreshToken. The error is: rpc error: code = Internal desc = server closed the stream without sending trailers
2022-03-07T09:57:51.202Z ←[33m[jfxr ]←[0m ←[1m←[31m[ERROR]←[0m [fa94fd4b0f66f76 ] [start_xray_server:115         ] [main                ] failed to create token, failed to execute gRPC call CreateOrRefreshToken. The error is: rpc error: code = Internal desc = server closed the stream without sending trailers
2022-03-07T09:57:51.202Z ←[33m[jfxr ]←[0m ←[1m←[31m[ERROR]←[0m [fa94fd4b0f66f76 ] [general_utils:23              ] [main                ] Critical error: exiting from Xray Program, reason: failed to create token, failed to execute gRPC call CreateOrRefreshToken. The error is: rpc error: code = Internal desc = server closed the stream without sending trailers
2022-03-07T09:57:56.203Z ←[33m[jfxr ]←[0m [PANIC] [fa94fd4b0f66f76 ] [general_utils:19              ] [main                ] Exiting from Xray. goroutine 1 [running]:
runtime/debug.Stack(0x12a05f200, 0xc0000e8ee0, 0xde)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
jfrog.com/xray/pkg/utils.WaitAndExit(0x3675800, 0xc0019bf8c0)
        /go/src/jfrog.com/xray/pkg/utils/general_utils.go:19 +0x39
jfrog.com/xray/pkg/utils.WaitAndExitWithError(0x3675800, 0xc0019bf8c0, 0x363eca0, 0xc00cbfba90)
        /go/src/jfrog.com/xray/pkg/utils/general_utils.go:24 +0xd2
jfrog.com/xray/internal/startup.InitXrayServer(0x3675800, 0xc0019bf8c0)
        /go/src/jfrog.com/xray/internal/startup/start_xray_server.go:116 +0x151f
main.main()
        /go/src/jfrog.com/xray/cmd/server/server.go:10 +0x4e

[sathishkumar3009]

@adleong

 Issues are fixed after adding these annotations.

 config.linkerd.io/skip-inbound-ports: 0-65535
 config.linkerd.io/skip-outbound-ports: 0-65535

[mateiidavid]

@sathishkumar3009 it seems that the annotations you put on the workloads work but fyi this will end up skipping any proxying on ports 0 through 65535. This is fine if you don't need Linkerd's proxy for that specific workload, but this does mean you will not get any linkerd behaviour on the workload's pods either.

Looking at your logs, I see this:

[ 1.060204s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=non-HTTP connection from 127.0.0.1:38980

I'm not familiar with Xray, is it trying to talk to your main application container? Does it run as a side car? More details on your set-up would help us troubleshoot and figure this out. It moreover seems that the X-ray container is trying to make a call to refresh a token? Does it need to talk to the k8s apiserver, or?

[sathishkumar3009]

@mateiidavid

 1. Single name space for Jfrog artifactory
 2.  Artifactory container is running in separate pod.
 3.  XRAY container is running in separate pod.
 4.  Xray is deployed as a separate helm charts from Jfrog and connected with artificatory.
 5. As you mentioned Xray will interact with the Artifactory container or pods.
 6. It is not required to communicate with k8s apiserver.

[mateiidavid]

It seems in your logs that Xray fails to create or refresh its token? What is supposed to happen there? And what server is it supposed to interact with? Is your server shutting down its half of the connection abruptly? Have you tried switching the log level on the proxy to linkerd=debug,warn? https://linkerd.io/2.11/tasks/modifying-proxy-log-level/ this should help you with the log level.