linkerd-await hangs initially but executed successfully on pod restart

[abekman-epam]

Hello everyone! I'm looking for help with a problem I encountered while upgrading k8s cluster (v1.19, EKS) from Linkerd 2.10.2 to stable-2.11.2
The problem is linkerd pods (destination & proxy-injector) are stuck for some time until linkerd-await timeout (2m) is expired and then got restarted. After that they start ok. If I manually modify deployments not to use linkerd-await they start ok too.
After that linkerd check result is green (except for "‼ cluster networks can be verified\n the following nodes do not expose a podCIDR:" but doesn't look relevant) Any suggestions on what can be checked to fix this behavior?

[mateiidavid]

Hey @abekman-epam, if linkerd-await times out, it means the proxy did not get into a ready state. Generally, the proxy is not ready when it cannot acquire identity, which is the first thing it attempts to do once the process is started. Removing await was a good first step -- since it's a preStart hook, it blocks other containers from starting -- however, it is only a symptom of the problem.

I would start by collecting the proxy logs (with a debug level, i.e linkerd=debug,warn) for the destination pod to figure out why things are going wrong.

[abekman-epam]

I see following errors, but can't understand what can cause them, as identity service is present

[     0.000644s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.001132s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.001146s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.001150s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.001153s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.001155s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.001158s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.001161s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.001596s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.106060s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.322536s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.746147s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.247838s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.749599s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.250322s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.751018s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.252735s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.754453s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.255162s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.756858s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.258546s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.759244s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     6.259920s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     6.761610s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     7.262869s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     7.764601s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     8.265369s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     8.766043s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     9.267721s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     9.769380s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    10.270707s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    10.772427s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    11.273142s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    11.774826s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    12.276493s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    12.777250s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    13.277998s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    13.779369s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    14.280101s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    14.781787s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    15.002119s]  WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized...
[    15.004635s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=request timed out
[    15.283594s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    15.784315s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    16.285047s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    16.786784s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    17.288512s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    17.789160s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    18.290844s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)

[danibaeyens]

I have the exact same symptoms and also the same log output.
I also have the podCIDR warning, and pods stay in ContainerCreating until linkerd-await times out. Once it gets restarted, containers immediately start. Identity pods are up and running and there are no changes in the cluster while it times out.

Any other pod which I mesh suffer from the same thing. Does not start until linkerd-await times out.

If I disable linkerd-await and get the linkerd-proxy logs, I get the following output

$ kubectl logs -n linkerd linkerd-destination-557c46d88d-bpb9s linkerd-proxy 
[     0.000549s] DEBUG ThreadId(01) linkerd_app::env: Only allowing connections targeting `LINKERD2_PROXY_INBOUND_IPS` allowed={100.72.39.43}
[     0.000780s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.000878s] DEBUG ThreadId(01) linkerd_app: building app
[     0.001357s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.001368s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.001371s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.001373s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.001375s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.001376s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.001379s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.001381s] DEBUG ThreadId(01) linkerd_app: spawning daemon thread
[     0.001614s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}: linkerd_proxy_discover::from_resolve: Changed change=Insert(127.0.0.1:8090, Target { addr: 127.0.0.1:8090, server_id: None(Loopback) })
[     0.001647s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=false
[     0.001653s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=false
[     0.001670s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     0.001675s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     0.001804s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.001822s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     0.001834s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.002466s] DEBUG ThreadId(02) daemon: linkerd_app: running admin thread
[     0.002512s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_proxy_identity::certify: Identity daemon running
[     0.002651s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_dns: resolve_srv name=linkerd-identity-headless.linkerd.svc.cluster.local.
[     0.105007s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.105028s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     0.105033s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     0.105038s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     0.105189s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.105205s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     0.105209s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.318479s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.318499s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     0.318504s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     0.318509s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     0.318630s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.318646s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     0.318649s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.720149s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.720183s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     0.720190s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     0.720196s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     0.720305s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.720322s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     0.720326s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     1.221922s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     1.221945s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     1.221949s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     1.221954s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     1.222091s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.222107s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     1.222110s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     1.723275s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     1.723297s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     1.723302s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     1.723307s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     1.723420s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.723436s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     1.723440s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     2.224006s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     2.224030s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     2.224036s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     2.224041s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     2.224159s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.224200s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     2.224205s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     2.725787s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     2.725819s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     2.725827s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     2.725833s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     2.725985s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.726013s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     2.726019s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     3.227625s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     3.227654s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     3.227660s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     3.227666s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     3.227783s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}:h2: linkerd_proxy_transport::connect: Connected local.addr=127.0.0.1:34808 keepalive=None
[     3.227919s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Connected
[     3.228481s] DEBUG ThreadId(01) policy:watch{port=9996}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.228509s] DEBUG ThreadId(01) policy:watch{port=9996}: linkerd_tonic_watch: Recovering
[     3.228543s] DEBUG ThreadId(01) policy:watch{port=8086}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.228559s] DEBUG ThreadId(01) policy:watch{port=8086}: linkerd_tonic_watch: Recovering
[     3.228573s] DEBUG ThreadId(01) policy:watch{port=9997}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.228584s] DEBUG ThreadId(01) policy:watch{port=9997}: linkerd_tonic_watch: Recovering
[     3.228621s] DEBUG ThreadId(01) policy:watch{port=8443}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.228632s] DEBUG ThreadId(01) policy:watch{port=8443}: linkerd_tonic_watch: Recovering
[     3.229571s] DEBUG ThreadId(01) policy:watch{port=9443}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.229591s] DEBUG ThreadId(01) policy:watch{port=9443}: linkerd_tonic_watch: Recovering
[     3.229603s] DEBUG ThreadId(01) policy:watch{port=9990}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.229608s] DEBUG ThreadId(01) policy:watch{port=9990}: linkerd_tonic_watch: Recovering
[     3.229619s] DEBUG ThreadId(01) policy:watch{port=4191}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.229625s] DEBUG ThreadId(01) policy:watch{port=4191}: linkerd_tonic_watch: Recovering
[     3.229652s] DEBUG ThreadId(01) policy:watch{port=8090}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.229663s] DEBUG ThreadId(01) policy:watch{port=8090}: linkerd_tonic_watch: Recovering
[     3.330257s] DEBUG ThreadId(01) policy:watch{port=9443}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.330279s] DEBUG ThreadId(01) policy:watch{port=9443}: linkerd_tonic_watch: Recovering
[     3.330312s] DEBUG ThreadId(01) policy:watch{port=8086}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.330321s] DEBUG ThreadId(01) policy:watch{port=8086}: linkerd_tonic_watch: Recovering
[     3.332553s] DEBUG ThreadId(01) policy:watch{port=9990}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.332565s] DEBUG ThreadId(01) policy:watch{port=9990}: linkerd_tonic_watch: Recovering
[     3.333779s] DEBUG ThreadId(01) policy:watch{port=4191}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.333791s] DEBUG ThreadId(01) policy:watch{port=4191}: linkerd_tonic_watch: Recovering
[     3.333832s] DEBUG ThreadId(01) policy:watch{port=9997}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.333844s] DEBUG ThreadId(01) policy:watch{port=9997}: linkerd_tonic_watch: Recovering
[     3.336049s] DEBUG ThreadId(01) policy:watch{port=8443}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.336060s] DEBUG ThreadId(01) policy:watch{port=8443}: linkerd_tonic_watch: Recovering
[     3.338255s] DEBUG ThreadId(01) policy:watch{port=9996}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.338266s] DEBUG ThreadId(01) policy:watch{port=9996}: linkerd_tonic_watch: Recovering
[     3.340472s] DEBUG ThreadId(01) policy:watch{port=8090}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.340484s] DEBUG ThreadId(01) policy:watch{port=8090}: linkerd_tonic_watch: Recovering
[     3.535081s] DEBUG ThreadId(01) policy:watch{port=8086}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.535102s] DEBUG ThreadId(01) policy:watch{port=8086}: linkerd_tonic_watch: Recovering
[     3.538378s] DEBUG ThreadId(01) policy:watch{port=9443}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.538393s] DEBUG ThreadId(01) policy:watch{port=9443}: linkerd_tonic_watch: Recovering
[     3.541612s] DEBUG ThreadId(01) policy:watch{port=9997}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.541626s] DEBUG ThreadId(01) policy:watch{port=9997}: linkerd_tonic_watch: Recovering
[     3.545849s] DEBUG ThreadId(01) policy:watch{port=9996}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.545863s] DEBUG ThreadId(01) policy:watch{port=9996}: linkerd_tonic_watch: Recovering
[     3.548079s] DEBUG ThreadId(01) policy:watch{port=9990}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.548092s] DEBUG ThreadId(01) policy:watch{port=9990}: linkerd_tonic_watch: Recovering
[     3.550327s] DEBUG ThreadId(01) policy:watch{port=4191}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.550340s] DEBUG ThreadId(01) policy:watch{port=4191}: linkerd_tonic_watch: Recovering
[     3.551546s] DEBUG ThreadId(01) policy:watch{port=8090}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.551559s] DEBUG ThreadId(01) policy:watch{port=8090}: linkerd_tonic_watch: Recovering
[     3.556794s] DEBUG ThreadId(01) policy:watch{port=8443}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.556808s] DEBUG ThreadId(01) policy:watch{port=8443}: linkerd_tonic_watch: Recovering
[     3.939323s] DEBUG ThreadId(01) policy:watch{port=9443}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.939351s] DEBUG ThreadId(01) policy:watch{port=9443}: linkerd_tonic_watch: Recovering
[     3.954757s] DEBUG ThreadId(01) policy:watch{port=8086}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.954783s] DEBUG ThreadId(01) policy:watch{port=8086}: linkerd_tonic_watch: Recovering
[     3.963128s] DEBUG ThreadId(01) policy:watch{port=9997}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.963160s] DEBUG ThreadId(01) policy:watch{port=9997}: linkerd_tonic_watch: Recovering
[     3.968450s] DEBUG ThreadId(01) policy:watch{port=9990}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.968466s] DEBUG ThreadId(01) policy:watch{port=9990}: linkerd_tonic_watch: Recovering
[     3.969672s] DEBUG ThreadId(01) policy:watch{port=9996}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.969686s] DEBUG ThreadId(01) policy:watch{port=9996}: linkerd_tonic_watch: Recovering
[     3.982928s] DEBUG ThreadId(01) policy:watch{port=8443}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.982943s] DEBUG ThreadId(01) policy:watch{port=8443}: linkerd_tonic_watch: Recovering
[     3.984147s] DEBUG ThreadId(01) policy:watch{port=4191}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.984159s] DEBUG ThreadId(01) policy:watch{port=4191}: linkerd_tonic_watch: Recovering
[     3.988386s] DEBUG ThreadId(01) policy:watch{port=8090}: linkerd_tonic_watch: Request failed status=status: NotFound, message: "unknown server", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 24 May 2022 04:44:07 GMT"} }
[     3.988400s] DEBUG ThreadId(01) policy:watch{port=8090}: linkerd_tonic_watch: Recovering
[     4.440352s] DEBUG ThreadId(01) policy:watch{port=9443}: linkerd_app_inbound::policy::discover: policy=ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" }
[     4.455810s] DEBUG ThreadId(01) policy:watch{port=8086}: linkerd_app_inbound::policy::discover: policy=ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" }
[     4.465224s] DEBUG ThreadId(01) policy:watch{port=9997}: linkerd_app_inbound::policy::discover: policy=ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" }
[     4.469539s] DEBUG ThreadId(01) policy:watch{port=9990}: linkerd_app_inbound::policy::discover: policy=ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" }
[     4.470844s] DEBUG ThreadId(01) policy:watch{port=9996}: linkerd_app_inbound::policy::discover: policy=ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" }
[     4.484218s] DEBUG ThreadId(01) policy:watch{port=8443}: linkerd_app_inbound::policy::discover: policy=ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" }
[     4.485489s] DEBUG ThreadId(01) policy:watch{port=4191}: linkerd_app_inbound::policy::discover: policy=ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" }
[     4.489748s] DEBUG ThreadId(01) policy:watch{port=8090}: linkerd_app_inbound::policy::discover: policy=ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" }
[     4.899492s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=10.150.13.245:34072}: linkerd_tls::server: Peeked bytes from TCP stream sz=109
[     4.899553s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=10.150.13.245:34072}: linkerd_detect: DetectResult protocol=Some(Http1) elapsed=18.726µs
[     4.899572s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=10.150.13.245:34072}: linkerd_proxy_http::server: Creating HTTP service version=Http1
[     4.899592s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=10.150.13.245:34072}: linkerd_proxy_http::server: Handling as HTTP version=Http1
[     4.899781s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=10.150.13.245:34072}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(0.0.0.0:4191), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=None(NoClientHello) client=10.150.13.245:34072
[     4.899915s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=10.150.13.245:34072}: linkerd_proxy_http::server: The client is shutting down the connection res=Ok(())
[     4.899957s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=10.150.13.245:34072}: linkerd_app_core::serve: Connection closed
[     5.004811s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_dns: ttl=4.999987125s addrs=[100.72.36.64:8080, 100.72.38.41:8080]
[     5.004833s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_proxy_dns_resolve: addrs=[100.72.36.64:8080, 100.72.38.41:8080] name=linkerd-identity-headless.linkerd.svc.cluster.local:8080
[     5.004874s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_proxy_discover::from_resolve: Changed change=Insert(100.72.36.64:8080, Target { addr: 100.72.36.64:8080, server_id: Some(ClientTls { server_id: ServerId(Name("linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local")), alpn: None }) })
[     5.004898s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_proxy_discover::from_resolve: Changed change=Insert(100.72.38.41:8080, Target { addr: 100.72.38.41:8080, server_id: Some(ClientTls { server_id: ServerId(Name("linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local")), alpn: None }) })
[     5.004916s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.36.64:8080}: linkerd_reconnect: Disconnected backoff=false
[     5.004920s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.36.64:8080}: linkerd_reconnect: Creating service backoff=false
[     5.004927s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.36.64:8080}: linkerd_tls::client: Initiating TLS connection server.id=linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
[     5.004935s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.36.64:8080}: linkerd_proxy_transport::connect: Connecting server.addr=100.72.36.64:8080
[     5.005048s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.38.41:8080}: linkerd_reconnect: Disconnected backoff=false
[     5.005057s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.38.41:8080}: linkerd_reconnect: Creating service backoff=false
[     5.005061s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.38.41:8080}: linkerd_tls::client: Initiating TLS connection server.id=linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
[     5.005065s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.38.41:8080}: linkerd_proxy_transport::connect: Connecting server.addr=100.72.38.41:8080
[     5.005313s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.36.64:8080}:h2: linkerd_proxy_transport::connect: Connected local.addr=100.72.39.43:35138 keepalive=Some(10s)
[     5.005492s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.38.41:8080}:h2: linkerd_proxy_transport::connect: Connected local.addr=100.72.39.43:57092 keepalive=Some(10s)
[     5.006564s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=100.72.36.64:8080}: linkerd_reconnect: Connected
[     5.010817s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_identity: certified linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     5.010832s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_proxy_identity::certify: daemon certified until SystemTime { tv_sec: 1653453869, tv_nsec: 0 }
[     5.010869s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     5.010977s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}: linkerd_app_inbound::accept: Accepted policy=AllowPolicy { dst: OrigDstAddr(100.72.39.43:9996), server: Receiver { shared: Shared { value: RwLock(PhantomData, RwLock { data: ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" } }), state: AtomicState(2), ref_count_rx: 2, notify_rx: Notify { state: 4, waiters: Mutex(PhantomData, Mutex { data: LinkedList { head: None, tail: None } }) }, notify_tx: Notify { state: 1, waiters: Mutex(PhantomData, Mutex { data: LinkedList { head: Some(0x7feb2d0bab78), tail: Some(0x7feb2d0bab78) } }) } }, version: Version(0) } }
[     5.011039s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}: linkerd_app_inbound::accept: Accepted policy=AllowPolicy { dst: OrigDstAddr(100.72.39.43:9997), server: Receiver { shared: Shared { value: RwLock(PhantomData, RwLock { data: ServerPolicy { protocol: Detect { timeout: 10s }, authorizations: [Authorization { networks: [Network { net: 0.0.0.0/0, except: [] }, Network { net: ::/0, except: [] }], authentication: Unauthenticated, name: "default:all-unauthenticated" }, Authorization { networks: [Network { net: 127.0.0.1/32, except: [] }, Network { net: ::1/128, except: [] }], authentication: Unauthenticated, name: "default:localhost" }], name: "default:all-unauthenticated" } }), state: AtomicState(2), ref_count_rx: 2, notify_rx: Notify { state: 4, waiters: Mutex(PhantomData, Mutex { data: LinkedList { head: None, tail: None } }) }, notify_tx: Notify { state: 1, waiters: Mutex(PhantomData, Mutex { data: LinkedList { head: Some(0x7feb2d0b8d78), tail: Some(0x7feb2d0b8d78) } }) } }, version: Version(0) } }
[     5.011089s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}: linkerd_tls::server: Peeked bytes from TCP stream sz=109
[     5.011115s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}: linkerd_detect: DetectResult protocol=Some(Http1) elapsed=9.994µs
[     5.011126s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}: linkerd_proxy_http::server: Creating HTTP service version=Http1
[     5.011154s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}: linkerd_proxy_http::server: Handling as HTTP version=Http1
[     5.011243s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(100.72.39.43:9996), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=None(NoClientHello) client=10.150.13.245:55632
[     5.011293s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}:http1: linkerd_cache: Caching new service target=Logical { logical: None, addr: Remote(ServerAddr(100.72.39.43:9996)), http: Http1, tls: None(NoClientHello), permit: Permit { dst: OrigDstAddr(100.72.39.43:9996), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } }, labels: {"saz_name": "default:all-unauthenticated", "srv_name": "default:all-unauthenticated"} }
[     5.011352s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}: linkerd_tls::server: Peeked bytes from TCP stream sz=109
[     5.011368s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}: linkerd_detect: DetectResult protocol=Some(Http1) elapsed=3.445µs
[     5.011377s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}: linkerd_proxy_http::server: Creating HTTP service version=Http1
[     5.011385s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}: linkerd_proxy_http::server: Handling as HTTP version=Http1
[     5.011408s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}:http{v=1.x}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(100.72.39.43:9997), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } } tls=None(NoClientHello) client=10.150.13.245:35122
[     5.011428s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}:http{v=1.x}:http1: linkerd_cache: Caching new service target=Logical { logical: None, addr: Remote(ServerAddr(100.72.39.43:9997)), http: Http1, tls: None(NoClientHello), permit: Permit { dst: OrigDstAddr(100.72.39.43:9997), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } }, labels: {"saz_name": "default:all-unauthenticated", "srv_name": "default:all-unauthenticated"} }
[     5.011466s] DEBUG ThreadId(01) evict{target=Logical { logical: None, addr: Remote(ServerAddr(100.72.39.43:9996)), http: Http1, tls: None(NoClientHello), permit: Permit { dst: OrigDstAddr(100.72.39.43:9996), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } }, labels: {"saz_name": "default:all-unauthenticated", "srv_name": "default:all-unauthenticated"} }}: linkerd_cache: Awaiting idleness
[     5.011486s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}:http1:profile: linkerd_service_profiles::default: Handling rejected discovery error=discovery rejected: inbound profile discovery requires DNS names
[     5.011501s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}:http1:profile: linkerd_reconnect: Disconnected backoff=false
[     5.011509s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}:http1:profile: linkerd_reconnect: Creating service backoff=false
[     5.011515s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}:http1:profile: linkerd_proxy_http::client: Building HTTP client settings=Http1
[     5.011521s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}:http1:profile: linkerd_reconnect: Connected
[     5.011535s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}:http1:profile:http1: linkerd_proxy_http::client: method=GET uri=http://100.72.39.43:9996/ready version=HTTP/1.1
[     5.011542s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}:http1:profile:http1: linkerd_proxy_http::client: headers={"host": "100.72.39.43:9996", "user-agent": "kube-probe/1.22", "accept": "*/*"}
[     5.011549s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:55632}:server{port=9996}:http{v=1.x}:http1:profile:http1: linkerd_proxy_http::h1: Caching new client use_absolute_form=false
[     5.011564s] DEBUG ThreadId(01) evict{target=Logical { logical: None, addr: Remote(ServerAddr(100.72.39.43:9997)), http: Http1, tls: None(NoClientHello), permit: Permit { dst: OrigDstAddr(100.72.39.43:9997), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel("default:all-unauthenticated"), authz: "default:all-unauthenticated" } }, labels: {"saz_name": "default:all-unauthenticated", "srv_name": "default:all-unauthenticated"} }}: linkerd_cache: Awaiting idleness
[     5.011574s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}:http{v=1.x}:http1:profile: linkerd_service_profiles::default: Handling rejected discovery error=discovery rejected: inbound profile discovery requires DNS names
[     5.011579s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}:http{v=1.x}:http1:profile: linkerd_reconnect: Disconnected backoff=false
[     5.011581s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}:http{v=1.x}:http1:profile: linkerd_reconnect: Creating service backoff=false
[     5.011584s] DEBUG ThreadId(01) inbound:accept{client.addr=10.150.13.245:35122}:server{port=9997}:http{v=1.x}:http1:profile: linkerd_proxy_http::client: Building HTTP client settings=Http1

[danibaeyens]

Thanks!
It looks like it's using the fallback mechanism, but still unable to connect. It seems that it's also detecting the connection but still not working (stuck in ContainerCreating) until restart, then it goes fine

Log of the first run:

[     0.002867s] DEBUG ThreadId(01) linkerd_app::env: Only allowing connections targeting `LINKERD2_PROXY_INBOUND_IPS` allowed={100.72.37.105}
[     0.003200s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.003310s] DEBUG ThreadId(01) linkerd_app: building app
[     0.004093s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.004104s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.004107s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.004110s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.004114s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.004117s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.004120s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.004124s] DEBUG ThreadId(01) linkerd_app: spawning daemon thread
[     0.004465s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}: linkerd_proxy_discover::from_resolve: Changed change=Insert(127.0.0.1:8090, Target { addr: 127.0.0.1:8090, server_id: None(Loopback) })
[     0.004530s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=false
[     0.004540s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=false
[     0.004581s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     0.004597s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     0.004851s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.004889s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     0.004898s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.005456s] DEBUG ThreadId(02) daemon: linkerd_app: running admin thread
[     0.005521s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_proxy_identity_client::certify: Identity daemon running
[     0.005537s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_proxy_identity_client::certify: Certifying identity
[     0.005699s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_dns: Resolving a SRV record name=linkerd-identity-headless.linkerd.svc.cluster.local.
[     0.094571s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=127.0.0.1:56272}: linkerd_tls::server: Peeked bytes from TCP stream sz=45
[     0.094667s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=127.0.0.1:56272}: linkerd_detect: DetectResult protocol=Some(Http1) elapsed=24.431µs
[     0.094694s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=127.0.0.1:56272}: linkerd_proxy_http::server: Creating HTTP service version=Http1
[     0.094723s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=127.0.0.1:56272}: linkerd_proxy_http::server: Handling as HTTP version=Http1
[     0.094880s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=127.0.0.1:56272}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(0.0.0.0:4191), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel { kind: "default", name: "all-unauthenticated" }, kind: "default", name: "all-unauthenticated" } } tls=None(NoClientHello) client=127.0.0.1:56272
[     0.107488s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.107508s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     0.107514s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     0.107520s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     0.107634s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.107658s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     0.107664s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.316053s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.316070s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     0.316075s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     0.316080s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     0.316188s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.316230s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     0.316236s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.746628s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     0.746657s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     0.746663s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     0.746669s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     0.746820s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.746848s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     0.746853s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     1.096781s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=127.0.0.1:56272}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(0.0.0.0:4191), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel { kind: "default", name: "all-unauthenticated" }, kind: "default", name: "all-unauthenticated" } } tls=None(NoClientHello) client=127.0.0.1:56272
[     1.247580s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     1.247618s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     1.247630s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     1.247640s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     1.247831s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.247854s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     1.247860s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     1.748584s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     1.748613s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     1.748620s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     1.748626s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     1.748759s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.748786s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     1.748792s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     2.098290s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=127.0.0.1:56272}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(0.0.0.0:4191), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel { kind: "default", name: "all-unauthenticated" }, kind: "default", name: "all-unauthenticated" } } tls=None(NoClientHello) client=127.0.0.1:56272
[     2.249522s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     2.249548s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     2.249555s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     2.249560s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     2.249704s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.249730s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     2.249735s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     2.751466s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     2.751512s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     2.751525s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     2.751536s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     2.751759s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.751802s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     2.751812s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     3.099788s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=127.0.0.1:56272}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(0.0.0.0:4191), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel { kind: "default", name: "all-unauthenticated" }, kind: "default", name: "all-unauthenticated" } } tls=None(NoClientHello) client=127.0.0.1:56272
[     3.252533s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     3.252560s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     3.252566s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     3.252573s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     3.252722s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.252747s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     3.252752s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     3.753494s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[     3.753521s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[     3.753528s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[     3.753535s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[     3.753656s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.753680s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[     3.753685s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true

[...] Continues retrying until [...]

[    15.005928s]  WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized...
[    15.008638s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_dns: Falling back to A record lookup srv.error=failed to resolve SRV record: request timed out
[    15.008659s] DEBUG ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_dns: Resolving an A record name=linkerd-identity-headless.linkerd.svc.cluster.local.
[    15.118751s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=127.0.0.1:56272}: linkerd_app_inbound::policy::authorize::http: Request authorized permit=Permit { dst: OrigDstAddr(0.0.0.0:4191), protocol: Detect { timeout: 10s }, labels: AuthzLabels { server: ServerLabel { kind: "default", name: "all-unauthenticated" }, kind: "default", name: "all-unauthenticated" } } tls=None(NoClientHello) client=127.0.0.1:56272
[    15.283450s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[    15.283488s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[    15.283500s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[    15.283510s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[    15.283708s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    15.283731s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[    15.283737s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[    15.784462s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true
[    15.784491s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Creating service backoff=true
[    15.784498s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_tls::client: Peer does not support TLS reason=loopback
[    15.784505s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_proxy_transport::connect: Connecting server.addr=127.0.0.1:8090
[    15.784633s]  WARN ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    15.784657s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Recovering
[    15.784663s] DEBUG ThreadId(01) policy:watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Disconnected backoff=true

[danibaeyens]

By the way, seeing your comments here: #7493 I'm using a different domain than cluster.local (although I'm replacing the domain on the logs). Still, as I added on a previous comment, it seems to resolve fine using the custom domain (also, identity works fine, as the second run works fine providing the right certificates with the custom domain)

[danibaeyens]

I've just found out this issue in AKS Azure/AKS#2750 , and I can fully reproduce it with the pod.yaml attached.

Interestingly, we are not in AKS, but a custom cluster on AWS with EC2 (not EKS), with AWS CNI and calico for network policies

If I remove the lifecycle section, it starts normally.

I've also found this one: aws/amazon-vpc-cni-k8s#1790 which seems related and without resolution

[danibaeyens]

Uhmm...
Could it be this one? projectcalico/calico#3499
Does it make sense @alpeb ?

@abekman-epam do you also use calico (maybe for the network policies)?

[abekman-epam]

@danibaeyens Yes, I use calico for network policies.

[danibaeyens]

@olix0r Just found out that there is a workaround for this issue:
openshift/cluster-network-operator@cbacf6d#diff-95566c5e08594ee74004728f0a4910304a230cd024f4fd102f16396c693b88ea

I've just tried, and works fine :)

[kleimkuhler]

Thanks for the investigation on this @danibaeyens. #9310 was recently opened to at least track the work on this so that this discussion doesn't stale out again.