getting "Sending fatal alert AccessDenied on startup"

[sumit-joshi-mt]

Hi,
I have one application which uses linkerd on kubernetes. On startup of pod I get following warning in linkerd proxy due to this in application I get connect: connection refused error for few seconds after which it gets resolved automatically This happens only with one pod which is created first, every time application is deployed:

Linkerd proxy version: edge-20.9.1

Error log in application:
syslog.log:{"level":"error","ts":1604298910.9524453,"msg":"Error in pinging serverrpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.20.29.171:80: connect: connection refused\"","host":"tdb-svc-loadgen-866c76c886-n4jcj","ReqId":""}

Linkerd-proxy log:

[     0.8540537s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.8579757s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.8585095s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.8588865s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.8592415s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.load-test.serviceaccount.identity.linkerd.cluster.local
[     0.8599837s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.8603816s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.9217697s]  INFO ThreadId(01) outbound: linkerd2_app_outbound: Serving addr=127.0.0.1:4140
[     0.9283841s]  INFO ThreadId(01) inbound: linkerd2_app_inbound: Serving addr=0.0.0.0:4143
[     0.48871691s]  WARN ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42030}: rustls::session: Sending fatal alert AccessDenied    
[     0.48990698s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42030}: linkerd2_app_core::serve: Connection closed error=unexpected error: no server certificate chain resolved
[     0.162941899s]  WARN ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42032}: rustls::session: Sending fatal alert AccessDenied    
[     0.163053192s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42032}: linkerd2_app_core::serve: Connection closed error=unexpected error: no server certificate chain resolved
[     0.371914523s]  WARN ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42038}: rustls::session: Sending fatal alert AccessDenied    
[     0.371997662s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42038}: linkerd2_app_core::serve: Connection closed error=unexpected error: no server certificate chain resolved
[     0.784393414s]  WARN ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42048}: rustls::session: Sending fatal alert AccessDenied    
[     0.784499315s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42048}: linkerd2_app_core::serve: Connection closed error=unexpected error: no server certificate chain resolved
[     1.287692402s]  WARN ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42058}: rustls::session: Sending fatal alert AccessDenied    
[     1.287795231s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.11.154.19:42058}: linkerd2_app_core::serve: Connection closed error=unexpected error: no server certificate chain resolved
[     1.572862034s]  INFO ThreadId(02) daemon:identity: linkerd2_app: Certified identity: default.load-test.serviceaccount.identity.linkerd.cluster.local
[     7.851052197s]  WARN ThreadId(01) trust_dns_proto::xfer::dns_exchange: io_stream hit an error, shutting down: io error: Connection reset by peer (os error 104)``

________________________________________________________________________________________________

Linkerd check output: 
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
    kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if HA mode is enabled
    see https://linkerd.io/checks/#l5d-injection-disabled for hints
√ multiple replicas of control plane pods

linkerd-addons
--------------
√ 'linkerd-config-addons' config map exists

linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

Status check results are

Additional information:
I tried with edge-20.10.4 also, with this release getting in all pods of application.

[olix0r]

@sumit-joshi-mt Does your application start sending requests immediately on startup? If so, it's possible that it's trying to do so before the proxy sidecar has initialized. This is a fundamental issue with Kubernetes -- there is no way to enforce ordering around process startup. We're eager for kubernetes to provide better primitives for sidecar containers, but in the meantime, you might be able to use linkerd-await to help your process wait until the proxy has been initialized.