Connection error when inject linkerd proxy

[huyturing]

linkerd check
kubernetes-api

---

√ can initialize the client
√ can query the Kubernetes API

kubernetes-version

√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence

√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config

√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity

√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-api

√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version

√ can determine the latest version
‼ cli is up-to-date
is running version 2.7.0 but the latest stable version is 2.7.1
see https://linkerd.io/checks/#l5d-version-cli for hints

control-plane-version

‼ control plane is up-to-date
is running version 2.7.0 but the latest stable version is 2.7.1
see https://linkerd.io/checks/#l5d-version-control for hints
√ control plane and cli versions match

Status check results are √

What is the issue?

I could not access to the application container after injecting linked proxy.

Linkerd proxy log:

[ 0.9691467s] INFO linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.9715950s] INFO linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.9719432s] INFO linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.9748461s] INFO linkerd2_proxy: Tap interface on 0.0.0.0:4190
[ 0.9751229s] INFO linkerd2_proxy: Local identity is default.workspace-develop.serviceaccount.identity.linkerd.cluster.local
[ 0.9758117s] INFO linkerd2_proxy: Identity verified via linkerd-identity.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.9761020s] INFO linkerd2_proxy: Destinations resolved via linkerd-dst.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.10407485s] INFO inbound: linkerd2_app_inbound: serving listen.addr=0.0.0.0:4143
[ 0.48536067s] INFO daemon:identity: linkerd2_app: Certified identity: default.workspace-develop.serviceaccount.identity.linkerd.cluster.local
[ 97.852191701s] WARN inbound:accept{peer.addr=10.52.2.50:41982}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 98.693787322s] WARN inbound:accept{peer.addr=10.52.2.49:35494}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 99.251036154s] WARN inbound:accept{peer.addr=10.52.2.50:41982}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 101.15255988s] WARN inbound:accept{peer.addr=10.52.2.49:35494}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 114.655722534s] WARN inbound:accept{peer.addr=10.52.2.50:41982}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 115.17375554s] WARN inbound:accept{peer.addr=10.52.2.49:35744}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 115.576425512s] WARN inbound:accept{peer.addr=10.52.2.50:41982}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 117.296066771s] WARN inbound:accept{peer.addr=10.52.2.53:48300}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 787.3892515s] WARN inbound:accept{peer.addr=10.52.2.51:56628}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 787.878268854s] WARN inbound:accept{peer.addr=10.52.2.50:51864}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 789.362538705s] WARN inbound:accept{peer.addr=10.52.2.50:51896}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)
[ 791.551621598s] WARN inbound:accept{peer.addr=10.52.2.49:45422}:source{peer.id=None(NoPeerName(NotProvidedByRemote)) target.addr=10.52.2.163:3535}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111) (address: 127.0.0.1:3535)

[huyturing]

it looks like when I inject the linkerd-proxy, my application can start the process, but can not bind the port T.T

this is strange, but when I remove linked, it's back to normal

[grampelberg]

Is your app listening on 3535? Do you have a readinessProbe configured?

[huyturing]

yes, my app is listening on 3535. I don't have readinessProbe configured. After getting this error, I tried to use both readiness and licenses configure to check but still the same error.

When I jumped in the application container, I ran netstat to check if the port is listened or not, and it was not. Tried to start the server manually in the container did not help.

But when I stopped the linkerd-proxy, it worked back, netstat show the listened port.

[grampelberg]

What is in your application logs? We're not doing anything that would stop you from listening on a port.

[huyturing]

yeah, but it actually happened.