linkerd-proxy sidecar container terminated after scaling up the deployments

[zhoudaxia233]

I am encountering an issue with linkerd-proxy sidecar container termination when scaling up my Kubernetes deployments. Prior to scaling up, my application with 3 replicas and linkerd-proxy sidecar containers worked without issues. However, after scaling up to 5 replicas, the sidecar container in the extra 2 replicas failed to function correctly.

I have checked the logs and found that the sidecar container terminates without any apparent reason. The application container in the same pod continues to work as expected. How can I troubleshoot and fix this issue?

time="2023-02-15T12:19:40Z" level=info msg="Found pre-existing key: /var/run/linkerd/identity/end-entity/key.p8"
time="2023-02-15T12:19:40Z" level=info msg="Found pre-existing CSR: /var/run/linkerd/identity/end-entity/csr.der"
[     0.001959s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.002661s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.002675s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.002677s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.002679s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.002681s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.default.serviceaccount.identity.linkerd.cluster.local
[     0.002684s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.002686s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.003925s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.71:8080}: rustls::conn: Sending fatal alert BadCertificate    
[     0.003925s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.72:8090}: rustls::conn: Sending fatal alert BadCertificate    
[     0.004002s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.71:8080}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     0.004002s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.72:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     0.115485s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.71:8080}: rustls::conn: Sending fatal alert BadCertificate    
[     0.115763s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.71:8080}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     0.116304s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.72:8090}: rustls::conn: Sending fatal alert BadCertificate    
[     0.116575s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.72:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     0.324315s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.71:8080}: rustls::conn: Sending fatal alert BadCertificate    
[     0.324591s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.71:8080}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     0.332870s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.72:8090}: rustls::conn: Sending fatal alert BadCertificate    
[     0.333113s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.72:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     0.744679s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.72:8090}: rustls::conn: Sending fatal alert BadCertificate

Thank you for your help!

[mateiidavid]

Can you run through linkerd check? It sounds like the newly created pods can't get their certificates created. Likely the issuer has expired?

[zhoudaxia233]

I just run linkerd check and all checks are passed √

[zhoudaxia233]

I've also run linkerd check --proxy, two checks failed:

linkerd-data-plane

× data plane proxies are ready
pod "auth-deploy-d697d9459-c66gg" status is Running

linkerd-viz-data-plane

‼ data plane proxy metrics are present in Prometheus
Data plane metrics not found for default/auth-deploy-d697d9459-qkbgg, default/auth-deploy-d697d9459-c66qg

[zhoudaxia233]

I've also tried to check the expiry of the certificates using kubectl get secret/linkerd-identity-issuer -n linkerd, after base64 decoding I got the following:

notBefore=Feb 13 21:52:39 2023 GMT
notAfter=Feb 13 21:52:59 2024 GMT

Apparently it's not expired.

[zhoudaxia233]

I've checked the local time on my machine, it is also correct.