Linkerd viz extension doesn't work after few hours inactivity

[zhoudaxia233]

I freshly set up Linkerd and the Linkerd Viz extension on my local machine and everything was working fine when I checked the dashboard to monitor all traffic. However, after a few hours of inactivity, I noticed that the dashboard was no longer working. Upon running the linkerd check command, I received an error message regarding the Viz extension self-check (× viz extension self-check). Upon further investigation, I found relevant logs in the metrics-api pod of the Linkerd-proxy container:

..........................
[   141.591673s]  WARN ThreadId(01) policy:watch{port=8085}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.16:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[   141.593804s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.15:8080}: rustls::conn: Sending fatal alert BadCertificate    
[   141.593864s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.15:8080}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[   142.094593s]  WARN ThreadId(01) policy:watch{port=8085}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.16:8090}: rustls::conn: Sending fatal alert BadCertificate    
[   142.094816s]  WARN ThreadId(01) policy:watch{port=8085}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.16:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[   142.097242s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.15:8080}: rustls::conn: Sending fatal alert BadCertificate    
[   142.097416s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.15:8080}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[   142.598424s]  WARN ThreadId(01) policy:watch{port=8085}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.16:8090}: rustls::conn: Sending fatal alert BadCertificate    
[   142.598698s]  WARN ThreadId(01) policy:watch{port=8085}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.16:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[   142.599952s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.15:8080}: rustls::conn: Sending fatal alert BadCertificate    
[   142.600193s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.15:8080}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired

I cannot fixed this issue, so I just destroyed the cluster on my laptop and rebuild everything. However, after a few hours inactivity the same issue appears. I've tried to restart the deployment, tried to delete the pod, none of them works.

I tried this on both minikube and kind.

Please help me, thank you!

[kleimkuhler]

What is the output of linkerd check when these errors start occurring?

[zhoudaxia233]

All checks are passed, except for:
× viz extension self-check

[kleimkuhler]

[   142.097416s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.15:8080}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[   142.598424s]  WARN ThreadId(01) policy:watch{port=8085}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.16:8090}: rustls::conn: Sending fatal alert BadCertificate    

These log lines point to something being incorrect about identity on the cluster. Are you configuring anything about identity during installation? Have you verified that the root/issuer certs are correct? Do you see the same issues when installing with the CLI which can generate it's own certs?

[zhoudaxia233]

1. No, I didn't do any customized configuration.
2. Yes, if you are referring to the secret linkerd-identity-issuer in linkerd namespace. It is valid. Also the root cert kube-root-ca.crtof k8s is valid.
3. I'm not sure if I understand you correctly. I always use the linkerd CLI to install linkerd into my k8s cluster (via linkerd inject). During installing, there's no error. So should be a No to your 3rd question.

[zhoudaxia233]

And it would be so kind if you could take a look at this discussion. Basically the same issue, but I provide much more details there.
Thank you for your help!