Linkerd viz extension doesn't work after few hours inactivity

[zhoudaxia233]

I've already asked the same question here a few days ago but I haven't got an answer. This error has been haunting me for WEEKs so I would like to ask again with more detail in hopes of getting a helpful response.

Upon running the linkerd check command, I received an error message regarding the Viz extension self-check (× viz extension self-check). Upon further investigation, I found relevant logs in the broken metrics-api pod (with status CrashLoopBackOff):

1. in metrics-api container:

time="2023-02-17T21:26:53Z" level=info msg="running version dev-undefined"
time="2023-02-17T21:26:53Z" level=info msg="starting admin server on :9995"
time="2023-02-17T21:26:56Z" level=fatal msg="Failed to initialize K8s API: Post \"https://10.96.0.1:443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\": read tcp 10.244.0.39:34860->10.96.0.1:443: read: connection reset by peer"

2. in linkerd-proxy container:

[     0.013709s]  INFO ThreadId(01) linkerd2_proxy: Local identity is metrics-api.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
[     0.013711s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.013713s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.016622s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: rustls::conn: Sending fatal alert BadCertificate    
[     0.016715s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     0.042606s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=metrics-api.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
[     0.121588s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: rustls::conn: Sending fatal alert BadCertificate    
[     0.121903s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     0.330366s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: rustls::conn: Sending fatal alert BadCertificate    
[     0.330674s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     0.741323s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: rustls::conn: Sending fatal alert BadCertificate    
[     0.741621s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     1.244060s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: rustls::conn: Sending fatal alert BadCertificate    
[     1.244629s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.244.0.18:8090}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     1.517283s]  WARN ThreadId(01) outbound:proxy{addr=10.96.0.1:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.244.0.18:8086}: rustls::conn: Sending fatal alert BadCertificate    
[     1.517357s]  WARN ThreadId(01) outbound:proxy{addr=10.96.0.1:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.244.0.18:8086}: linkerd_reconnect: Failed to connect error=invalid peer certificate contents: invalid peer certificate: CertExpired
[     1.629176s]  WARN ThreadId(01) outbound:proxy{addr=10.96.0.1:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.244.0.18:8086}: rustls::conn: Sending fatal alert BadCertificate

I have two metrics-api pods running in namespace linkerd-viz, however, one of them is broken(as above), another works fine.

NAME                            READY   STATUS             RESTARTS        AGE
metrics-api-58878cc567-9hbvh    2/2     Running            2 (32h ago)     46h
metrics-api-7c9469d5db-mbnqp    1/2     CrashLoopBackOff   8 (4m50s ago)   21m

What's weird is, even in the normal metrics-api pod (which is metrics-api-58878cc567-9hbvh), the logs seem abnormal:

1. in metrics-api container:

time="2023-02-17T14:55:49Z" level=error msg="queryProm failed with: Query failed: \"histogram_quantile(0.95, sum(irate(response_latency_ms_bucket{direction=\\\"outbound\\\", dst_deployment=\\\"auth-deploy\\\", dst_namespace=\\\"cert-test\\\"}[1m])) by (le, namespace, statefulset))\": context canceled"
time="2023-02-17T14:55:49Z" level=error msg="queryProm failed with: Query failed: \"sum(increase(response_total{direction=\\\"outbound\\\", dst_deployment=\\\"auth-deploy\\\", dst_namespace=\\\"cert-test\\\"}[1m])) by (namespace, statefulset, classification, tls)\": Post \"http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query\": context canceled"
time="2023-02-17T14:55:49Z" level=error msg="queryProm failed with: Query failed: \"sum(increase(response_total{direction=\\\"outbound\\\", dst_deployment=\\\"auth-deploy\\\", dst_namespace=\\\"cert-test\\\"}[1m])) by (namespace, replicationcontroller, classification, tls)\": Post \"http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query\": context canceled"
time="2023-02-17T14:55:49Z" level=error msg="queryProm failed with: Query failed: \"histogram_quantile(0.99, sum(irate(response_latency_ms_bucket{deployment=\\\"auth-deploy\\\", direction=\\\"inbound\\\", namespace=\\\"cert-test\\\"}[1m])) by (le, namespace, deployment))\": server_error: server error: 504"
time="2023-02-17T14:55:49Z" level=error msg="queryProm failed with: Query failed: \"sum(tcp_open_connections{deployment=\\\"auth-deploy\\\", direction=\\\"inbound\\\", namespace=\\\"cert-test\\\"}) by (namespace, deployment)\": server_error: server error: 504"
time="2023-02-17T14:55:49Z" level=error msg="queryProm failed with: Query failed: \"sum(increase(response_total{deployment=\\\"auth-deploy\\\", direction=\\\"inbound\\\", namespace=\\\"cert-test\\\"}[1m])) by (namespace, deployment, classification, tls)\": server_error: server error: 504"

2. in linkerd-proxy container

[ 81321.055332s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "connection error: broken pipe", details: [], metadata: MetadataMap { headers: {} } error.sources=[connection error: broken pipe, broken pipe]
[ 81331.061288s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "connection error: broken pipe", details: [], metadata: MetadataMap { headers: {} } error.sources=[connection error: broken pipe, broken pipe]
[ 81341.067220s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "connection error: broken pipe", details: [], metadata: MetadataMap { headers: {} } error.sources=[connection error: broken pipe, broken pipe]
[ 81351.073234s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "connection error: broken pipe", details: [], metadata: MetadataMap { headers: {} } error.sources=[connection error: broken pipe, broken pipe]
[ 81361.080106s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "connection error: broken pipe", details: [], metadata: MetadataMap { headers: {} } error.sources=[connection error: broken pipe, broken pipe]

I've also checked the identity certificate of the broken pod metrics-api-7c9469d5db-mbnqp (with status CrashLoopBackOff) to make sure it's still valid.

POD metrics-api-7c9469d5db-mbnqp (1 of 1)

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 17 (0x11)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=identity.linkerd.
        Validity
            Not Before: Feb 17 21:20:25 2023 UTC
            Not After : Feb 18 21:21:05 2023 UTC
        Subject: CN=metrics-api.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
        . . . . . .

And it is valid! (NOW is Feb 17 22:00:00 2023 UTC)

I've tried to restart the deployment again, tried to destroy the cluster and rebuilt everything from scratch, tried Minikube, tried Kind, just to name a few.

(Not sure if this information helps, but I use Ubuntu 18.04.6 LTS)
My Linkerd version:
Client version: stable-2.12.4
Server version: stable-2.12.4