Replacing expired certificates

[lucazz]

Heya everyone,

We're rotating the CA TLS certs in a Linkerd2 install, following the steps described in here: https://linkerd.io/2.10/tasks/replacing_expired_certificates/

But these pods here still keep getting the old certs being fed to them

data plane proxies certificate match CA
    Some pods do not have the current trust bundle and must be restarted:
        * linkerd/linkerd-grafana-55996db648-wrxxk
        * linkerd/linkerd-grafana-58fb86b575-m5jw6
        * linkerd/linkerd-prometheus-645b6f9d87-tljcv
        * linkerd/linkerd-prometheus-6f77d45bb4-x6pfv
        * linkerd/linkerd-tap-5975bf497f-nqwjw
        * linkerd/linkerd-tap-5975bf497f-v79d5
        * linkerd/linkerd-tap-6467884566-nhjz5
        * linkerd/linkerd-tap-6467884566-qvn8k
        * linkerd/linkerd-web-67696f9c5b-ttjkv
        * linkerd/linkerd-web-6d8d7c4bd9-7qcp7
    see https://linkerd.io/checks/#l5d-identity-data-plane-proxies-certs-match-ca for hints

They keep logging messages as such to stderr/stdout:

linkerd-prometheus-645b6f9d87-tljcv linkerd-proxy [  3633.742991207s]  WARN dst: rustls::session: Sending fatal alert BadCertificate
linkerd-prometheus-6f77d45bb4-x6pfv linkerd-proxy [  2723.807109948s]  WARN dst: rustls::session: Sending fatal alert BadCertificate

The upgrade process was pretty uneventful but still, we're stuck in this weird phase.
Any ideas on what we're missing here? :thinking_face:

[lucazz]

Manually updating TLS the deployment manifests for: linkerd-grafana, linkerd-prometheus and linkerd-web fixes these now.

[lucazz]

The only pods failing now are the linkerd-tap ones, and they fail with the following message:

time="2021-05-13T19:53:08Z" level=info msg="running version stable-2.8.1"
time="2021-05-13T19:53:08Z" level=fatal msg="failed to read config file: open /var/run/linkerd/config/global: no such file or directory"
~ ❯❯❯

[olix0r]

It sounds like the pod isn't mounting a required configmap? I believe this was the linkerd-config configmap (this has changed since stable-2.8)?