Issue with certificate in production environment

[deggja]

Hello guys!

I have a problem and I'm quite new to linkerd so I'm struggling a bit unfortunately. Here goes.

I've been running Linkerd for exactly one year and I've stumbled onto some issues with my certificates that expired today.

I've attempted to recreate the anchor certificate and the intermediate certificate as documented in the linkerd docs here: https://linkerd.io/2.12/tasks/manually-rotating-control-plane-tls-credentials/#rotating-the-identity-issuer-certificate

I've then attempted to upgrade Linkerd with the following
command:

linkerd upgrade \
    --identity-issuer-certificate-file=./issuer-new.crt \
    --identity-issuer-key-file=./issuer-new.key \
    | kubectl apply -f -

Found here: https://linkerd.io/2.12/tasks/manually-rotating-control-plane-tls-credentials/#rotating-the-identity-issuer-certificate

After doing this, all my linkerd related pods have restarted, but I can see the following being reported in my linkerd-proxy containers (every second).

ERROR ThreadId(01) inbound:server{port=8080}: rustls::conn: TLS alert received: AlertMessagePayload {
    level: Fatal,
    description: BadCertificate,

So obviously I'm doing something wrong here. Any pointers in the right direction would be appreciated.

However, when running linkerd check - the only error it is reporting is the following:

× tap API server has valid cert
    certificate is not valid anymore

So I followed what I found of documentation on linkerd.io and tried looking for solutions online, with no luck so far. Hopefully I did something right here.. but I cant seem to find how to attack this particular issue (might be my bad googling skills). Any pointers in the right direction is greatly appreciated.

My app is currently dead 😞

BTW: Posted the same question in the Linkerd slack community: https://linkerd.slack.com/archives/C89RTCWJF/p1671033205770679

[wmorgan]

Sorry to hear that. Those instructions should work... does linkerd check --proxy show anything informative?

That said, if your app is currently non-functional anyways because of this, then (assuming you aren't doing anything related to multi-cluster) one failsafe option is to unmesh all your pods, remove Linkerd, and re-install from scratch. You could override the trust anchor and the identity certificates with your own long-lived versions (e.g. 30 years), or you can set up cert-manager to automatically rotate the issuer certs, depending on your security posture.

[deggja]

Sorry to hear that. Those instructions should work... does linkerd check --proxy show anything informative?

That said, if your app is currently non-functional anyways because of this, then (assuming you aren't doing anything related to multi-cluster) one failsafe option is to unmesh all your pods, remove Linkerd, and re-install from scratch. You could override the trust anchor and the identity certificates with your own long-lived versions (e.g. 30 years), or you can set up cert-manager to automatically rotate the issuer certs, depending on your security posture.

Hey @wmorgan, thanks for jumping in on this!

Running linkerd check passes on everything expect this extension check for linkerd-viz:

× tap API server has valid cert
    certificate is not valid anymore. Expired on 2022-12-14T14:40:36Z
    see https://linkerd.io/2.12/checks/#l5d-tap-cert-valid for hints

I've tried to solve this but I haven't gotten any closer as of yet. I just attempted the following:

1. Generate a new trust anchor (https://linkerd.io/2.12/tasks/manually-rotating-control-plane-tls-credentials/#generate-a-new-trust-anchor)

2. Bundle original trust anchor with the new one and then deploying it (https://linkerd.io/2.12/tasks/manually-rotating-control-plane-tls-credentials/#deploying-the-new-bundle-to-linkerd)

3. Generating new root and issuer certificates as described here: https://linkerd.io/2.12/tasks/generate-certificates/#generating-the-certificates-with-step

4. Lastly, attempting to upgrade Linkerd by running the following upgrade command (from here: https://linkerd.io/2.12/tasks/replacing_expired_certificates/)

linkerd upgrade \
    --identity-issuer-certificate-file=./issuer-new.crt \
    --identity-issuer-key-file=./issuer-new.key \
    --identity-trust-anchors-file=./ca-new.crt \
    --force \
    | kubectl apply -f -

But with no luck.

I have a feeling i'm missing something though and this is failing due to me being a total noob. I'm considering uninstalling and doing a fresh install but I would really prefer solving the issue.

Thank you again.

[deggja]

The issue was solved by uninstalling and doing a clean re-install of Linkerd using helm.