Broken Linkerd: proxy-injector webhook has valid cert: cert is not issued by the trust anchor: x509: certificate signed by unknown authority

[jseiser]

1. We have had 2 clusters break, with the same error, 5 days apart.

Linkerd Check Passes

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
‼ trust anchors are valid for at least 60 days
    Anchors expiring soon:
        * 300668091862607334720282139186536717139 root.linkerd.cluster.local will expire on 2025-07-21T13:57:35Z
    see https://linkerd.io/2/checks/#l5d-identity-trustAnchors-not-expiring-soon for hints
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2025-06-28T03:27:07Z
    see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

All pods are unable to restart, because they fail with certificate related errors.

{"timestamp":"2025-06-22T21:34:17.333260Z","level":"WARN","fields":{"message":"Failed to connect","error":"endpoint 10.11.38.159:8086: invalid peer certificate: BadSignature"},"target":"linkerd_reconnect","spans"
:[{"name":"dst"},{"addr":"linkerd-dst-headless.linkerd.svc.cluster.local:8086","name":"controller"},{"addr":"10.11.38.159:8086","name":"endpoint"}],"threadId":"ThreadId(1)"}
{"timestamp":"2025-06-22T21:34:17.333800Z","level":"WARN","fields":{"message":"Failed to connect","error":"endpoint 10.11.38.159:8090: invalid peer certificate: BadSignature"},"target":"linkerd_reconnect","spans"
:[{"name":"policy"},{"addr":"linkerd-policy.linkerd.svc.cluster.local:8090","name":"controller"},{"addr":"10.11.38.159:8090","name":"endpoint"}],"threadId":"ThreadId(1)"}
{"timestamp":"2025-06-22T21:34:17.337858Z","level":"INFO","fields":{"message":"Unavailable; entering failfast","timeout":30.0},"target":"linkerd_proxy_balance_queue::worker","spans":[{"name":"identity"},{"server.
addr":"linkerd-identity-headless.linkerd.svc.cluster.local:8080","name":"identity"},{"addr":"linkerd-identity-headless.linkerd.svc.cluster.local:8080","name":"controller"}],"threadId":"ThreadId(2)"}
{"timestamp":"2025-06-22T21:34:17.337906Z","level":"ERROR","fields":{"message":"Failed to obtain identity","error":"status: Unknown, message: \"controller linkerd-identity-headless.linkerd.svc.cluster.local:8080:
 service in fail-fast\", details: [], metadata: MetadataMap { headers: {} }"},"target":"linkerd_proxy_identity_client::certify","spans":[{"name":"identity"},{"server.addr":"linkerd-identity-headless.linkerd.svc.c
luster.local:8080","name":"identity"}],"threadId":"ThreadId(2)"}
{"timestamp":"2025-06-22T21:34:17.339895Z","level":"WARN","fields":{"message":"Waiting for identity to be initialized..."},"target":"linkerd_app","threadId":"ThreadId(1)"}

both clusters broke, within a week of upgrading to "2025.6.1". No amount of restarting etc would fix anything.

To fix, we had to remove linkerd, unmesh everything, re-create all certificates and re-deploy.

[kflynn]

Can you do this, just as rank paranoia?

inspect_cert () {
  sub_selector='\(.extensions.subject_key_id | .[0:16])... \(.subject_dn)'
  iss_selector='\(.extensions.authority_key_id // "................" | .[0:16])... \(.issuer_dn)'

  step certificate inspect --format json \
    | jq -r "\"Issuer:  $iss_selector\",\"Subject: $sub_selector\""
}

echo "Trust anchor:"
kubectl get configmap -n cert-manager linkerd-identity-trust-roots \
        -o jsonpath='{ .data.ca-bundle\.crt }' \
    | inspect_cert

echo ""
echo "Identity issuer:"
kubectl get secret -n linkerd linkerd-identity-issuer \
        -o jsonpath='{ .data.ca\.crt }' \
    | base64 -d | inspect_cert

That should output something like

Trust anchor:
Issuer:  ................... CN=root.linkerd.cluster.local
Subject: 5c455d3e9bd77e91... CN=root.linkerd.cluster.local

Identity issuer:
Issuer:  5c455d3e9bd77e91... CN=root.linkerd.cluster.local
Subject: 56bfe071553c16ad... CN=identity.linkerd.cluster.local

depending on how exactly you've set up the trust anchor in cert-manager, but the critical bit is that it's looking at the things that Linkerd should be looking at, and the Issuer of the identity issuer must exactly match the Subject of the trust anchor.

Based on what you're saying about your cert-manager setup, this should already be the case, but it's the first thing I'd want to check.