failed calling webhook "linkerd-proxy-injector.linkerd.io"

[marcoEgger]

Hello,

we got a problem in out kubernetes cluster (GKE) with all deployments when linkerd is installed. When updating/scaling/restarting deployments the underlying pods are not updated/scaled/restarted and the existing ones keep running. GKE displays this for all deployments Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": failed to call webhook: Post "https://linkerd-proxy-injector.linkerd.svc:443/?timeout=10s": context deadline exceeded

As soon as I uninstalled (with force, because when I uninject deployments the pods don't update) linkerd everything comes back to normal. I installed linkerd via CLI tool (with --ha flag), linkerd check results are all green. kube-system namespace has label config.linkerd.io/admission-webhooks: disabled

What can cause this and how can I resolve that?

Thanks in advance

[adleong]

Hi @marcoEgger! It sounds like after uninstalling and reinstalling, things are working now, which makes it hard to diagnose what the problem was. If you see this problem again, I would recommend taking a look at the logs of the proxy-injector container to see if there are any clues in there about why it was failing to response to webhook requests.

[marcoEgger]

Hey @adleong no the reinstallation did not fix it. The logs didn't state anything. Restating the kube-dns pods however solved the problem. I don't know what was causing it to fail.

[kleimkuhler]

Is it possible there is a firewall interfering with linkerd-proxy-injector? If so, as mentioned in #8707 there is a fix for adding the ports to the firewall rule.