Vault Agent Injector and Linkerd compatibility issue

[ihordyrman]

Hello everyone,

I'm currently facing an issue with my Kubernetes environment where I'm using HashiCorp's Vault Agent Injector to inject secrets into my applications, and I'm trying to add Linkerd to enable mTLS for all my apps.

After injecting Linkerd into my applications, some pods are failing to start. The logs indicate that the issue is related to the vault-agent-init container:

2023-04-19T16:31:44.311Z [ERROR] agent.auth.handler: error authenticating: error="Put \"http://vault.vault.svc:8200/v1/auth/kubernetes/login\": dial tcp 10.0.35.60:8200: connect: connection refused" backoff=3m59.72s
2023-04-19T16:35:44.033Z [INFO]  agent.auth.handler: authenticating
2023-04-19T16:35:44.035Z [ERROR] agent.auth.handler: error authenticating: error="Put \"http://vault.vault.svc:8200/v1/auth/kubernetes/login\": dial tcp 10.0.35.60:8200: connect: connection refused" backoff=3m52.85s
2023-04-19T16:39:36.894Z [INFO]  agent.auth.handler: authenticating
2023-04-19T16:39:36.896Z [ERROR] agent.auth.handler: error authenticating: error="Put \"http://vault.vault.svc:8200/v1/auth/kubernetes/login\": dial tcp 10.0.35.60:8200: connect: connection refused" backoff=4m59.52s

I suspect that the Linkerd proxy is interfering with the communication between the Vault Agent and the Vault server. I tried using the config.linkerd.io/skip-outbound-ports annotation to bypass the Linkerd proxy for the Vault server's communication, and it does work. However, I would like to know if there's a way to configure the Vault Agent Injector and my applications to work with Linkerd's mTLS without skipping the outbound ports.

Any guidance or suggestions on how to properly configure this setup would be greatly appreciated.

Thank you in advance!

[mikebell90]

This is a generic issue for linkerd and all meshes. If the initContainer doing ipTables rerouting runs before another initContainer that makes http calls, the second will fail because the proxy isn't up (and in a chicken egg fashion CANNOT be up until all initContainers succeed)

In our environment we use helm charts for all apps and directly inject the vault initContainer there. That guarantees it runs first.

If you are using mutating webhooks only though, this may be difficult to achieve, though maybe linkerd or vault has a "prepend" or "append" choice. Otherwise as I recall mutating webhooks run serially and alphabetically, which is not terribly reliable, but is another option

[ihordyrman]

The fix was easier than I thought.
I just had to use vault.hashicorp.com/agent-init-first: "true" annotation to make my vault injector run before linkerd.

[mikebell90]

That will work until some mutator after that changes the order