Ingress webhook to workaround the CCM issue

[makhov]

Create mutating webhook to add KUBERNETES_SERVICE_HOST env var to the pods with given label.

This breaks the CCM chicken-and-egg: CCM pods can reach the API server whithout local HAProxy, report node addresses, so kubernetes default service can be configured

[apedriza]

Nothing new to add from my side, but here are my two cents; IMO, overriding CCM env vars under the hood is not fully transparent to users. In practice, users still need to configure CCM for this scenario anyway, either by adding labels so pods are selected by the mutating webhook, or by setting the env vars if the webhook is not running.

I think it is fair to give users full responsibility for configuring those CCM env vars. That way ensures user knows what's going on in the CCM envs and there are no lines to maintain in k0smotron side. This is one of the benefits of being able to configure an app running in the cluster, so automation might not be needed for this. Documentation of this limitation and the required CCM configuration may be enough..?

[makhov]

I agree with all concerns. The webhook in question is optional and should be explicitly enabled and user needs to specify the label. All CCMs comes with some labels which can be used here, so no need to change it.
But I'm still on the fence about this change, not sure if it worth adding to the k0smotron code.