Allow externalTrafficPolicy configuration on Service

[Wouter0100]

We're using BGP to announce the LoadBalancer service created by Kamaji for a tenant control plane. As an improvement, we'd like to directly route traffic to nodes that're actually running the tenant control plane pods.

Specifically, we want to set the spec.externalTrafficPolicy to Local, which is currently Cluster (the default).

How would you like to have this implemented in the Kamaji TCP spec?

What we would be able to do is something like:

spec:
  controlPlane:
      service:
        spec:
            externalTrafficPolicy: Local
        serviceType: LoadBalancer

or, to have it the same as additionalMetadata:

spec:
  controlPlane:
      service:
        additionalSpec:
            externalTrafficPolicy: Local
        serviceType: LoadBalancer

[prometherion]

Definitely useful, although this looks more like a network configuration: we recently merged #620 and #611 in the .spec.networkProfile field, I think it would be better to handle this information there, as we're already doing with allowAddressAsExternalIP.

[Wouter0100]

Makes sense! In that case just a .spec.networkProfile.externalTrafficPolicy would suffice? 🚀

[prometherion]

Yes, I think this can be enough.

In the future we would need a refactoring of the specification since now it's a bit counter-intuitive, the plan is sticking to the current specification without introducing breaking changes.

[Wouter0100]

We have found other problems with this option, since Konnectivity fails to work with externalTrafficPolicy: Local in our specific setup. Konnectivity agent traffic is with Local always routed to a single Konnectivity server, causing the other apiserver to not have any konnectivity connections available.

[Marlinc]

For reference, Konnectivity agents expects to be load balanced across proxy servers as mentioned here kubernetes-sigs/apiserver-network-proxy#570 (comment)

[prometherion]

It means we would need to have a separate Service for Konnectivity, but this would increase the complexity of the Kubernetes objects 🤔