:sparkles: Multiple FailoverIPs

[Mattes83]

Describe the solution you'd like:
We are using Metallb (https://metallb.universe.tf/) to announce public IPv4 addresses for our services of type LoadBalancer. Therefore we need the possibility to assign multiple FailoverIPs to an IonosCloudMachine.

As there is already the possibility to assign a single FailoverIP on an IonosCloudMachine I'd propose to make the field a list as

// If the machine is a control plane machine, this field will not be taken into account.
//+kubebuilder:validation:XValidation:rule="self == oldSelf",message="failoverIPs is immutable"
//+kubebuilder:validation:XValidation:rule=`self.all(ip, ip == "AUTO" || ip.matches("((25[0-5]|(2[0-4]|1\\d|[1-9]|)\\d)\\.?\\b){4}$"))`,message="failoverIPs entries must be either 'AUTO' or a valid IPv4 address"
//+optional
FailoverIPs []string `json:"failoverIPs,omitempty"

Please let me know if you are okay with this api change. I'd then start working on this.

Anything else you would like to add:
Will we need a new apiversion for this change?

[piepmatz]

Hi @Mattes83, as this would be a breaking change and we'd rather like to avoid that as the 1st thing after OpenSourcing in the beginning of June, let's clarify whether we can incorporate at least the API change quickly.

I'm not familiar with MetalLB. Can you please elaborate on why using it would result in requiring multiple failover IPs on a single machine? Is there some prior work in other infrastructure providers doing somthing similar?

[Mattes83]

Hi, thanks for your reply.
For the discussion we can replace MetalLB with kube-vip, which will be more familiar as you use it for the api endpoint of the control planes. Both kube-vip and MetalLB use ARP to announce their addresses on layer 2. If we have multiple services of Type LoadBalancer with the same port we will need multiple IPs. It might also be usefull to have multiple/distinct ips for eg firewall rules.

I am not sure how other providers are handling this.

[piepmatz]

I'm afraid I don't understand your use case yet. We'll have few more rounds of Q&A I guess. ;)

In the current FailoverIP singular form, the field expresses that a single IP will be shared among all machines of the same MachineDeployment. A controller then uses it to configure a failover group via Cloud API.

How do the semantics change with the proposed FailoverIPs plural field?
What will the controller use it for?

The MetalLB usage I've seen so far does not require setting up a failover group via Cloud API, it's rather an in-cluster-only solution to support services of type LoadBalancer. So I also don't understand yet why we need a CRD change to support MetalLB, shouldn't it work just fine without that? Also for my understanding: Did you have services in the management cluster or in a workload cluster in mind?

[Mattes83]

How do the semantics change with the proposed FailoverIPs plural field? What will the controller use it for?

The controller would create multiple failover groups, one for each FailoverIP.

The MetalLB usage I've seen so far does not require setting up a failover group via Cloud API, it's rather an in-cluster-only solution to support services of type LoadBalancer.

MetalLB should work out-of-the-box for private IPs. When we use public IPs we need to make sure the traffic reaches the right lan. In our homebrew capic solution we solved this by creating failovergroups to somehow register the public IPs with the right lan.

Also for my understanding: Did you have services in the management cluster or in a workload cluster in mind?

Its about services in the workload cluster.

Is there already another working LoadBalancer solution for ionos where we can assign static IPs? We do not necessarily need to use MetalLB.

It might also be a solution to deploy a custom operator in the workload cluster to manage the failover groups if you do not want to have it integrated into capic.

[piepmatz]

Thanks for the reply, the picture is getting clearer. ;)

Is there already another working LoadBalancer solution for ionos where we can assign static IPs? We do not necessarily need to use MetalLB.

Let's follow that approach for now. Yes, there is a CCM used in our Managed K8s offering. It supports services of type LoadBalancer. The CCM itself is not yet open source, but we plan to have a Helm Chart and some docs on using it available open source by end of next week. Would that be a viable solution for you?

[Mattes83]

I guess I found the documentation in #113.
This should be sufficient provided we can somehow control which ips get assigned to which service.

Background: We have imported our own IP range into a datacenter where we want to create our workload cluster.
We need some LoadBalancer Services to have an IP from this pool and not some random public ionos IPs.

[piepmatz]

Right, #113 will contain some docs, but is still WIP.

Our CCM supports assigning an IP explicitly to a service.

We have imported our own IP range into a datacenter

That's interesting. I've heard of the possibility to "bring your own IP range", but didn't have any points of contact with it yet. Do you know whether those IPs will be used when you reserve an IP block using https://api.ionos.com/docs/cloud/v6/#tag/IP-blocks/operation/ipblocksPost?

[farodin91]

I have multiple questions to the CCM.

Is LoadBalancerClass supported? https://kubernetes.io/docs/concepts/services-networking/service/#load-balancer-class
Could you define a ip block by annotations?

[piepmatz]

Is LoadBalancerClass supported?

Not sure what you mean. Our CCM acts as the cloud provider's default implementation, so it only considers services that don't set such a class. Or in other words: If you install an alternative load balancer implementation for a certain class, our CCM will ignore those services and won't interfere.

Could you define a ip block by annotations?

No, we continue to use spec.loadBalancerIP as long as that field is only deprecated. Once it gets removed, we'll switch to annotations. Here's an example: https://docs.ionos.com/cloud/containers/managed-kubernetes/use-cases/create-ip-address#example

[Mattes83]

Do you know whether those IPs will be used when you reserve an IP block using https://api.ionos.com/docs/cloud/v6/#tag/IP-blocks/operation/ipblocksPost?

We did not use this endpoint yet. The IPs from our IP range were moved from IONOS support and are avaiable as an IP block ready to use.

No, we continue to use spec.loadBalancerIP as long as that field is only deprecated. Once it gets removed, we'll switch to annotations. Here's an example: https://docs.ionos.com/cloud/containers/managed-kubernetes/use-cases/create-ip-address#example

It would be nice to be able to just say "take an ip from this block" instead of assigning single ips, but I guess we can work with the current solution.

[piepmatz]

Do you know whether those IPs will be used when you reserve an IP block using https://api.ionos.com/docs/cloud/v6/#tag/IP-blocks/operation/ipblocksPost?

We did not use this endpoint yet. The IPs from our IP range were moved from IONOS support and are avaiable as an IP block ready to use.

I talked to the support team and learnt that you don't need to use that API endpoint, because it will give you a new IP block from the range offered by IONOS Cloud. Instead, as the result of bringing your own range an IP block exists already. At least to me that's new. :)

No, we continue to use spec.loadBalancerIP as long as that field is only deprecated. Once it gets removed, we'll switch to annotations. Here's an example: https://docs.ionos.com/cloud/containers/managed-kubernetes/use-cases/create-ip-address#example

It would be nice to be able to just say "take an ip from this block" instead of assigning single ips, but I guess we can work with the current solution.

Our CCM support using the IP set in the service's spec.loadBalancerIP field as well as reserving an IP block automatically when the field is unset. So in your case you would indeed need to specify a concrete IP if it should be taken from your IP block. Our CCM does not support something like "take an IP automatically, but only from the following block". That would be a new feature, but I don't see much priority here, as it wasn't requested before.