Skip to content

Commit 6aa7494

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #973 from stackhpc/disable-api-server-fip
✨ Allow clusters without a floating IP for the API server
2 parents 7a9b9d1 + f3432b3 commit 6aa7494

File tree

8 files changed

+217
-55
lines changed

8 files changed

+217
-55
lines changed

‎api/v1alpha4/openstackcluster_types.go‎

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -58,16 +58,41 @@ type OpenStackClusterSpec struct {
5858
ExternalNetworkID string `json:"externalNetworkId,omitempty"`
5959

6060
// ManagedAPIServerLoadBalancer defines whether a LoadBalancer for the
61-
// APIServer should be created. If set to true the following properties are
62-
// mandatory: APIServerFloatingIP, APIServerPort
61+
// APIServer should be created.
6362
// +optional
6463
ManagedAPIServerLoadBalancer bool `json:"managedAPIServerLoadBalancer"`
6564

66-
// APIServerFloatingIP is the floatingIP which will be associated
67-
// to the APIServer. The floatingIP will be created if it not
68-
// already exists.
65+
// DisableAPIServerFloatingIP determines whether or not to attempt to attach a floating
66+
// IP to the API server. This allows for the creation of clusters when attaching a floating
67+
// IP to the API server (and hence, in many cases, exposing the API server to the internet)
68+
// is not possible or desirable, e.g. if using a shared VLAN for communication between
69+
// management and workload clusters or when the management cluster is inside the
70+
// project network.
71+
// This option requires that the API server use a VIP on the cluster network so that the
72+
// underlying machines can change without changing ControlPlaneEndpoint.Host.
73+
// When using a managed load balancer, this VIP will be managed automatically.
74+
// If not using a managed load balancer, cluster configuration will fail without additional
75+
// configuration to manage the VIP on the control plane machines, which falls outside of
76+
// the scope of this controller.
77+
// +optional
78+
DisableAPIServerFloatingIP bool `json:"disableAPIServerFloatingIP"`
79+
80+
// APIServerFloatingIP is the floatingIP which will be associated with the API server.
81+
// The floatingIP will be created if it does not already exist.
82+
// If not specified, a new floatingIP is allocated.
83+
// This field is not used if DisableAPIServerFloatingIP is set to true.
6984
APIServerFloatingIP string `json:"apiServerFloatingIP,omitempty"`
7085

86+
// APIServerFixedIP is the fixed IP which will be associated with the API server.
87+
// In the case where the API server has a floating IP but not a managed load balancer,
88+
// this field is not used.
89+
// If a managed load balancer is used and this field is not specified, a fixed IP will
90+
// be dynamically allocated for the load balancer.
91+
// If a managed load balancer is not used AND the API server floating IP is disabled,
92+
// this field MUST be specified and should correspond to a pre-allocated port that
93+
// holds the fixed IP to be used as a VIP.
94+
APIServerFixedIP string `json:"apiServerFixedIP,omitempty"`
95+
7196
// APIServerPort is the port on which the listener on the APIServer
7297
// will be created
7398
APIServerPort int `json:"apiServerPort,omitempty"`

‎config/crd/bases/infrastructure.cluster.x-k8s.io_openstackclusters.yaml‎

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1078,10 +1078,22 @@ spec:
10781078
groups are configured so that all ingress and egress between cluster
10791079
nodes is permitted, allowing CNIs other than Calico to be used.
10801080
type: boolean
1081+
apiServerFixedIP:
1082+
description: APIServerFixedIP is the fixed IP which will be associated
1083+
with the API server. In the case where the API server has a floating
1084+
IP but not a managed load balancer, this field is not used. If a
1085+
managed load balancer is used and this field is not specified, a
1086+
fixed IP will be dynamically allocated for the load balancer. If
1087+
a managed load balancer is not used AND the API server floating
1088+
IP is disabled, this field MUST be specified and should correspond
1089+
to a pre-allocated port that holds the fixed IP to be used as a
1090+
VIP.
1091+
type: string
10811092
apiServerFloatingIP:
10821093
description: APIServerFloatingIP is the floatingIP which will be associated
1083-
to the APIServer. The floatingIP will be created if it not already
1084-
exists.
1094+
with the API server. The floatingIP will be created if it does not
1095+
already exist. If not specified, a new floatingIP is allocated.
1096+
This field is not used if DisableAPIServerFloatingIP is set to true.
10851097
type: string
10861098
apiServerLoadBalancerAdditionalPorts:
10871099
description: APIServerLoadBalancerAdditionalPorts adds additional
@@ -1451,6 +1463,21 @@ spec:
14511463
- host
14521464
- port
14531465
type: object
1466+
disableAPIServerFloatingIP:
1467+
description: DisableAPIServerFloatingIP determines whether or not
1468+
to attempt to attach a floating IP to the API server. This allows
1469+
for the creation of clusters when attaching a floating IP to the
1470+
API server (and hence, in many cases, exposing the API server to
1471+
the internet) is not possible or desirable, e.g. if using a shared
1472+
VLAN for communication between management and workload clusters
1473+
or when the management cluster is inside the project network. This
1474+
option requires that the API server use a VIP on the cluster network
1475+
so that the underlying machines can change without changing ControlPlaneEndpoint.Host.
1476+
When using a managed load balancer, this VIP will be managed automatically.
1477+
If not using a managed load balancer, cluster configuration will
1478+
fail without additional configuration to manage the VIP on the control
1479+
plane machines, which falls outside of the scope of this controller.
1480+
type: boolean
14541481
disablePortSecurity:
14551482
description: DisablePortSecurity disables the port security of the
14561483
network created for the Kubernetes cluster, which also disables
@@ -1554,9 +1581,8 @@ spec:
15541581
- name
15551582
type: object
15561583
managedAPIServerLoadBalancer:
1557-
description: 'ManagedAPIServerLoadBalancer defines whether a LoadBalancer
1558-
for the APIServer should be created. If set to true the following
1559-
properties are mandatory: APIServerFloatingIP, APIServerPort'
1584+
description: ManagedAPIServerLoadBalancer defines whether a LoadBalancer
1585+
for the APIServer should be created.
15601586
type: boolean
15611587
managedSecurityGroups:
15621588
description: ManagedSecurityGroups determines whether OpenStack security

‎config/crd/bases/infrastructure.cluster.x-k8s.io_openstackclustertemplates.yaml‎

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -57,10 +57,24 @@ spec:
5757
and egress between cluster nodes is permitted, allowing
5858
CNIs other than Calico to be used.
5959
type: boolean
60+
apiServerFixedIP:
61+
description: APIServerFixedIP is the fixed IP which will be
62+
associated with the API server. In the case where the API
63+
server has a floating IP but not a managed load balancer,
64+
this field is not used. If a managed load balancer is used
65+
and this field is not specified, a fixed IP will be dynamically
66+
allocated for the load balancer. If a managed load balancer
67+
is not used AND the API server floating IP is disabled,
68+
this field MUST be specified and should correspond to a
69+
pre-allocated port that holds the fixed IP to be used as
70+
a VIP.
71+
type: string
6072
apiServerFloatingIP:
6173
description: APIServerFloatingIP is the floatingIP which will
62-
be associated to the APIServer. The floatingIP will be created
63-
if it not already exists.
74+
be associated with the API server. The floatingIP will be
75+
created if it does not already exist. If not specified,
76+
a new floatingIP is allocated. This field is not used if
77+
DisableAPIServerFloatingIP is set to true.
6478
type: string
6579
apiServerLoadBalancerAdditionalPorts:
6680
description: APIServerLoadBalancerAdditionalPorts adds additional
@@ -438,6 +452,23 @@ spec:
438452
- host
439453
- port
440454
type: object
455+
disableAPIServerFloatingIP:
456+
description: DisableAPIServerFloatingIP determines whether
457+
or not to attempt to attach a floating IP to the API server.
458+
This allows for the creation of clusters when attaching
459+
a floating IP to the API server (and hence, in many cases,
460+
exposing the API server to the internet) is not possible
461+
or desirable, e.g. if using a shared VLAN for communication
462+
between management and workload clusters or when the management
463+
cluster is inside the project network. This option requires
464+
that the API server use a VIP on the cluster network so
465+
that the underlying machines can change without changing
466+
ControlPlaneEndpoint.Host. When using a managed load balancer,
467+
this VIP will be managed automatically. If not using a managed
468+
load balancer, cluster configuration will fail without additional
469+
configuration to manage the VIP on the control plane machines,
470+
which falls outside of the scope of this controller.
471+
type: boolean
441472
disablePortSecurity:
442473
description: DisablePortSecurity disables the port security
443474
of the network created for the Kubernetes cluster, which
@@ -545,10 +576,8 @@ spec:
545576
- name
546577
type: object
547578
managedAPIServerLoadBalancer:
548-
description: 'ManagedAPIServerLoadBalancer defines whether
549-
a LoadBalancer for the APIServer should be created. If set
550-
to true the following properties are mandatory: APIServerFloatingIP,
551-
APIServerPort'
579+
description: ManagedAPIServerLoadBalancer defines whether
580+
a LoadBalancer for the APIServer should be created.
552581
type: boolean
553582
managedSecurityGroups:
554583
description: ManagedSecurityGroups determines whether OpenStack

‎controllers/openstackcluster_controller.go‎

Lines changed: 50 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -438,44 +438,75 @@ func reconcileNetworkComponents(log logr.Logger, osProviderClient *gophercloud.P
438438
return errors.Errorf("failed to reconcile router: %v", err)
439439
}
440440
}
441-
if !openStackCluster.Spec.ControlPlaneEndpoint.IsValid() {
442-
var port int32
443-
if openStackCluster.Spec.APIServerPort == 0 {
444-
port = 6443
445-
} else {
446-
port = int32(openStackCluster.Spec.APIServerPort)
447-
}
448-
fp, err := networkingService.GetOrCreateFloatingIP(openStackCluster, clusterName, openStackCluster.Spec.APIServerFloatingIP)
449-
if err != nil {
450-
handleUpdateOSCError(openStackCluster, errors.Errorf("Floating IP cannot be got or created: %v", err))
451-
return errors.Errorf("Floating IP cannot be got or created: %v", err)
452-
}
453-
// Set APIEndpoints so the Cluster API Cluster Controller can pull them
454-
openStackCluster.Spec.ControlPlaneEndpoint = clusterv1.APIEndpoint{
455-
Host: fp.FloatingIP,
456-
Port: port,
457-
}
458-
}
459441

460442
err = networkingService.ReconcileSecurityGroups(openStackCluster, clusterName)
461443
if err != nil {
462444
handleUpdateOSCError(openStackCluster, errors.Errorf("failed to reconcile security groups: %v", err))
463445
return errors.Errorf("failed to reconcile security groups: %v", err)
464446
}
465447

448+
// Calculate the port that we will use for the API server
449+
var apiServerPort int
450+
switch {
451+
case openStackCluster.Spec.ControlPlaneEndpoint.IsValid():
452+
apiServerPort = int(openStackCluster.Spec.ControlPlaneEndpoint.Port)
453+
case openStackCluster.Spec.APIServerPort != 0:
454+
apiServerPort = openStackCluster.Spec.APIServerPort
455+
default:
456+
apiServerPort = 6443
457+
}
458+
466459
if openStackCluster.Spec.ManagedAPIServerLoadBalancer {
467460
loadBalancerService, err := loadbalancer.NewService(osProviderClient, clientOpts, log)
468461
if err != nil {
469462
return err
470463
}
471464

472-
err = loadBalancerService.ReconcileLoadBalancer(openStackCluster, clusterName)
465+
err = loadBalancerService.ReconcileLoadBalancer(openStackCluster, clusterName, apiServerPort)
473466
if err != nil {
474467
handleUpdateOSCError(openStackCluster, errors.Errorf("failed to reconcile load balancer: %v", err))
475468
return errors.Errorf("failed to reconcile load balancer: %v", err)
476469
}
477470
}
478471

472+
if !openStackCluster.Spec.ControlPlaneEndpoint.IsValid() {
473+
var host string
474+
// If there is a load balancer use the floating IP for it if set, falling back to the internal IP
475+
switch {
476+
case openStackCluster.Spec.ManagedAPIServerLoadBalancer:
477+
if openStackCluster.Status.Network.APIServerLoadBalancer.IP != "" {
478+
host = openStackCluster.Status.Network.APIServerLoadBalancer.IP
479+
} else {
480+
host = openStackCluster.Status.Network.APIServerLoadBalancer.InternalIP
481+
}
482+
case !openStackCluster.Spec.DisableAPIServerFloatingIP:
483+
// If floating IPs are not disabled, get one to use as the VIP for the control plane
484+
fp, err := networkingService.GetOrCreateFloatingIP(openStackCluster, clusterName, openStackCluster.Spec.APIServerFloatingIP)
485+
if err != nil {
486+
handleUpdateOSCError(openStackCluster, errors.Errorf("Floating IP cannot be got or created: %v", err))
487+
return errors.Errorf("Floating IP cannot be got or created: %v", err)
488+
}
489+
host = fp.FloatingIP
490+
case openStackCluster.Spec.APIServerFixedIP != "":
491+
// If a fixed IP was specified, assume that the user is providing the extra configuration
492+
// to use that IP as the VIP for the API server, e.g. using keepalived or kube-vip
493+
host = openStackCluster.Spec.APIServerFixedIP
494+
default:
495+
// For now, we do not provide a managed VIP without either a load balancer or a floating IP
496+
// In the future, we could manage a VIP port on the cluster network and set allowedAddressPairs
497+
// accordingly when creating control plane machines
498+
// However this would require us to deploy software on the control plane hosts to manage the
499+
// VIP (e.g. keepalived/kube-vip)
500+
return errors.New("unable to determine VIP for API server")
501+
}
502+
503+
// Set APIEndpoints so the Cluster API Cluster Controller can pull them
504+
openStackCluster.Spec.ControlPlaneEndpoint = clusterv1.APIEndpoint{
505+
Host: host,
506+
Port: int32(apiServerPort),
507+
}
508+
}
509+
479510
return nil
480511
}
481512

‎controllers/openstackmachine_controller.go‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -360,8 +360,12 @@ func (r *OpenStackMachineReconciler) reconcileNormal(ctx context.Context, logger
360360
handleUpdateMachineError(logger, openStackMachine, errors.Errorf("LoadBalancerMember cannot be reconciled: %v", err))
361361
return ctrl.Result{}, nil
362362
}
363-
} else if util.IsControlPlaneMachine(machine) {
364-
fp, err := networkingService.GetOrCreateFloatingIP(openStackCluster, clusterName, openStackCluster.Spec.ControlPlaneEndpoint.Host)
363+
} else if util.IsControlPlaneMachine(machine) && !openStackCluster.Spec.DisableAPIServerFloatingIP {
364+
floatingIPAddress := openStackCluster.Spec.ControlPlaneEndpoint.Host
365+
if openStackCluster.Spec.APIServerFloatingIP != "" {
366+
floatingIPAddress = openStackCluster.Spec.APIServerFloatingIP
367+
}
368+
fp, err := networkingService.GetOrCreateFloatingIP(openStackCluster, clusterName, floatingIPAddress)
365369
if err != nil {
366370
handleUpdateMachineError(logger, openStackMachine, errors.Errorf("Floating IP cannot be got or created: %v", err))
367371
return ctrl.Result{}, nil

‎docs/book/src/clusteropenstack/configuration.md‎

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@
1414
- [Optional Configuration](#optional-configuration)
1515
- [Log level](#log-level)
1616
- [External network](#external-network)
17-
- [Floating IP](#floating-ip)
17+
- [API server floating IP](#api-server-floating-ip)
18+
- [Disabling the API server floating IP](#disabling-the-api-server-floating-ip)
1819
- [Network Filters](#network-filters)
1920
- [Multiple Networks](#multiple-networks)
2021
- [Subnet Filters](#subnet-filters)
@@ -134,9 +135,11 @@ openstack network list --external
134135

135136
Note: If your openstack cluster does not already have a public network, you should contact your cloud service provider. We will not review how to troubleshoot this here.
136137

137-
## Floating IP
138+
## API server floating IP
138139

139-
A floating IP is automatically created and associated with the load balancer or controller node, but you can specify the floating IP explicitly by `spec.apiServerFloatingIP` of `OpenStackCluster`.
140+
Unless explicitly disabled, a floating IP is automatically created and associated with the load balancer
141+
or controller node. If required, you can specify the floating IP explicitly by `spec.apiServerFloatingIP`
142+
of `OpenStackCluster`.
140143

141144
You have to be able to create a floating IP in your OpenStack in advance. You can create one using,
142145

@@ -146,6 +149,31 @@ openstack floating ip create <public network>
146149

147150
Note: Only user with admin role can create a floating IP with specific IP.
148151

152+
### Disabling the API server floating IP
153+
154+
It is possible to provision a cluster without a floating IP for the API server by setting
155+
`OpenStackCluster.spec.disableAPIServerFloatingIP: true` (the default is `false`). This will
156+
prevent a floating IP from being allocated.
157+
158+
> **WARNING**
159+
>
160+
> If the API server does not have a floating IP, workload clusters will only deploy successfully
161+
> when the management cluster and workload cluster control plane nodes are on the same network.
162+
> This can be a project-specific network, if the management cluster lives in the same project
163+
> as the workload cluster, or a network that is shared across multiple projects.
164+
>
165+
> In particular, this means that the cluster **cannot** use `OpenStackCluster.spec.nodeCidr`
166+
> to provision a new network for the cluster. Instead, use `OpenStackCluster.spec.network`
167+
> to explicitly specify the same network as the management cluster is on.
168+
169+
When the API server floating IP is disabled, it is **not possible** to provision a cluster
170+
without a load balancer without additional configuration (an advanced use-case that is not
171+
documented here). This is because the API server must still have a
172+
[virtual IP](https://en.wikipedia.org/wiki/Virtual_IP_address) that is not associated with
173+
a particular control plane node in order to allow the nodes to change underneath, e.g.
174+
during an upgrade. When the API server has a floating IP, this role is fulfilled by the
175+
floating IP even if there is no load balancer. When the API server does not have a floating
176+
IP, the load balancer virtual IP on the cluster network is used.
149177

150178
## Network Filters
151179

0 commit comments

Comments
 (0)