Nginx Ingress Controller without LoadBalancer

[UncleSamSwiss]

The readme states

• highly-available via kube-vip and dynamic peering (no load-balancer required)

I was able to reconfigure the Nginx Ingress Helm chart to deploy the controller service as a NodePort, but now I guess I'm missing a piece of configuration (probably adding something to kube-vip). Do you have any example how to expose a service to the public IP address, the same way it is done for port 6443.

I'm using your great module on Infomaniak for a private project and I can't afford paying an extra 13.- for the additional IP and the OpenStack load balancer.

[zifeo]

@UncleSamSwiss Thanks for the point. That sentence is meant to be read slightly differently: there is no need for an openstack load-balancer to have the control-pane (the servers in rke2) highly available. However due to the nature of the module with an isolated/private network, it is strongly recommended to have a load-balancer for any incoming network connection. You might be able to get around that by attaching a floating IP to a node and try this however you would loose the high-availability in case that node goes down.

[UncleSamSwiss]

Thanks for your answer. It helps a lot! I currently don't care about HA since I only have one node. If I ever need to scale, I also don't mind adding a real load balancer.

[UncleSamSwiss]

@zifeo I got it working but not without some "manual intervention" (i.e. not using Terraform). Reason for this being that I couldn't set the allowed_address_pairs for the agent VMs. Would you be interested in a PR adding the possibility to set additional allowed_address_pairs for nodes or is this requirement too specific for my use case?

[zifeo]

@UncleSamSwiss that does make sense per pool of agents, feel free to add that in a PR 👍

[UncleSamSwiss]

By the way, I'm wondering why this aproach is not "high-available"?
What I'm doing (based on this module and your response above):

1. Add a floating IP
2. Add a port with admin_state_up = false similar to yours for the VIP:

   terraform-openstack-rke2/network.tf

   Lines 64 to 76 in 6b8ca67

   	resource "openstack_networking_port_v2" "dummy" {
   	# this port enables the fip to be associated dynamically via allowed_address_pairs
   	# if associated directly to a server, it will steal the ip without being ready
   	name = "${var.name}-vip"
   	network_id = openstack_networking_network_v2.net.id
   	security_group_ids = [openstack_networking_secgroup_v2.server.id]
   	admin_state_up = false
   	fixed_ip {
   	subnet_id = openstack_networking_subnet_v2.servers.id
   	ip_address = local.internal_vip
   	}
   	}

3. Associate the FIP with the port
4. Add rules for ports 80 and 443 to the agent secgroup
5. Deploy kube-vip using the helm chart and setting the address to to the VIP
6. As written above, manually add the required "allowed address pair"

From what I understand, kube-vip should now be able to do the fail-over (but not the load-balancing) expected from a load-balancer.

[zifeo]

The FIP assignment is already taken care by the existing kube-vip so no need to change anything. You just need to open the ports and install the ingress-nginx on the server nodes (no need to enable again rke2-ingress-nginx because it would anyway require some tuning but simply deploy another helm release) by making sure the service is using node ports and not a load-balancer.

[UncleSamSwiss]

Thanks for your quick response. I guessed as much (and tried that), but unfortunately using node ports is not possible as RKE2 (at least as configured in this module) only allows node ports in the 30000 range. Would I be able to make a port mapping (NATting) somewhere in OpenStack?

[UncleSamSwiss]

I guess https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/resources/networking_portforwarding_v2 should help me with the port forwarding...

[zifeo]

What is the exact error you are getting? Does it happen also with host network and host ports enabled (on the chart)? Curious also to hear if the openstack port forward could help :)

[UncleSamSwiss]

So, port forward doesn't work, as there is already a binding. Error message: `Cannot create port forwarding to floating IP 195.15.x.x (6ae87f7c-xxxx-xxxx-xxxx-f6b1f56d374a) with port d4e8c473-xxxx-xxxx-xxxx-b06a150fbc26 using fixed IP 192.168.42.4, as that port already has a binding floating IP.

But your approach of using host network and host ports worked.

Thanks a lot for your help!

[UncleSamSwiss]

Thanks to the suggestions by @zifeo, I was able to get it working with these resources in a new terraform project (not the one creating the cluster):

data "openstack_networking_secgroup_v2" "rke2_cluster_server" {
  name = "rke2-cluster-server" // change this to the name of your secgroup (based on the name of the cluster)
}

resource "openstack_networking_secgroup_rule_v2" "ingress_http" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 80
  port_range_max    = 80
  remote_ip_prefix  = "0.0.0.0/0"
  security_group_id = data.openstack_networking_secgroup_v2.rke2_cluster_server.id
}

resource "openstack_networking_secgroup_rule_v2" "ingress_https" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 443
  port_range_max    = 443
  remote_ip_prefix  = "0.0.0.0/0"
  security_group_id = data.openstack_networking_secgroup_v2.rke2_cluster_server.id
}

resource "helm_release" "ingress_nginx" {
  name             = "ingress-nginx"
  repository       = "https://kubernetes.github.io/ingress-nginx"
  chart            = "ingress-nginx"
  version          = "4.9.0"
  namespace        = "global-ingress-nginx"
  create_namespace = true
  recreate_pods    = true

  values = [
    <<-EOT
controller:
  kind: DaemonSet
  service:
    type: NodePort
  hostPort:
    enabled: true
  hostNetwork: true
  ingressClassResource:
    default: true
  watchIngressWithoutClass: true
  nodeSelector:
    node-role.kubernetes.io/master: 'true'
  tolerations:
    - key: CriticalAddonsOnly
      operator: Exists
      effect: NoExecute
    - key: node.cloudprovider.kubernetes.io/uninitialized
      value: 'true'
      effect: NoSchedule
EOT
  ]
}