Merge pull request #1638 from RichardORCL/main

Added information on how to have multiple egress IPs and use OCI vLANs with Openshift

Author: fhasan80

cloud-infrastructure/virtualization-solutions/openshift-on-oci/README.md

@@ -14,6 +14,7 @@ Reviewed: 06.11.2024
 
 ## Reference Architectures & Step-by-step Guides
 
+- [Use multiple and floating Egress IP(s) by leveraging OCI VLANs](openshift-floating-egress-ip/README.md)
 - [Enable Seamless Access to Red Hat OpenShift Container Platform on OCI from On-Premises to VCNs in the Same Region](https://docs.oracle.com/en/learn/oci-openshift-vcn/)
 
 ## OpenShift Day-2 - Operations

cloud-infrastructure/virtualization-solutions/openshift-on-oci/openshift-floating-egress-ip/README.md

@@ -0,0 +1,236 @@
+# OCI-OpenShift-Floating-EgressIP
+
+If you want to use a fixed egress IP address in openshift for your pods, you need to make sure that the egress IP is "floatable" across your worker nodes. 
+In case the host fails that was assigned to the EgressIP. 
+
+Traditionally this is difficult to do in a public cloud using Layer 3 networking, as the cloud control plane controls
+the assigned IP Addresses and another instance can not just reclaim an IP address in case of failure. 
+
+The Oracle Cloud (OCI) does not just support Layer 3 network subnets, but also has the capability to support Layer 2 VLANs.
+In these VLANs any instance can claim IP addresses using normal ARP. Initially this VLAN concept was designed for use with 
+the Oracle Cloud VMware Solution, but it can also be used for normal OCI Compute Instances (VMs and Bare metal servers).
+https://docs.oracle.com/en-us/iaas/Content/VMware/Tasks/ocvsmanagingl2net.htm
+
+We can use these VLANs for our OpenShift worker nodes as secondary NIC interfaces and route the EgressIP traffic over.
+
+
+## Create VLAN and assign addition vNIC(s) to worker nodes attached to the VLAN in OCI
+
+1. Create a VLAN and assign it a CIDR range. The first available address in this CIDR range will be used by OCI for the gateway service.
+All other addresses are free to be used. There is no DHCP service in the VLAN.
+
+In this example, we have created a VLAN with the CIDR: 10.0.101.0/24. Hence the default gateway of this segment will be 10.0.101.1
+
+2. On each worker node VM add a vNIC attached to the created VLAN
+
+**TIP: Automate Step 1 and Step 2**: Use the provided [vlan.tf](tf_Create_and_Add_VLAN/vlan.tf) file to automate the creation of the VLAN and the assignment of vNICs to worker nodes. This Terraform script will handle the setup for you, ensuring that the VLAN is created with the specified CIDR range and that each worker node VM has a vNIC attached to the created VLAN.
+
+
+3. Login to one of the worker nodes and confirm a 2nd NIC is visible and note down the device name (In this example it is ens5)
+
+```
+oc debug node/[your node name]
+chroot /host
+ifconfig
+```
+
+
+<img src="images/vnicsetup.jpg">
+
+
+## Setup IP configuration for worker nodes for the new VLAN vNICs
+
+1. Install openshift-nmstate operator via the operator hub. After installation, click on 'create instance' on the operator and click 'create'
+
+Check if the NodeNetworkConfigurationPolicy CRD exists
+
+linux: 
+
+```oc get crds | grep nodenetworkconfigurationpolicies```
+
+windows: 
+
+```oc get crds | findstr nodenetworkconfigurationpolicies```
+
+<img src="images/netconfigpolicy.jpg">
+
+2. Create node_config.yaml file:
+
+You can create 1 file per worker node or just add multiple worker nodes into one file. The example below show the configuration of 2 workers nodes.
+
+**IMPORTANT:** Ensure you use the correct device name, in my example it is 'ens5', but check in your environment. Set the correct IP address and prefix. 
+
+**IMPORTANT:** All nodes that can run worload pods must have VLAN attached to them and have an individual IP setup in this VLAN
+
+Reminder: OCI does not provide DHCP on a VLAN, so you need to maintain assigned IP addresses yourself. You need 1 IP address per worker node and you need 1 IP address per egress IP you want to use.
+
+The next-hop-address needs to be set to the VLANs gateway, which is the first available address in the assigned CIDR block of the VLAN. In this example this is 10.0.101.1
+
+```
+apiVersion: nmstate.io/v1
+kind: NodeNetworkConfigurationPolicy
+metadata:
+  name: node1-ens5-vlan-static-ip
+spec:
+  nodeSelector:
+    kubernetes.io/hostname: os-compute-1.private.openshiftvcn.oraclevcn.com
+  desiredState:
+    interfaces:
+      - name: ens5
+        type: ethernet
+        state: up
+        ipv4:
+          enabled: true
+          address:
+            - ip: 10.0.101.11
+              prefix-length: 24
+          dhcp: false
+        mtu: 9000
+    routes:
+      config:
+        - destination: 0.0.0.0/0
+          next-hop-address: 10.0.101.1
+          next-hop-interface: ens5
+          metric: 500
+---
+apiVersion: nmstate.io/v1
+kind: NodeNetworkConfigurationPolicy
+metadata:
+  name: node2-ens5-vlan-static-ip
+spec:
+  nodeSelector:
+    kubernetes.io/hostname: os-compute-2.private.openshiftvcn.oraclevcn.com
+  desiredState:
+    interfaces:
+      - name: ens5
+        type: ethernet
+        state: up
+        ipv4:
+          enabled: true
+          address:
+            - ip: 10.0.101.12
+              prefix-length: 24
+          dhcp: false
+        mtu: 9000
+    routes:
+      config:
+        - destination: 0.0.0.0/0
+          next-hop-address: 10.0.101.1
+          next-hop-interface: ens5
+          metric: 500
+---
+
+```
+
+3. Apply the configuration:
+```
+oc apply -f node_config.yaml
+```
+
+4. enable ipForwarding to the network.operator
+
+run
+```
+oc edit network.operator
+```
+
+add the line: "ipForwarding: Global"  directly after the gatewayConfig line
+
+```
+			  defaultNetwork:
+			    ovnKubernetesConfig:
+			      egressIPConfig: {}
+			      gatewayConfig:
+			        ipForwarding: Global
+```
+
+The Opsnshift environment and its worker nodes are now setup correctly to use the VLANs and have the egressIP(s) float on it.
+
+## Setup EgressIP(s) and assign them to namespaces
+
+## Egress Labels 
+We first need to allow the hosts you want to use as egress controllers. We can do this by assigning them
+the correct k8s label:
+
+```
+oc get nodes --show-labels | grep egress
+oc label node <node-name> k8s.ovn.org/egress-assignable=true
+```
+
+Ensure you have set the egress-assignable=true label to all the hosts you have connected to the VLAN(s)
+
+## Namespaces
+Lets create 2 example namespaces, each with it's own deployement in it and a label called egress.
+
+```
+oc new-project test1
+oc label namespace test1 egress=egress_ip1
+oc new-app httpd
+
+oc new-project test2
+oc label namespace test2 egress=egress_ip2
+oc new-app httpd
+```
+
+We can not create an egressip config file and apply it to the OpenShift cluster.
+Example egressip.yaml
+
+```
+apiVersion: k8s.ovn.org/v1
+kind: EgressIP
+metadata:
+  name: egressips-oci-vlan-01
+spec:
+  egressIPs:
+  - 10.0.101.101
+  namespaceSelector:
+    matchLabels:
+      egress: egress_ip1
+---
+apiVersion: k8s.ovn.org/v1
+kind: EgressIP
+metadata:
+  name: egressips-oci-vlan-02
+spec:
+  egressIPs:
+  - 10.0.101.102
+  namespaceSelector:
+    matchLabels:
+      egress: egress_ip2
+```
+
+Create the above egressip.yaml and apply to your cluster
+```
+oc apply -f egressip.yaml
+```
+
+You can not check if the EgressIPs are running and from what node the out going traffic goes.
+```
+oc get egressip
+```
+
+<img src="./images/egressips.jpg">
+
+Having completed this setup, the 2 projects will both now have a unique fixed EgressIP. The workernode routing the Egress traffic can now fail over to any other worker node fully automatically.
+
+## Having Public IP addresses for the EgressIP
+
+Using the external access feature on the VLAN in OCI, you can assign each private IP Address to a Public IP Address. Do make sure that the VLAN has a Internet Gateway service (not NAT Gateway)
+
+<img src='images/publicip.jpg'>
+
+Now when you pods communicate with resources on the internet, per namespace you will have different source public IP Addresses as well.
+
+<img src='images/result.jpg'>
+
+## Known limitation
+
+The amount of VNICs attached to a compute instance (worker node) is tied to the size of the Compute instance: https://docs.oracle.com/en-us/iaas/Content/Compute/References/computeshapes.htm#vm-standard
+
+If you want to connect to multiple VLANs, keep this limit in mind. There is no restriction to how many EgressIPs you can use within a VLAN. 
+
+
+
+
+
+

cloud-infrastructure/virtualization-solutions/openshift-on-oci/openshift-floating-egress-ip/images/egressips.jpg

@@ -0,0 +1,93 @@
+#====================================================
+# Create VLAN and assign to OpenShift Worker Nodes 
+#====================================================
+
+terraform {
+  required_providers {
+    oci = {
+      source  = "oracle/oci"
+      version = "~> 5.0"
+    }
+  }
+}
+
+provider "oci" {
+}
+
+#====================================================
+# Define variables
+#====================================================
+variable "OpenShiftNameSpace" {
+  description = "The namespace for the OpenShift environment"
+  type        = string
+  default     = "---OCI Namespace that is being used for your OpenShift Cluster---"
+}
+
+variable "compartment_id" {
+  description = "The compartment OCID in which to create resources"
+  type        = string
+  default     = "---OCID of the Compartment the OpenShift cluster is in---"
+}
+
+variable "vcn_id" {
+  description = "The OCID of the VCN in which the VLAN (subnet) will be created"
+  type        = string
+  default     = "---OCID of the VCN your are using---"
+}
+
+variable "vlan_cidr" {
+  description = "The CIDR block for the VLAN (subnet) to be created"
+  type        = string
+  default     = "10.0.101.0/24"  # Change to CIDR range that is part of your VCN and is still available
+}
+
+variable "vlan_name" {
+  description = "Name of the VLAN"
+  type        = string
+  default     = "EgressVLAN"   
+}
+
+#====================================================
+# Create VLAN in existing (specified) VCN
+#====================================================
+resource "oci_core_vlan" "new_vlan" {
+  compartment_id = var.compartment_id
+  vcn_id         = var.vcn_id
+  cidr_block     = var.vlan_cidr
+  display_name   = "vlan-${var.vlan_name}"
+}
+
+#====================================================
+# DATA SOURCE: GET Openshift Worker Nodes based on tag
+#====================================================
+data "oci_core_instances" "compute_instances" {
+  compartment_id = var.compartment_id
+
+  filter {
+    # Use defined tag filtering syntax: defined_tags.<namespace>.<tag-key>
+    name   = "defined_tags.${var.OpenShiftNameSpace}.instance-role"
+    values = ["compute"]
+  }
+}
+
+#====================================================
+# LOCALS: Create list of Worker nodes instances
+#====================================================
+locals {
+  compute_instances = {
+    for instance in data.oci_core_instances.compute_instances.instances :
+    instance.id => instance
+  }
+}
+
+#====================================================
+# Attach VLAN to each worker node
+#====================================================
+resource "oci_core_vnic_attachment" "vlan_vnic" {
+  for_each = local.compute_instances
+    instance_id = each.key
+    create_vnic_details{
+      display_name = "vlan-${var.vlan_name}"    
+      vlan_id = oci_core_vlan.new_vlan.id
+    }
+}