Merge pull request openshift#7727 from shiftstack/add-support-byon-dualstack-upi

OSASINFRA-3261: OpenStack: support dualstack in UPI

Author: openshift-merge-bot[bot]

docs/user/openstack/install_upi.md

@@ -27,6 +27,7 @@ of this method of installation.
   - [OpenShift Configuration Directory](#openshift-configuration-directory)
   - [Red Hat Enterprise Linux CoreOS (RHCOS)](#red-hat-enterprise-linux-coreos-rhcos)
   - [API and Ingress Floating IP Addresses](#api-and-ingress-floating-ip-addresses)
+  - [Create network, API and ingress ports](#create-network-api-and-ingress-ports)
   - [Install Config](#install-config)
     - [Configure the machineNetwork.CIDR apiVIP and ingressVIP](#configure-the-machinenetworkcidr-apivip-and-ingressvip)
     - [Empty Compute Pools](#empty-compute-pools)
@@ -44,7 +45,7 @@ of this method of installation.
     - [Master Ignition](#master-ignition)
   - [Network Topology](#network-topology)
     - [Security Groups](#security-groups)
-    - [Network, Subnet and external router](#network-subnet-and-external-router)
+    - [Update Network, Subnet, Router and ports](#update-network-subnet-router-and-ports)
     - [Subnet DNS (optional)](#subnet-dns-optional)
   - [Bootstrap](#bootstrap)
   - [Control Plane](#control-plane)
@@ -59,7 +60,7 @@ of this method of installation.
 
 ## Prerequisites
 
-The file `inventory.yaml` contains the variables most likely to need customization.
+The `inventory.yaml` file contains variables which should be reviewed and adjusted if needed.
 
 > **Note**
 > Some of the default pods (e.g. the `openshift-router`) require at least two nodes so that is the effective minimum.
@@ -149,7 +150,7 @@ Make sure that `python` points to Python3:
 sudo alternatives --set python /usr/bin/python3
 ```
 
-To avoid packages not found or mismatchs, we use pip to install the dependencies:
+To avoid packages not found or mismatches, we use pip to install the dependencies:
 ```sh
 python3 -m pip install --upgrade pip
 python3 -m pip install yq openstackclient openstacksdk netaddr
@@ -227,7 +228,7 @@ access between OpenStack KVM hypervisors and the cluster nodes.
 To enable this feature, you must add the `hw_qemu_guest_agent=yes` property to the image:
 
 ```
-$ openstack image "rhcos-${CLUSTER_NAME}" set --property hw_qemu_guest_agent=yes
+$ openstack image set --property hw_qemu_guest_agent=yes "rhcos-${CLUSTER_NAME}"
 ```
 
 Finally validate that the image was successfully created:
@@ -269,6 +270,19 @@ api.openshift.example.com.    A 203.0.113.23
 
 They will need to be available to your developers, end users as well as the OpenShift installer process later in this guide.
 
+## Create network, API and ingress ports
+
+Please note that value of the API and Ingress VIPs fields will be overwritten in the `inventory.yaml` with the respective addresses assigned to the Ports. Run the following playbook to create necessary resources:
+
+<!--- e2e-openstack-upi: INCLUDE START --->
+```sh
+$ ansible-playbook -i inventory.yaml network.yaml
+```
+<!--- e2e-openstack-upi: INCLUDE END --->
+
+> **Note**
+> These OpenStack resources will be deleted by the `down-network.yaml` playbook.
+
 ## Install Config
 
 Run the `create install-config` subcommand and fill in the desired entries:
@@ -304,76 +318,51 @@ $ tree
 ```
 
 ### Configure the machineNetwork.CIDR apiVIP and ingressVIP
+
 The `machineNetwork` represents the OpenStack network which will be used to connect all the OpenShift cluster nodes.
 The `machineNetwork.CIDR` defines the IP range, in CIDR notation, from which the installer will choose what IP addresses
-to assign the nodes.  The `apiVIP` and `ingressVIP` are the IP addresses the installer will assign to the cluster API and
+to assign the nodes.  The `apiVIPs` and `ingressVIPs` are the IP addresses the installer will assign to the cluster API and
 ingress VIPs, respectively.
-In the previous steps, the installer added default values for the `machineNetwork.CIDR`, and then it picked the
-5th and 7th IP addresses from that range to assign to `apiVIP` and `ingressVIP`.
-`machineNetwork.CIDR` needs to match the IP range specified by `os_subnet_range` in the `inventory.yaml` file.
-
-When the installer creates the manifest files from an existing `install-config.yaml` file, it validates that the
-`apiVIP` and `ingressVIP` fall within the IP range specified by `machineNetwork.CIDR`. If they do not, it errors out.
-If you change the value of `machineNetwork.CIDR` you must make sure the `apiVIP` and `ingressVIP` values still fall within
-the new range. There are two options for setting the `apiVIP` and `ingressVIP`. If you know the values you want to use,
-you can specify them in the `install-config.yaml` file. If you want the installer to pick the 5th and 7th IP addresses in the
-new range, you need to remove the `apiVIP` and `ingressVIP` entries from the `install-config.yaml` file.
 
-To illustrate the process, we will use '192.0.2.0/24' as an example. It defines a usable IP range from
-192.0.2.1 to 192.0.2.254. There are some IP addresses that should be avoided because they are usually taken up or
-reserved. For example, the first address (.1) is usually assigned to a router. The DHCP and DNS servers will use a few
-more addresses, usually .2, .3, .11 and .12. The actual addresses used by these services depend on the configuration of
-the OpenStack deployment in use. You should check your OpenStack deployment.
+In the previous step, ansible playbook added default values for the
+`machineNetwork.CIDR`, and then it assigned selected by Neutron IP addresses for
+`apiVIPs` and `ingressVIPs` to appropriate fields inventory file - os_ingressVIP
+and os_apiVIP for single stack installation, and additionally os_ingressVIP6 and
+os_apiVIP6 for dualstack out of `machineNetwork.CIDR`.
 
+Following script will fill into `intall-config.yaml` the value for `machineNetwork`, `apiVIPs`, `ingressVIPs`, `controlPlanePort`
+for single-stack and dual-stack and `networkType`, `clusterNetwork` and `serviceNetwork` only for dual-stack, using `inventory.yaml`
+values:
 
-The following script modifies the value of `machineNetwork.CIDR` in the `install-config.yaml` file to match the `os_subnet_range` defined in `inventory.yaml`.
 <!--- e2e-openstack-upi: INCLUDE START --->
 ```sh
 $ python -c 'import yaml
-installconfig_path = "install-config.yaml"
-installconfig = yaml.safe_load(open(installconfig_path))
-inventory = yaml.safe_load(open("inventory.yaml"))
-inventory_subnet_range = inventory["all"]["hosts"]["localhost"]["os_subnet_range"]
-installconfig["networking"]["machineNetwork"][0]["cidr"] = inventory_subnet_range
-open(installconfig_path, "w").write(yaml.dump(installconfig, default_flow_style=False))'
-```
-<!--- e2e-openstack-upi: INCLUDE END --->
-
-Next, we need to correct the `apiVIP` and `ingressVIP` values.
-
-The following script will clear the values from the `install-config.yaml` file so that the installer will pick
-the 5th and 7th IP addresses in the new range, 192.0.2.5 and 192.0.2.7.
-<!--- e2e-openstack-upi: INCLUDE START --->
-```sh
-$ python -c 'import yaml
-import sys
 path = "install-config.yaml"
 data = yaml.safe_load(open(path))
-if "apiVIP" in data["platform"]["openstack"]:
-   del data["platform"]["openstack"]["apiVIP"]
-if "ingressVIP" in data["platform"]["openstack"]:
-   del data["platform"]["openstack"]["ingressVIP"]
+inventory = yaml.safe_load(open("inventory.yaml"))["all"]["hosts"]["localhost"]
+machine_net = [{"cidr": inventory["os_subnet_range"]}]
+api_vips = [inventory["os_apiVIP"]]
+ingress_vips = [inventory["os_ingressVIP"]]
+ctrl_plane_port = {"network": {"name": inventory["os_network"]}, "fixedIPs": [{"subnet": {"name": inventory["os_subnet"]}}]}
+if inventory.get("os_subnet6"):
+    machine_net.append({"cidr": inventory["os_subnet6_range"]})
+    api_vips.append(inventory["os_apiVIP6"])
+    ingress_vips.append(inventory["os_ingressVIP6"])
+    data["networking"]["networkType"] = "OVNKubernetes"
+    data["networking"]["clusterNetwork"].append({"cidr": inventory["cluster_network6_cidr"], "hostPrefix": inventory["cluster_network6_prefix"]})
+    data["networking"]["serviceNetwork"].append(inventory["service_subnet6_range"])
+    ctrl_plane_port["fixedIPs"].append({"subnet": {"name": inventory["os_subnet6"]}})
+data["networking"]["machineNetwork"] = machine_net
+data["platform"]["openstack"]["apiVIPs"] = api_vips
+data["platform"]["openstack"]["ingressVIPs"] = ingress_vips
+data["platform"]["openstack"]["controlPlanePort"] = ctrl_plane_port
+del data["platform"]["openstack"]["externalDNS"]
 open(path, "w").write(yaml.dump(data, default_flow_style=False))'
 ```
 <!--- e2e-openstack-upi: INCLUDE END --->
 
-If you want to specify the values yourself, you can use the following script, which sets them to 192.0.2.8
-and 192.0.2.9.
-
-```sh
-$ python -c 'import yaml
-import sys
-path = "install-config.yaml"
-data = yaml.safe_load(open(path))
-if "apiVIP" in data["platform"]["openstack"]:
-   data["platform"]["openstack"]["apiVIP"] = "192.0.2.8"
-if "ingressVIP" in data["platform"]["openstack"]:
-   data["platform"]["openstack"]["ingressVIP"] = "192.0.2.9"
-open(path, "w").write(yaml.dump(data, default_flow_style=False))'
-```
-
 > **Note**
-> All the scripts in this guide work with Python 3 as well as Python 2.
+> All the scripts in this guide work only with Python 3.
 > You can also choose to edit the `install-config.yaml` file by hand.
 
 ### Empty Compute Pools
@@ -398,7 +387,7 @@ open(path, "w").write(yaml.dump(data, default_flow_style=False))'
 
 By default the `networkType` is set to `OVNKubernetes` on the `install-config.yaml`.
 
-If an installation with OpenShift SDN is desired, you must modify the `networkType` field.
+If an installation with OpenShift SDN is desired, you must modify the `networkType` field. Note, that dual-stack only supports `OVNKubernetes` network type.
 
 This command will do it for you:
 
@@ -689,12 +678,12 @@ Create a file called `$INFRA_ID-bootstrap-ignition.json` (fill in your `infraID`
     "config": {
       "merge": [
         {
-	  "httpHeaders": [
-	    {
-	      "name": "X-Auth-Token",
-	      "value": "${GLANCE_TOKEN}"
-	    }
-	  ],
+          "httpHeaders": [
+            {
+              "name": "X-Auth-Token",
+              "value": "${GLANCE_TOKEN}"
+            }
+          ],
           "source": "${BOOTSTRAP_URL}"
         }
       ]
@@ -755,7 +744,7 @@ else:
 
 infra_id = os.environ.get('INFRA_ID', 'openshift')
 
-bootstrap_ignition_shim = infra_id+'-bootstrap-ignition.json'
+bootstrap_ignition_shim = infra_id + '-bootstrap-ignition.json'
 
 with open(bootstrap_ignition_shim, 'r') as f:
     ignition_data = json.load(f)
@@ -821,18 +810,14 @@ $ ansible-playbook -i inventory.yaml security-groups.yaml
 <!--- e2e-openstack-upi: INCLUDE END --->
 
 The playbook creates one Security group for the Control Plane and one for the Compute nodes, then attaches rules for enabling communication between the nodes.
-### Network, Subnet and external router
+### Update Network, Subnet, Router and ports
 <!--- e2e-openstack-upi: INCLUDE START --->
 ```sh
-$ ansible-playbook -i inventory.yaml network.yaml
+$ ansible-playbook -i inventory.yaml update-network-resources.yaml
 ```
 <!--- e2e-openstack-upi: INCLUDE END --->
 
-The playbook creates a network and a subnet. The subnet obeys `os_subnet_range`; however the first ten IP addresses are removed from the allocation pool. These addresses will be used for the VRRP addresses managed by keepalived for high availability. For more information, read the [networking infrastructure design document][net-infra].
-
-Outside connectivity will be provided by attaching the floating IP addresses (IPs in the inventory) to the corresponding routers.
-
-[net-infra]: https://github.com/openshift/installer/blob/master/docs/design/openstack/networking-infrastructure.md
+The playbook sets tags to network, subnets, ports and router. It also attaches the floating IP to the API and Ingress ports and set the security group on those ports.
 
 ### Subnet DNS (optional)
 

upi/openstack/bootstrap.yaml

@@ -19,6 +19,18 @@
       - "{{ os_sg_master }}"
       allowed_address_pairs:
       - ip_address: "{{ os_apiVIP }}"
+    when: os_subnet6 is not defined
+
+  - name: 'Create the bootstrap dualstack server port'
+    os_port:
+      name: "{{ os_port_bootstrap }}"
+      network: "{{ os_network }}"
+      security_groups:
+      - "{{ os_sg_master }}"
+      allowed_address_pairs:
+      - ip_address: "{{ os_apiVIP }}"
+      - ip_address: "{{ os_apiVIP6 }}"
+    when: os_subnet6 is defined
 
   - name: 'Set bootstrap port tag'
     command:

upi/openstack/common.yaml

@@ -10,12 +10,6 @@
       cluster_id_tag: "openshiftClusterID={{ infraID }}"
       primary_cluster_network_tag: "{{ infraID }}-primaryClusterNetwork"
       os_infra_id: "{{ infraID }}"
-      os_network: "{{ infraID }}-network"
-      os_subnet: "{{ infraID }}-nodes"
-      os_router: "{{ infraID }}-external-router"
-      # Port names
-      os_port_api: "{{ infraID }}-api-port"
-      os_port_ingress: "{{ infraID }}-ingress-port"
       os_port_bootstrap: "{{ infraID }}-bootstrap-port"
       os_port_master: "{{ infraID }}-master-port"
       os_port_worker: "{{ infraID }}-worker-port"

upi/openstack/compute-nodes.yaml

@@ -21,6 +21,20 @@
       - ip_address: "{{ os_ingressVIP }}"
     with_indexed_items: "{{ [os_port_worker] * os_compute_nodes_number }}"
     register: ports
+    when: os_subnet6 is not defined
+
+  - name: 'Create the dualstack Compute ports'
+    openstack.cloud.port:
+      name: "{{ item.1 }}-{{ item.0 }}"
+      network: "{{ os_network }}"
+      security_groups:
+      - "{{ os_sg_worker }}"
+      allowed_address_pairs:
+      - ip_address: "{{ os_ingressVIP }}"
+      - ip_address: "{{ os_ingressVIP6 }}"
+    with_indexed_items: "{{ [os_port_worker] * os_compute_nodes_number }}"
+    register: ports
+    when: os_subnet6 is defined
 
   - name: 'Set Compute ports tag'
     ansible.builtin.command:

upi/openstack/control-plane.yaml

@@ -22,6 +22,22 @@
       - ip_address: "{{ os_ingressVIP }}"
     with_indexed_items: "{{ [os_port_master] * os_cp_nodes_number }}"
     register: ports
+    when: os_subnet6 is not defined
+
+  - name: 'Create the dualstack Control Plane ports'
+    openstack.cloud.port:
+      name: "{{ item.1 }}-{{ item.0 }}"
+      network: "{{ os_network }}"
+      security_groups:
+      - "{{ os_sg_master }}"
+      allowed_address_pairs:
+      - ip_address: "{{ os_apiVIP }}"
+      - ip_address: "{{ os_apiVIP6 }}"
+      - ip_address: "{{ os_ingressVIP }}"
+      - ip_address: "{{ os_ingressVIP6 }}"
+    with_indexed_items: "{{ [os_port_master] * os_cp_nodes_number }}"
+    register: ports
+    when: os_subnet6 is defined
 
   - name: 'Set Control Plane ports tag'
     ansible.builtin.command:

upi/openstack/inventory.yaml

@@ -4,10 +4,15 @@ all:
       ansible_connection: local
       ansible_python_interpreter: "{{ansible_playbook_python}}"
 
+      # Network resource names
+      os_network: ocp-network
+      os_port_api: ocp-api-port
+      os_port_ingress: ocp-ingress-port
+      os_router: ocp-external-router
+      os_subnet: ocp-subnet-v4
+
       # User-provided values
       os_subnet_range: '10.0.0.0/16'
-      # uncomment for dual stack
-      # os_subnet6_range: 'd2e:6f44:5dd8:c956::/64'
       os_flavor_master: 'm1.xlarge'
       os_flavor_worker: 'm1.large'
       os_image_rhcos: 'rhcos'
@@ -52,15 +57,43 @@ all:
       # in case of install failure.
       os_bootstrap_fip: '203.0.113.20'
 
-      # An IP address that will be assigned to the API VIP.
+      # An IPv4 address that will be assigned to the API VIP.
       # Be aware that the 10 and 11 of the machineNetwork will
       # be taken by neutron dhcp by default, and wont be available.
+      # This value will be overwritten by the network.yaml playbook.
       os_apiVIP: "{{ os_subnet_range | ansible.utils.next_nth_usable(5) }}"
 
-      # An IP address that will be assigned to the ingress VIP.
+      # An IPv4 address that will be assigned to the ingress VIP.
       # Be aware that the 10 and 11 of the machineNetwork will
       # be taken by neutron dhcp by default, and wont be available.
+      # This value will be overwritten by the network.yaml playbook.
       os_ingressVIP: "{{ os_subnet_range | ansible.utils.next_nth_usable(7) }}"
 
-      # Set control-plane nodes to schedule workloads when number of compute nodes is zero
+      # Set control-plane nodes to schedule workloads when number of compute
+      # nodes is zero
       os_master_schedulable: "{{ os_compute_nodes_number | int == 0 }}"
+
+      # Name of the IPv6 subnet. Uncomment to enable dual-stack support
+      #os_subnet6: ocp-subnet-v6
+
+      # IPv6 subnet CIDR
+      os_subnet6_range: 'fd2e:6f44:5dd8:c956::/64'
+
+      # Modes are one of: slaac, dhcpv6-stateful or dhcpv6-stateless
+      os_subnet6_address_mode: slaac
+      os_subnet6_router_advertisements_mode: slaac
+
+      # IPv6 service subnet cidr
+      service_subnet6_range: 'fd02::/112'
+
+      # IPv6 cluster network details
+      cluster_network6_cidr: 'fd01::/48'
+      cluster_network6_prefix: 64
+
+      # An IPv6 address that will be assigned to the API VIP.
+      # This value will be overwritten by the network.yaml playbook.
+      os_apiVIP6: ""
+
+      # An IPv6 address that will be assigned to the ingress VIP.
+      # This value will be overwritten by the network.yaml playbook.
+      os_ingressVIP6: ""