Merge pull request #41484 from maxwelldb/osp-ovs-dpdk-worker-osdocs-3059

Bob Furu · web-flow · commit eb673db07557 · 2022-03-08T09:29:05.000-05:00
diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml
@@ -343,6 +343,8 @@ Topics:
     File: installing-openstack-installer-kuryr
   - Name: Installing a cluster that supports SR-IOV compute machines on OpenStack
     File: installing-openstack-installer-sr-iov
+  - Name: Installing a cluster on OpenStack that supports DPDK-connected compute machines
+    File: installing-openstack-installer-ovs-dpdk
   - Name: Installing a cluster on OpenStack on your own infrastructure
     File: installing-openstack-user
   - Name: Installing a cluster on OpenStack with Kuryr on your own infrastructure
diff --git a/installing/installing_openstack/installing-openstack-installer-ovs-dpdk.adoc b/installing/installing_openstack/installing-openstack-installer-ovs-dpdk.adoc
@@ -0,0 +1,88 @@
+:_content-type: ASSEMBLY
+[id="installing-openstack-installer-ovs-dpdk"]
+= Installing a cluster on OpenStack that supports DPDK-connected compute machines
+include::modules/common-attributes.adoc[]
+:context: installing-openstack-installer-ovs-dpdk
+
+toc::[]
+
+:FeatureName: Installing a cluster on {rh-openstack} that supports DPDK-connected compute machines
+
+include::snippets/technology-preview.adoc[]
+
+If your {rh-openstack-first} deployment has Open vSwitch with the Data Plane Development Kit (OVS-DPDK) enabled, you can install an {product-title} cluster on it. Clusters that run on such {rh-openstack} deployments use OVS-DPDK features by providing access to link:https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html[poll mode drivers]. 
+
+== Prerequisites
+
+* Review details about the
+xref:../../architecture/architecture-installation.adoc#architecture-installation[{product-title} installation and update]
+processes.
+** Verify that {product-title} {product-version} is compatible with your {rh-openstack} version by using the "Supported platforms for OpenShift clusters" section. You can also compare platform support across different versions by viewing the link:https://access.redhat.com/articles/4679401[{product-title} on {rh-openstack} support matrix].
+
+* Have a storage service installed in {rh-openstack}, like block storage (Cinder) or object storage (Swift). Object storage is the recommended storage technology for {product-title} registry cluster deployment. For more information, see xref:../../scalability_and_performance/optimizing-storage.adoc#optimizing-storage[Optimizing storage].
+
+* Have the metadata service enabled in {rh-openstack}.
+
+* Plan your {rh-openstack} OVS-DPDK deployment by referring to link:https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/network_functions_virtualization_planning_and_configuration_guide/assembly_ovsdpdk_parameters[Planning your OVS-DPDK deployment] in the Network Functions Virtualization Planning and Configuration Guide.
+
+* Configure your {rh-openstack} OVS-DPDK deployment according to link:https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/network_functions_virtualization_planning_and_configuration_guide/part-dpdk-configure[Configuring an OVS-DPDK deployment] in the Network Functions Virtualization Planning and Configuration Guide.
+
+** You must complete link:https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/network_functions_virtualization_planning_and_configuration_guide/part-dpdk-configure#p-ovs-dpdk-flavor-deploy-instance[Creating a flavor and deploying an instance for OVS-DPDK] before you install a cluster on {rh-openstack}. 
+
+include::modules/installation-osp-default-deployment.adoc[leveloffset=+1]
+include::modules/installation-osp-control-compute-machines.adoc[leveloffset=+2]
+include::modules/installation-osp-bootstrap-machine.adoc[leveloffset=+2]
+include::modules/cluster-entitlements.adoc[leveloffset=+1]
+include::modules/installation-osp-enabling-swift.adoc[leveloffset=+1]
+include::modules/installation-osp-verifying-external-network.adoc[leveloffset=+1]
+include::modules/installation-osp-describing-cloud-parameters.adoc[leveloffset=+1]
+include::modules/installation-obtaining-installer.adoc[leveloffset=+1]
+include::modules/installation-initializing.adoc[leveloffset=+1]
+include::modules/installation-configure-proxy.adoc[leveloffset=+2]
+include::modules/installation-configuration-parameters.adoc[leveloffset=+1]
+include::modules/installation-osp-custom-subnet.adoc[leveloffset=+2]
+include::modules/installation-osp-deploying-bare-metal-machines.adoc[leveloffset=+2]
+include::modules/installation-osp-config-yaml.adoc[leveloffset=+2]
+include::modules/ssh-agent-using.adoc[leveloffset=+1]
+include::modules/installation-osp-accessing-api.adoc[leveloffset=+1]
+include::modules/installation-osp-accessing-api-floating.adoc[leveloffset=+2]
+include::modules/installation-osp-accessing-api-no-floating.adoc[leveloffset=+2]
+include::modules/installation-osp-configuring-sr-iov.adoc[leveloffset=+1]
+include::modules/installation-launching-installer.adoc[leveloffset=+1]
+include::modules/installation-osp-verifying-cluster-status.adoc[leveloffset=+1]
+include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1]
+
+The cluster is operational. Before you can add OVS-DPDK compute machines though, you must perform additional tasks.
+
+include::modules/networking-osp-enabling-metadata.adoc[leveloffset=+1]
+include::modules/networking-osp-enabling-vfio-noiommu.adoc[leveloffset=+1]
+include::modules/installation-osp-dpdk-binding-vfio-pci.adoc[leveloffset=+1]
+include::modules/installation-osp-dpdk-exposing-host-interface.adoc[leveloffset=+1]
+
+.Additional resources
+
+* xref:../../networking/multiple_networks/configuring-additional-network.adoc#nw-multus-host-device-object_configuring-additional-network[Creating an additional network attachment with the Cluster Network Operator]
+
+The cluster is installed and prepared for configuration. You must now perform the OVS-DPDK configuration tasks in <<next-steps_installing-openstack-installer-ovs-dpdk, Next steps>>.
+
+include::modules/cluster-telemetry.adoc[leveloffset=+1]
+
+.Additional resources
+
+* See xref:../../support/remote_health_monitoring/about-remote-health-monitoring.adoc#about-remote-health-monitoring[About remote health monitoring] for more information about the Telemetry service
+
+[id="additional-resources_installing-openstack-installer-ovs-dpdk"]
+== Additional resources
+* See xref:../../scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.adoc#cnf-understanding-low-latency_cnf-master[Performance Addon Operator for low latency nodes] for information about configuring your deployment for real-time running and low latency.
+
+[id="next-steps_installing-openstack-installer-ovs-dpdk"]
+== Next steps
+
+* To complete OVS-DPDK configuration for your cluster:
+** xref:../../scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.adoc#installing-the-performance-addon-operator_cnf-master[Install the Performance Addon Operator].
+** xref:../../scalability_and_performance/what-huge-pages-do-and-how-they-are-consumed-by-apps.adoc#what-huge-pages-do_huge-pages[Configure the Performance Addon Operator with huge pages support].
+* xref:../../post_installation_configuration/cluster-tasks.adoc#available_cluster_customizations[Customize your cluster].
+* If necessary, you can
+xref:../../support/remote_health_monitoring/opting-out-of-remote-health-reporting.adoc#opting-out-remote-health-reporting_opting-out-remote-health-reporting[opt out of remote health reporting].
+* If you need to enable external access to node ports, xref:../../networking/configuring_ingress_cluster_traffic/configuring-ingress-cluster-traffic-nodeport.adoc#nw-using-nodeport_configuring-ingress-cluster-traffic-nodeport[configure ingress cluster traffic by using a node port].
+* If you did not configure {rh-openstack} to accept application traffic over floating IP addresses, xref:../../post_installation_configuration/network-configuration.adoc#installation-osp-configuring-api-floating-ip_post-install-network-configuration[configure {rh-openstack} access with floating IP addresses].
diff --git a/modules/installation-osp-control-compute-machines.adoc b/modules/installation-osp-control-compute-machines.adoc
@@ -10,6 +10,9 @@ endif::[]
 ifeval::["{context}" == "installing-openstack-installer-sr-iov"]
 :osp-sr-iov:
 endif::[]
+ifeval::["{context}" == "installing-openstack-installer-ovs-dpdk"]
+:osp-sr-iov:
+endif::[]
 
 [id="installation-osp-control-compute-machines_{context}"]
 = Control plane and compute machines
@@ -49,3 +52,6 @@ endif::[]
 ifeval::["{context}" == "installing-openstack-installer-sr-iov"]
 :!osp-sr-iov:
 endif::[]
+ifeval::["{context}" == "installing-openstack-installer-ovs-dpdk"]
+:!osp-sr-iov:
+endif::[]
diff --git a/modules/installation-osp-dpdk-binding-vfio-pci.adoc b/modules/installation-osp-dpdk-binding-vfio-pci.adoc
@@ -0,0 +1,193 @@
+:_content-type: PROCEDURE
+[id="installation-osp-dpdk-binding-vfio-pci_{context}"]
+= Binding the vfio-pci kernel driver to NICs
+
+Compute machines that connect to a virtual function I/O (VFIO) network require the `vfio-pci` kernel driver to be bound to the ports that are attached to a configured network. Create a machine set for workers that attach to this VFIO network.
+
+.Procedure
+
+. From a command line, retrieve VFIO network UUIDs:
++
+[source,terminal]
+----
+$ openstack network show <VFIO_network_name> -f value -c id
+----
+
+. Create a machine set on your cluster from the following template:
++
+[%collapsible]
+====
+[source,yaml]
+----
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: worker
+  name: 99-vhostuser-bind
+spec:
+  config:
+    ignition:
+      version: 2.2.0
+    systemd:
+      units:
+      - name: vhostuser-bind.service
+        enabled: true
+        contents: |
+          [Unit]
+          Description=Vhostuser Interface vfio-pci Bind
+          Wants=network-online.target
+          After=network-online.target ignition-firstboot-complete.service
+          [Service]
+          Type=oneshot
+          EnvironmentFile=/etc/vhostuser-bind.conf
+          ExecStart=/usr/local/bin/vhostuser $ARG
+          [Install]
+          WantedBy=multi-user.target
+    storage:
+      files:
+      - contents:
+          inline: vfio-pci
+        filesystem: root
+        mode: 0644
+        path: /etc/modules-load.d/vfio-pci.conf
+      - contents:
+          inline: |
+            #!/bin/bash
+            set -e
+            if [[ "$#" -lt 1 ]]; then
+                echo "Nework ID not provided, nothing to do"
+                exit
+            fi
+            
+            source /etc/vhostuser-bind.conf
+            
+            NW_DATA="/var/config/openstack/latest/network_data.json"
+            if [ ! -f ${NW_DATA} ]; then
+                echo "Network data file not found, trying to download it from nova metadata"
+                if ! curl http://169.254.169.254/openstack/latest/network_data.json > /tmp/network_data.json; then
+                    echo "Failed to download network data file"
+                    exit 1
+                fi
+                NW_DATA="/tmp/network_data.json"
+            fi
+            function parseNetwork() {
+                local nwid=$1
+                local pcis=()
+                echo "Network ID is $nwid"
+                links=$(jq '.networks[] | select(.network_id == "'$nwid'") | .link' $NW_DATA)
+                if [ ${#links} -gt 0 ]; then
+                    for link in $links; do
+                        echo "Link Name: $link"
+                        mac=$(jq -r '.links[] | select(.id == '$link') | .ethernet_mac_address'  $NW_DATA)
+                        if [ -n $mac ]; then
+                            pci=$(bindDriver $mac)
+                            pci_ret=$?
+                            if [[ "$pci_ret" -eq 0 ]]; then
+                                echo "$pci bind succesful"
+                            fi
+                        fi
+                    done
+                fi
+            }
+            
+            function bindDriver() {
+                local mac=$1
+                for file in /sys/class/net/*; do
+                    dev_mac=$(cat $file/address)
+                    if [[ "$mac" == "$dev_mac" ]]; then
+                        name=${file##*\/}
+                        bus_str=$(ethtool -i $name | grep bus)
+                        dev_t=${bus_str#*:}
+                        dev=${dev_t#[[:space:]]}
+            
+                        echo $dev
+            
+                        devlink="/sys/bus/pci/devices/$dev"
+                        syspath=$(realpath "$devlink")
+                        if [ ! -f "$syspath/driver/unbind" ]; then
+                            echo "File $syspath/driver/unbind not found"
+                            return 1
+                        fi
+                        if ! echo "$dev">"$syspath/driver/unbind"; then
+                            return 1
+                        fi
+            
+                        if [ ! -f "$syspath/driver_override" ]; then
+                            echo "File $syspath/driver_override not found"
+                            return 1
+                        fi
+                        if ! echo "vfio-pci">"$syspath/driver_override"; then
+                            return 1
+                        fi
+            
+                        if [ ! -f "/sys/bus/pci/drivers/vfio-pci/bind" ]; then
+                            echo "File /sys/bus/pci/drivers/vfio-pci/bind not found"
+                            return 1
+                        fi
+                        if ! echo "$dev">"/sys/bus/pci/drivers/vfio-pci/bind"; then
+                          return 1
+                        fi
+                        return 0
+                    fi
+                done
+                return 1
+            }
+            
+            for nwid in "$@"; do
+                parseNetwork $nwid
+            done
+        filesystem: root
+        mode: 0744
+        path: /usr/local/bin/vhostuser
+      - contents:
+          inline: |
+            ARG="be22563c-041e-44a0-9cbd-aa391b439a39,ec200105-fb85-4181-a6af-35816da6baf7" <1> 
+        filesystem: root
+        mode: 0644
+        path: /etc/vhostuser-bind.conf
+----
+<1> Replace this value with a comma-separated list of VFIO network UUIDs. 
+====
++
+On boot for machines that are part of this set, the MAC addresses of ports are translated into PCI bus IDs. The `vfio-pci` module is bound to any port that is assocated with a network that is identified by the {rh-openstack} network ID.
+
+.Verification
+
+. On a compute node, from a command line, retrieve the name of the node by entering:
++
+[source,terminal]
+----
+$ oc get nodes
+----
+
+. Create a shell to debug the node:
++
+[source,terminal]
+----
+$ oc debug node/<node_name>
+----
+
+. Change the root directory for the current running process:
++
+[source,terminal]
+----
+$ chroot /host
+----
+
+. Enter the following command to list the kernel drivers that are handling each device on your machine:
++
+[source,terminal]
+----
+$ lspci -k
+----
++
+.Example output
+[source,terminal]
+----
+00:07.0 Ethernet controller: Red Hat, Inc. Virtio network device
+Subsystem: Red Hat, Inc. Device 0001
+Kernel driver in use: vfio-pci
+----
++
+In the output of the command, VFIO ethernet controllers use the `vfio-pci` kernel driver.
diff --git a/modules/installation-osp-dpdk-exposing-host-interface.adoc b/modules/installation-osp-dpdk-exposing-host-interface.adoc
@@ -0,0 +1,29 @@
+:_content-type: PROCEDURE
+[id="installation-osp-dpdk-exposing-host-interface_{context}"]
+= Exposing the host-device interface to the pod
+
+You can use the Container Network Interface (CNI) plug-in to expose an interface that is on the host to the pod. The plug-in moves the interface from the namespace of the host network to the namespace of the pod. The pod then has direct control of the interface.
+
+.Procedure
+
+* Create an additional network attachment with the host-device CNI plug-in by using the following object as an example:
++
+[source,yaml]
+----
+    apiVersion: k8s.cni.cncf.io/v1
+    kind: NetworkAttachmentDefinition
+    metadata:
+     name: vhostuser1
+     namespace: default
+    spec:
+     config: '{ "cniVersion": "0.3.1", "name": "hostonly", "type": "host-device", "pciBusId": "0000:00:04.0", "ipam": { } }'
+----
+
+.Verification
+
+* From a command line, run the following command to see if networks are created in the namespace:
++
+[source,terminal]
+----
+$ oc -n <your_cnf_namespace> get net-attach-def
+----
