OCPBUGS-48847: Documented IPoIB support for nmstate

Author: dfitzmau

installing/installing_bare_metal_ipi/ipi-install-post-installation-configuration.adoc

@@ -8,10 +8,15 @@ toc::[]
 
 After successfully deploying an installer-provisioned cluster, consider the following postinstallation procedures.
 
+// Optional: Configuring NTP for disconnected clusters
 include::modules/ipi-install-configuring-ntp-for-disconnected-clusters.adoc[leveloffset=+1]
 
+// Enabling a provisioning network after installation
 include::modules/nw-enabling-a-provisioning-network-after-installation.adoc[leveloffset=+1]
 
+// Creating an InfiniBand interface on nodes
+include::modules/virt-creating-infiniband-interface-on-nodes.adoc[leveloffset=+1]
+
 // Configuring an external load balancer
 include::modules/nw-osp-services-external-load-balancer.adoc[leveloffset=+1]
 

modules/nw-enabling-a-provisioning-network-after-installation.adoc

@@ -4,7 +4,6 @@
 
 :_mod-docs-content-type: PROCEDURE
 [id="enabling-a-provisioning-network-after-installation_{context}"]
-
 = Enabling a provisioning network after installation
 
 The assisted installer and installer-provisioned installation for bare metal clusters provide the ability to deploy a cluster without a `provisioning` network. This capability is for scenarios such as proof-of-concept clusters or deploying exclusively with Redfish virtual media when each node's baseboard management controller is routable via the `baremetal` network.

modules/virt-creating-infiniband-interface-on-nodes.adoc

@@ -0,0 +1,64 @@
+// Module included in the following assemblies:
+//
+// * installing/installing_bare_metal/ipi/ipi-install-post-installation-configuration.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="virt-creating-infiniband-interface-on-nodes_{context}"]
+= Creating an IP over InfiniBand interface on nodes
+
+On the {product-title} web console, you can install a Red{nbsp}Hat certified third-party Operator, such as the NVIDIA Network Operator, that supports InfiniBand (IPoIB) mode. Typically, you would use the third-party Operator with other vendor infrastructure to manage resources in an {product-title} cluster. To create an IPoIB interface on nodes in your cluster, you must define an InfiniBand (IPoIB) interface in a `NodeNetworkConfigurationPolicy` (NNCP) manifest file. 
+
+[IMPORTANT]
+====
+The {product-title} documentation describes defining only the IPoIB interface configuration in a `NodeNetworkConfigurationPolicy` (NNCP) manifest file. You must refer to the NVIDIA and other third-party vendor documentation for the majority of the configuring steps. Red{nbsp}Hat support does not extend to anything external to the NNCP configuration. 
+
+For more information about the NVIDIA Operator, see link:https://docs.nvidia.com/networking/display/kubernetes2410/getting+started+with+red+hat+openshift[Getting Started with Red{nbsp}Hat OpenShift] (NVIDIA Docs Hub).
+====
+
+.Prerequisites
+
+* You installed a Red{nbsp}Hat certified third-party Operator that supports an IPoIB interface.
+
+
+.Procedure
+
+. Create or edit a `NodeNetworkConfigurationPolicy` (NNCP) manifest file, and then specify an IPoIB interface in the file.
++
+
+[source,yaml]
+----
+apiVersion: nmstate.io/v1
+kind: NodeNetworkConfigurationPolicy
+metadata:
+  name: worker-0-ipoib
+spec:
+# ...
+    interfaces:
+    - description: ""
+      infiniband:
+        mode: datagram <1>
+        pkey: "0xffff" <2>
+      ipv4:
+        address:
+        - ip: 100.125.3.4
+          prefix-length: 16
+        dhcp: false
+        enabled: true
+      ipv6:
+        enabled: false
+      name: ibp27s0
+      state: up
+      type: infiniband <3>
+# ...
+----
+<1> `datagram` is the default mode for an IPoIB interface, and this mode improves optimizes performance and latency. `connected` mode is a supported mode but consider only using this mode when you need to adjust the maximum transmission unit (MTU) value to improve node connectivity with surrounding network devices.  
+<2> Supports a string or an integer value. The parameter defines the protection key, or _P-key_, for the interface for the purposes of authentication and encrypted communications with a third-party vendor, such as NVIDIA. Values `None` and `0xffff` indicate the protection key for the base interface in an InfiniBand system.
+<3> Sets the type of interface to `infiniband `.
+
+. Apply the NNCP configuration to each node in your cluster by running the following command. The Kubernetes NMState Operator can then create an IPoIB interface on each node. 
++
+[source,yaml]
+----
+$ oc apply -f <nncp_file_name> <1>
+----
+<1> Replace `<nncp_file_name>` with the name of your NNCP file.