Merge pull request #88285 from dfitzmau/OCPBUGS-48847-16

[enterprise-4.16] OCPBUGS-48847: Documented IPoIB support for nmstate

Author: dfitzmau

modules/creating-manifest-file-customized-br-ex-bridge.adoc

@@ -18,7 +18,7 @@ endif::[]
 
 :_mod-docs-content-type: PROCEDURE
 [id="creating-manifest-file-customized-br-ex-bridge_{context}"]
-== Optional: Creating a manifest object that includes a customized `br-ex` bridge
+= Creating a manifest object that includes a customized `br-ex` bridge
 
 ifndef::postinstall-bare-metal-ipi,postinstall-bare-metal-upi[]
 As an alternative to using the `configure-ovs.sh` shell script to set a `br-ex` bridge on a bare-metal platform, you can create a `MachineConfig` object that includes an NMState configuration file. The NMState configuration file creates a customized `br-ex` bridge network configuration on each node in your cluster.

modules/virt-creating-infiniband-interface-on-nodes.adoc

@@ -0,0 +1,64 @@
+// Module included in the following assemblies:
+//
+// * installing/installing_bare_metal/ipi/ipi-install-post-installation-configuration.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="virt-creating-infiniband-interface-on-nodes_{context}"]
+= Creating an IP over InfiniBand interface on nodes
+
+On the {product-title} web console, you can install a Red{nbsp}Hat certified third-party Operator, such as the NVIDIA Network Operator, that supports InfiniBand (IPoIB) mode. Typically, you would use the third-party Operator with other vendor infrastructure to manage resources in an {product-title} cluster. To create an IPoIB interface on nodes in your cluster, you must define an InfiniBand (IPoIB) interface in a `NodeNetworkConfigurationPolicy` (NNCP) manifest file. 
+
+[IMPORTANT]
+====
+The {product-title} documentation describes defining only the IPoIB interface configuration in a `NodeNetworkConfigurationPolicy` (NNCP) manifest file. You must refer to the NVIDIA and other third-party vendor documentation for the majority of the configuring steps. Red{nbsp}Hat support does not extend to anything external to the NNCP configuration. 
+
+For more information about the NVIDIA Operator, see link:https://docs.nvidia.com/networking/display/kubernetes2410/getting+started+with+red+hat+openshift[Getting Started with Red{nbsp}Hat OpenShift] (NVIDIA Docs Hub).
+====
+
+.Prerequisites
+
+* You installed a Red{nbsp}Hat certified third-party Operator that supports an IPoIB interface.
+
+
+.Procedure
+
+. Create or edit a `NodeNetworkConfigurationPolicy` (NNCP) manifest file, and then specify an IPoIB interface in the file.
++
+
+[source,yaml]
+----
+apiVersion: nmstate.io/v1
+kind: NodeNetworkConfigurationPolicy
+metadata:
+  name: worker-0-ipoib
+spec:
+# ...
+    interfaces:
+    - description: ""
+      infiniband:
+        mode: datagram <1>
+        pkey: "0xffff" <2>
+      ipv4:
+        address:
+        - ip: 100.125.3.4
+          prefix-length: 16
+        dhcp: false
+        enabled: true
+      ipv6:
+        enabled: false
+      name: ibp27s0
+      state: up
+      type: infiniband <3>
+# ...
+----
+<1> `datagram` is the default mode for an IPoIB interface, and this mode improves optimizes performance and latency. `connected` mode is a supported mode but consider only using this mode when you need to adjust the maximum transmission unit (MTU) value to improve node connectivity with surrounding network devices.  
+<2> Supports a string or an integer value. The parameter defines the protection key, or _P-key_, for the interface for the purposes of authentication and encrypted communications with a third-party vendor, such as NVIDIA. Values `None` and `0xffff` indicate the protection key for the base interface in an InfiniBand system.
+<3> Sets the type of interface to `infiniband `.
+
+. Apply the NNCP configuration to each node in your cluster by running the following command. The Kubernetes NMState Operator can then create an IPoIB interface on each node. 
++
+[source,yaml]
+----
+$ oc apply -f <nncp_file_name> <1>
+----
+<1> Replace `<nncp_file_name>` with the name of your NNCP file.

post_installation_configuration/post-install-bare-metal-configuration.adoc

@@ -8,6 +8,9 @@ toc::[]
 
 When deploying {product-title} on bare-metal hosts, there are times when you need to make changes to the host either before or after provisioning. This can include inspecting the host's hardware, firmware, and firmware details. It can also include formatting disks or changing modifiable firmware settings.
 
+// Creating an InfiniBand interface on nodes
+include::modules/virt-creating-infiniband-interface-on-nodes.adoc[leveloffset=+1]
+
 // Services for a user-managed load balancer
 include::modules/nw-osp-services-external-load-balancer.adoc[leveloffset=+1]