Add MTU configuration change for installed clusters

- https://issues.redhat.com/browse/OSDOCS-1810

Author: jab-rh

_topic_maps/_topic_map.yml

@@ -949,6 +949,8 @@ Topics:
   Distros: openshift-enterprise,openshift-origin
 - Name: Verifying connectivity to an endpoint
   File: verifying-connectivity-endpoint
+- Name: Changing the cluster network MTU
+  File: changing-cluster-network-mtu
 - Name: Configuring the node port service range
   File: configuring-node-port-service-range
 - Name: Configuring IP failover

modules/nw-cluster-mtu-change-about.adoc

@@ -0,0 +1,81 @@
+// Module included in the following assemblies:
+//
+// * networking/changing-cluster-network-mtu.adoc
+
+:_content-type: CONCEPT
+[id="nw-cluster-mtu-change-about_{context}"]
+= About the cluster MTU
+
+During installation the maximum transmission unit (MTU) for the cluster network is detected automatically based on the MTU of the primary network interface of nodes in the cluster. You do not normally need to override the detected MTU.
+
+You might want to change the MTU of the cluster network for several reasons:
+
+* The MTU detected during cluster installation is not correct for your infrastructure
+* Your cluster infrastructure now requires a different MTU, such as from the addition of nodes that need a different MTU for optimal performance
+
+You can change the cluster MTU for only the OVN-Kubernetes and OpenShift SDN cluster network providers.
+
+// https://github.com/openshift/enhancements/blob/master/enhancements/network/allow-mtu-changes.md
+[id="service-interruption-considerations_{context}"]
+== Service interruption considerations
+
+When you initiate an MTU change on your cluster the following effects might impact service availability:
+
+* At least two rolling reboots are required to complete the migration to a new MTU. During this time, some nodes are not available as they restart.
+
+* Specific applications deployed to the cluster with shorter timeout intervals than the absolute TCP timeout interval might experience disruption during the MTU change.
+
+[id="mtu-value-selection_{context}"]
+== MTU value selection
+
+When planning your MTU migration there are two related but distinct MTU values to consider.
+
+* *Hardware MTU*: This MTU value is set based on the specifics of your network infrastructure.
+* *Cluster network MTU*: This MTU value is always less than your hardware MTU to account for the cluster network overlay overhead. The specific overhead is determined by your cluster network provider:
+** *OVN-Kubernetes*: `100` bytes
+** *OpenShift SDN*: `50` bytes
+
+If your cluster requires different MTU values for different nodes, you must subtract the overhead value for your cluster network provider from the lowest MTU value that is used by any node in your cluster. For example, if some nodes in your cluster have an MTU of `9001`, and some have an MTU of `1500`, you must set this value to `1400`.
+
+[id="how-the-migration-process-works_{context}"]
+== How the migration process works
+
+The following table summarizes the migration process by segmenting between the user-initiated steps in the process and the actions that the migration performs in response.
+
+.Live migration of the cluster MTU
+[cols="1a,1a",options="header"]
+|===
+
+|User-initiated steps|{product-title} activity
+
+|
+Set the following values in the Cluster Network Operator configuration:
+
+- `spec.migration.mtu.machine.to`
+- `spec.migration.mtu.network.from`
+- `spec.migration.mtu.network.to`
+
+|
+*Cluster Network Operator (CNO)*: Confirms that each field is set to a valid value.
+
+- The `mtu.machine.to` must be set to either the new hardware MTU or to the current hardware MTU if the MTU for the hardware is not changing. This value is transient and is used as part of the migration process. Separately, if you specify a hardware MTU that is different from your existing hardware MTU value, you must manually configure the MTU to persist by other means, such as with a machine config, DHCP setting, or a Linux kernel command line.
+- The `mtu.network.from` field must equal the `network.status.clusterNetworkMTU` field, which is the current MTU of the cluster network.
+- The `mtu.network.to` field must be set to the target cluster network MTU and must be lower than the hardware MTU to allow for the overlay overhead of the cluster network provider. For OVN-Kubernetes, the overhead is `100` bytes and for OpenShift SDN the overhead is `50` bytes.
+
+If the values provided are valid, the CNO writes out a new temporary configuration with the MTU for the cluster network set to the value of the `mtu.network.to` field.
+
+*Machine Config Operator (MCO)*: Performs a rolling reboot of each node in the cluster.
+
+|Reconfigure the MTU of the primary network interface for the nodes on the cluster. You can use a variety of methods to accomplish this, including:
+
+- Deploying a new NetworkManager connection profile with the MTU change
+- Changing the MTU through a DHCP server setting
+- Changing the MTU through boot parameters
+|N/A
+
+|Set the `mtu` value in the CNO configuration for the cluster network provider and set `spec.migration` to `null`.
+
+|
+*Machine Config Operator (MCO)*: Performs a rolling reboot of each node in the cluster with the new MTU configuration.
+
+|===