Merge pull request #28972 from SNiemann15/multipathing_ibmz

add multipathing to installation procedure

Author: Bob Furu

modules/installation-complete-user-infra.adoc

@@ -12,10 +12,20 @@ endif::[]
 ifeval::["{context}" == "installing-restricted-networks-bare-metal"]
 :restricted:
 endif::[]
+ifeval::["{context}" == "installing-ibm-z"]
+:ibm-z:
+endif::[]
 ifeval::["{context}" == "installing-restricted-networks-ibm-z"]
+:ibm-z:
+:restricted:
+endif::[]
+ifeval::["{context}" == "installing-ibm-power"]
+:ibm-power:
+endif::[]
+ifeval::["{context}" == "installing-restricted-networks-ibm-power"]
+:ibm-power:
 :restricted:
 endif::[]
-
 [id="installation-complete-user-infra_{context}"]
 = Completing installation on user-provisioned infrastructure
 
@@ -130,6 +140,73 @@ command.
 +
 If the pod logs display, the Kubernetes API server can communicate with the
 cluster machines.
+ifdef::ibm-z,ibm-power[]
+. For an installation with FCP, additional steps are required to enable multipathing. 
+.. To enable multipathing on master nodes, apply the following machine config file: 
++
+[source,yaml]
+--
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: "master"
+  name: 99-master-kargs-mpath
+spec:
+  kernelArguments:
+    - 'rd.multipath=default'
+    - 'root=/dev/disk/by-label/dm-mpath-root'
+--
+
+.. To enable multipathing on worker nodes, apply the following machine config file: 
++
+[source,yaml]
+--
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: "worker"
+  name: 99-master-kargs-mpath
+spec:
+  kernelArguments:
+    - 'rd.multipath=default'
+    - 'root=/dev/disk/by-label/dm-mpath-root'
+--
+endif::ibm-z,ibm-power[]
+ifdef::ibm-power[]
+.. To display a boot list and specify the possible boot devices if the system is booted in normal mode, enter the following command:
++
+[source,terminal]
+----
+$ bootlist -m normal -o
+sda
+----
+.. To update the boot list for normal mode and add alternate device names, enter the following command:
++
+[source,terminal]
+----
+$ bootlist -m normal -o /dev/sdc /dev/sdd /dev/sde
+sdc
+sdd
+sde
+----
++
+If the original boot disk path is down, the node reboots from the alternate device registered in the normal boot device list. 
+endif::ibm-power[]
+ifdef::ibm-z,ibm-power[]
+.. All the worker nodes are restarted. To monitor the process, enter the following command: 
++
+[source,terminal]
+----
+$ oc get nodes -w
+----
++
+[NOTE]
+====
+If you have additional machine types such as infrastructure nodes, repeat the process for these types.   
+====
+endif::ibm-z,ibm-power[]
 
 ifdef::restricted[]
 . Register your cluster on the link:https://cloud.redhat.com/openshift/register[Cluster registration] page.
@@ -144,6 +221,17 @@ endif::[]
 ifdef::openshift-origin[]
 :!restricted:
 endif::[]
+ifeval::["{context}" == "installing-ibm-z"]
+:!ibm-z:
+endif::[]
 ifeval::["{context}" == "installing-restricted-networks-ibm-z"]
+:!ibm-z:
 :!restricted:
 endif::[]
+ifeval::["{context}" == "installing-ibm-power"]
+:!ibm-power:
+endif::[]
+ifeval::["{context}" == "installing-restricted-networks-ibm-power"]
+:!ibm-power:
+:restricted:
+endif::[]

modules/installation-ibm-z-kvm-user-infra-machines-iso.adoc

@@ -16,7 +16,7 @@ Before you install a cluster on IBM Z infrastructure that you provision, you mus
 
 . Install the KVM/QEMU hypervisor on the {op-system-base} KVM host.
 
-. Obtain the {op-system-base} QEMU copy-on-write (QCOW2) disk image file from the link:https://access.redhat.com/downloads/content/290[Product Downloads] page on the Red Hat customer portal or the link:https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/4.2/latest/[{op-system} image mirror] page.
+. Obtain the {op-system-base} QEMU copy-on-write (QCOW2) disk image file from the link:https://access.redhat.com/downloads/content/290[Product Downloads] page on the Red Hat Customer Portal or from the link:https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/latest/[{op-system} image mirror] page.
 +
 [IMPORTANT]
 ====

modules/installation-ibm-z-user-infra-machines-iso.adoc

@@ -9,13 +9,13 @@ Before you install a cluster on IBM Z infrastructure that you provision, you mus
 
 .Prerequisites
 
-* An FTP server running on your provisioning machine that is accessible to the machines you create.
+* An HTTP or HTTPS server running on your provisioning machine that is accessible to the machines you create.
 
 .Procedure
 
 . Log in to Linux on your provisioning machine.
 
-. Obtain the {op-system-first} kernel, initramfs, and rootfs files from the link:https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/4.2/latest/[{op-system} image mirror].
+. Obtain the {op-system-first} kernel, initramfs, and rootfs files from the link:https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/latest/[{op-system} image mirror].
 +
 [IMPORTANT]
 ====
@@ -36,23 +36,25 @@ The rootfs image is the same for FCP and DASD.
 ====
 +
 . Create parameter files. The following parameters are specific for a particular virtual machine:
-** For `coreos.inst.install_dev=`, specify `dasda` for a DASD installation, or `sda` for FCP. Note that FCP requires `zfcp.allow_lun_scan=0`.
-** For `rd.dasd=`, specifys the DASD where {op-system} is to be installed.
-** `rd.zfcp=<adapter>,<wwpn>,<lun>` specifies the FCP disk to install {op-system} on.
 ** For `ip=`, specify the following seven entries:
 ... The IP address for the machine.
 ... An empty string.
 ... The gateway.
 ... The netmask.
 ... The machine host and domain name in the form `hostname.domainname`. Omit this value to let {op-system} decide.
 ... The network interface name. Omit this value to let {op-system} decide.
-... If you use static IP addresses, an empty string.
+... If you use static IP addresses, specify `none`.
 ** For `coreos.inst.ignition_url=`, specify the Ignition file for the machine role. Use `bootstrap.ign`, `master.ign`, or `worker.ign`. Only HTTP and HTTPS protocols are supported.
 ** For `coreos.live.rootfs_url=`, specify the matching rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported.
-** All other parameters can stay as they are.
+
+** For installations on DASD-type disks, complete the following tasks:
+... For `coreos.inst.install_dev=`, specify `dasda`. 
+... Use `rd.dasd=` to specify the DASD where {op-system} is to be installed.
+... Leave all other parameters unchanged. 
 +
 Example parameter file, `bootstrap-0.parm`, for the bootstrap machine:
 +
+[source,terminal]
 ----
 rd.neednet=1 console=ttysclp0 coreos.inst.install_dev=dasda coreos.live.rootfs_url=http://
 cl1.provide.example.com:8080/assets/rhcos-live-rootfs.s390x.img
@@ -62,6 +64,40 @@ rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 zfcp.allow_lun_scan=0
 !condev rd.dasd=0.0.3490
 ----
 
+
+** For installations on FCP-type disks, complete the following tasks:
+... Use `rd.zfcp=<adapter>,<wwpn>,<lun>` to specify the FCP disk where {op-system} is to be installed. For multipathing repeat this step for each additional path. 
+... For multipathing, set the following parameter: `rd.multipath=default`.
+... For multipathing, set the install device as: `coreos.inst.install_dev=/dev/mapper/mpatha`.
+... For single-path installation, set the install device as: `coreos.inst.install_dev=sda`.
++
+[NOTE]
+====
+If additional LUNs are configured with NPIV, FCP requires `zfcp.allow_lun_scan=0`.  
+====
+... Leave all other parameters unchanged.
++
+[NOTE]
+====
+Additional post-installation steps are required to fully enable multipathing. See “Completing installation on user-provisioned infrastructure” for more information.  
+====
++
+The following is an example parameter file `bootstrap-0.parm` for the bootstrap machine with multipathing:
++
+[source,terminal]
+----
+rd.neednet=1 rd.multipath=default console=ttysclp0 coreos.inst.install_dev=/dev/mapper/mpatha 
+coreos.live.rootfs_url=http://cl1.provide.example.com:8080/assets/rhcos-live-rootfs.s390x.img
+coreos.inst.ignition_url=http://cl1.provide.example.com:8080/ignition/bootstrap.ign
+ip=172.18.78.2::172.18.78.1:255.255.255.0:::none nameserver=172.18.78.1
+rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 zfcp.allow_lun_scan=0 cio_ignore=all,
+!condev 
+rd.zfcp=0.0.1987,0x50050763070bc5e3,0x4008400B00000000
+rd.zfcp=0.0.19C7,0x50050763070bc5e3,0x4008400B00000000
+rd.zfcp=0.0.1987,0x50050763071bc5e3,0x4008400B00000000
+rd.zfcp=0.0.19C7,0x50050763071bc5e3,0x4008400B00000000
+----
+
 . Transfer the initramfs, kernel, parameter files, and {op-system} images to z/VM, for example with FTP. For details about how to transfer the files with FTP and boot from the virtual reader, see link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/installation_guide/sect-installing-zvm-s390[Installing under Z/VM].
 . Punch the files to the virtual reader of the z/VM guest virtual machine that is to become your bootstrap node.
 +

modules/installation-infrastructure-user-infra.adoc

@@ -25,7 +25,7 @@ page before you create the supporting infrastructure for your cluster.
 
 ifdef::ibm-z[]
 . Set up static IP addresses.
-. Set up an FTP server.
+. Set up an HTTP or HTTPS server to provide Ignition files to the cluster nodes.
 endif::ibm-z[]
 ifndef::ibm-z[]
 . Configure DHCP or set static IP addresses on each node.

modules/installation-network-user-infra.adoc

@@ -44,14 +44,14 @@ All the {op-system-first} machines require network in `initramfs` during boot
 to fetch Ignition config from the machine config server.
 
 ifdef::ibm-z[]
-During the initial boot, the machines require an FTP server in order to
+During the initial boot, the machines require an HTTP or HTTPS server to
 establish a network connection to download their Ignition config files.
 
 Ensure that the machines have persistent IP addresses and host names.
 endif::ibm-z[]
 ifndef::ibm-z[]
 During the initial boot, the machines require either a DHCP server
-or that static IP addresses be set on each host in the cluster in order to
+or that static IP addresses be set on each host in the cluster to
 establish a network connection, which allows them to download their Ignition config files.
 
 It is recommended to use the DHCP server to manage the machines for the cluster