Merge pull request #57587 from kelbrown20/OSDOCS-5324-bare-metal-on-multi-arch

OSDOCS-5324: Multi-arch on bare metal docs TP and cluster compatibility section

Author: kelbrown20

modules/machine-user-infra-machines-iso.adoc

@@ -2,19 +2,46 @@
 //
 // * machine_management/user_infra/adding-bare-metal-compute-user-infra.adoc
 // * post_installation_configuration/node-tasks.adoc
+ifeval::["{context}" == "multi-architecture-configuration"]
+:multi:
+endif::[]
 
 :_content-type: PROCEDURE
 [id="machine-user-infra-machines-iso_{context}"]
-= Creating more {op-system} machines using an ISO image
+= Creating {op-system} machines using an ISO image
 
 You can create more {op-system-first} compute machines for your bare metal cluster by using an ISO image to create the machines.
 
 .Prerequisites
 
 * Obtain the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation.
+* You must have the OpenShift CLI (`oc`)  installed. 
 
 .Procedure
 
+. Extract the Ignition config file from the cluster by running the following command:
++
+[source,terminal]
+----
+$ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign
+----
+
+. Upload the `worker.ign` Ignition config file you exported from your cluster to your HTTP server. Note the URLs of these files. 
+
+. You can validate that the ignition files are available on the URLs. The following example gets the Ignition config files for the compute node: 
++
+[source,terminal]
+----
+$ curl -k http://<HTTP_server>/worker.ign 
+----
+
+. You can access the ISO image for booting your new machine by running to following command:
++
+[source,terminal]
+----
+RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.<architecture>.artifacts.metal.formats.iso.disk.location')
+----
+
 . Use the ISO file to install {op-system} on more compute machines. Use the same method that you used when you created machines before you installed the cluster:
 ** Burn the ISO image to a disk and boot it directly.
 ** Use ISO redirection with a LOM interface.
@@ -55,3 +82,7 @@ Ensure that the installation is successful on each node before commencing with t
 ====
 
 . Continue to create more compute machines for your cluster.
+
+ifeval::["{context}" == "multi-architecture-configuration"]
+:!multi:
+endif::[]

modules/machine-user-infra-machines-pxe.adoc

@@ -2,10 +2,11 @@
 //
 // * machine_management/user_infra/adding-bare-metal-compute-user-infra.adoc
 // * post_installation_configuration/node-tasks.adoc
+// * post_installation_configuration/multi-architecture-configuration.adoc
 
 :_content-type: PROCEDURE
 [id="machine-user-infra-machines-pxe_{context}"]
-= Creating more {op-system} machines by PXE or iPXE booting
+= Creating {op-system} machines by PXE or iPXE booting
 
 You can create more {op-system-first} compute machines for your bare metal cluster by using PXE or iPXE booting.
 
@@ -33,25 +34,51 @@ LABEL pxeboot
 <1> Specify the location of the live `kernel` file that you uploaded to your HTTP server.
 <2> Specify locations of the {op-system} files that you uploaded to your HTTP server. The `initrd` parameter value is the location of the live `initramfs` file, the `coreos.inst.ignition_url` parameter value is the location of the worker Ignition config file, and the `coreos.live.rootfs_url` parameter value is the location of the live `rootfs` file. The `coreos.inst.ignition_url` and `coreos.live.rootfs_url` parameters only support HTTP and HTTPS.
 +
-+
 [NOTE]
 ====
-This configuration does not enable serial console access on machines with a graphical console.  To configure a different console, add one or more `console=` arguments to the `APPEND` line.  For example, add `console=tty0 console=ttyS0` to set the first PC serial port as the primary console and the graphical console as a secondary console.  For more information, see link:https://access.redhat.com/articles/7212[How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?].
+This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more `console=` arguments to the `APPEND` line. For example, add `console=tty0 console=ttyS0` to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see link:https://access.redhat.com/articles/7212[How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?].
 ====
 
-** For iPXE:
+** For iPXE (`x86_64` + `aarch64`):
 +
 ----
-kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img <1>
-initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img <2>
+kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign <1> <2>
+initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img <3>
+boot
 ----
-<1> Specify locations of the {op-system} files that you uploaded to your HTTP server. The `kernel` parameter value is the location of the `kernel` file, the `initrd=main` argument is needed for booting on UEFI systems, the `coreos.inst.ignition_url` parameter value is the location of the worker Ignition config file, and the `coreos.live.rootfs_url` parameter value is the location of the live `rootfs` file. The `coreos.inst.ignition_url` and `coreos.live.rootfs_url` parameters only support HTTP and HTTPS.
-<2> Specify the location of the `initramfs` file that you uploaded to your HTTP server.
+<1> Specify the locations of the {op-system} files that you uploaded to your
+HTTP server. The `kernel` parameter value is the location of the `kernel` file,
+the `initrd=main` argument is needed for booting on UEFI systems,
+the `coreos.live.rootfs_url` parameter value is the location of the `rootfs` file,
+and the `coreos.inst.ignition_url` parameter value is the
+location of the worker Ignition config file.
+<2> If you use multiple NICs, specify a single interface in the `ip` option.
+For example, to use DHCP on a NIC that is named `eno1`, set `ip=eno1:dhcp`.
+<3> Specify the location of the `initramfs` file that you uploaded to your HTTP server.
 +
+[NOTE]
+====
+This configuration does not enable serial console access on machines with a graphical console To configure a different console, add one or more `console=` arguments to the `kernel` line. For example, add `console=tty0 console=ttyS0` to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see link:https://access.redhat.com/articles/7212[How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?] and "Enabling the serial console for PXE and ISO installation" in the "Advanced {op-system} installation configuration" section.
+====
 +
 [NOTE]
 ====
-This configuration does not enable serial console access on machines with a graphical console.  To configure a different console, add one or more `console=` arguments to the `kernel` line.  For example, add `console=tty0 console=ttyS0` to set the first PC serial port as the primary console and the graphical console as a secondary console.  For more information, see link:https://access.redhat.com/articles/7212[How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?].
+To network boot the CoreOS `kernel` on `aarch64` architecture, you need to use a version of iPXE build with the `IMAGE_GZIP` option enabled. See link:https://ipxe.org/buildcfg/image_gzip[`IMAGE_GZIP` option in iPXE].
 ====
 
-. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.
+** For PXE (with UEFI and GRUB as second stage) on `aarch64`:
++
+----
+menuentry 'Install CoreOS' {
+    linux rhcos-<version>-live-kernel-<architecture>  coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign <1> <2>
+    initrd rhcos-<version>-live-initramfs.<architecture>.img <3>
+}
+----
+<1> Specify the locations of the {op-system} files that you uploaded to your
+HTTP/TFTP server. The `kernel` parameter value is the location of the `kernel` file on your TFTP server.
+The `coreos.live.rootfs_url` parameter value is the location of the `rootfs` file, and the `coreos.inst.ignition_url` parameter value is the location of the worker Ignition config file on your HTTP Server.
+<2> If you use multiple NICs, specify a single interface in the `ip` option.
+For example, to use DHCP on a NIC that is named `eno1`, set `ip=eno1:dhcp`.
+<3> Specify the location of the `initramfs` file that you uploaded to your TFTP server.
+
+. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

modules/multi-architecture-verifying-cluster-compatibility.adoc

@@ -0,0 +1,41 @@
+// Module included in the following assemblies:
+
+// * post_installation_configuration/multi-architecture-configuration.adoc
+
+:_content-type: PROCEDURE
+[id="multi-architecture-verifying-cluster-compatibility_{context}"]
+
+= Verifying cluster compatibility 
+
+Before you can start adding compute nodes of different architectures to your cluster, you must verify that your cluster is multi-architecture compatible. 
+
+.Prerequisites
+
+* You installed the OpenShift CLI (`oc`)
+
+.Procedure 
+
+* You can check that your cluster uses the architecture payload by running the following command:
++
+[source,terminal]
+----
+$ oc adm release info -o json | jq .metadata.metadata
+----
+
+.Verification 
+
+. If you see the following output, then your cluster is using the multi-architecture payload: 
++
+[source,terminal]
+----
+$ "release.openshift.io/architecture": "multi"
+----
+You can then begin adding multi-arch compute nodes to your cluster.
+
+. If you see the following output, then your cluster is not using the multi-architecture payload:
++
+[source,terminal]
+----
+$ null
+----
+To migrate your cluster to one that supports multi-architecture compute machines, follow the procedure in "Migrating to a cluster with multi-architecture compute machines". 

post_installation_configuration/multi-architecture-configuration.adoc

@@ -20,6 +20,12 @@ The Cluster Samples Operator is not supported on clusters with multi-architectur
 
 For information on migrating your single-architecture cluster to a cluster that supports multi-architecture compute machines, see xref:../updating/migrating-to-multi-payload.adoc#migrating-to-multi-payload[Migrating to a cluster with multi-architecture compute machines].
 
+include::modules/multi-architecture-verifying-cluster-compatibility.adoc[leveloffset=+1]
+
+[role="_additional-resources"] 
+.Additional resources
+* xref:../updating/migrating-to-multi-payload.adoc#migrating-to-multi-payload[Migrating to a cluster with multi-architecture compute machines].
+
 == Creating a cluster with multi-architecture compute machine on Azure 
 
 To deploy an Azure cluster with multi-architecture compute machines, you must first create a single-architecture Azure installer-provisioned cluster that uses the multi-architecture installer binary. For more information on Azure installations, see xref:../installing/installing_azure/installing-azure-customizations.adoc[Installing a cluster on Azure with customizations]. You can then add an `arm64` compute machine set to your cluster to create a cluster with multi-architecture compute machines. 
@@ -44,4 +50,21 @@ include::modules/multi-architecture-modify-machine-set-aws.adoc[leveloffset=+2]
 .Additional resources
 * xref:../installing/installing_aws/installing-aws-customizations.adoc#installation-aws-arm-tested-machine-types_installing-aws-customizations[Tested instance types for AWS 64-bit ARM]
 
+== Creating a cluster with multi-architecture compute machine on bare metal (Technology Preview)
+
+To create a cluster with multi-architecture compute machines on bare metal, you must have an existing single-architecture bare metal cluster. For more information on bare metal installations, see xref:../installing/installing_bare_metal/installing-bare-metal.adoc#installing-bare-metal[Installing a user provisioned cluster on bare metal]. You can then add 64-bit ARM compute machines to your {product-title} cluster on bare metal. 
+
+Before you can add 64-bit ARM nodes to your bare metal cluster, you must upgrade your cluster to one that uses the multi-architecture payload. For more information on migrating to the multi-architecture payload, see xref:../updating/migrating-to-multi-payload.adoc#migrating-to-multi-payload[Migrating to a cluster with multi-architecture compute machines].
+
+The following procedures explain how to create a {op-system} compute machine using an ISO image or network PXE booting. This will allow you to add ARM64 nodes to your bare metal cluster and deploy a cluster with multi-architecture compute machines. 
+
+:Featurename: Clusters with multi-architecture compute machines on bare metal user-provisioned installations 
+include::snippets/technology-preview.adoc[leveloffset=+2]
+
+include::modules/machine-user-infra-machines-iso.adoc[leveloffset=+2]
+
+include::modules/machine-user-infra-machines-pxe.adoc[leveloffset=+2]
+
+include::modules/installation-approve-csrs.adoc[leveloffset=+2]
+
 include::modules/multi-architecture-import-imagestreams.adoc[leveloffset=+1]