Skip to content

Commit f4d81e4

Browse files
kalexand-rhkalexand-rh
authored
Merge pull request #46224 from kelbrown20/OSDOCS-3699-heterogeneous-cluster-support
OSDOCS-3601: Heterogeneous cluster support
2 parents 0cad7d9 + 119b831 commit f4d81e4

File tree

5 files changed

+293
-0
lines changed

5 files changed

+293
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -492,6 +492,8 @@ Topics:
492492
File: configuring-private-cluster
493493
- Name: Bare metal configuration
494494
File: bare-metal-configuration
495+
- Name: Configuring a heterogeneous cluster
496+
File: deploy-heterogeneous-configuration
495497
- Name: Machine configuration tasks
496498
File: machine-configuration-tasks
497499
- Name: Cluster tasks

‎modules/mixed-arch-creating-arm64-bootimage.adoc

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
//Module included in the following assemblies
2+
//
3+
//post_installation_configuration/cluster-tasks.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="mixed-arch-creating-arm64-bootimage_{context}"]
7+
8+
= Creating an `arm64` boot image using the Azure image gallery
9+
10+
To configure your heterogeneous cluster, you must create an `arm64` boot image and add it to your Azure machine set. The following procedure describes how to manually generate an `arm64` boot image.
11+
12+
.Prerequisites
13+
14+
* You installed the Azure CLI (`az`).
15+
* You created a single-architecture Azure installer-provisioned cluster with the heterogeneous installer binary.
16+
17+
.Procedure
18+
. Log in to your Azure account:
19+
+
20+
[source,terminal]
21+
----
22+
$ az login
23+
----
24+
. Create a storage account and upload the `arm64` virtual hard disk (VHD) to your storage account. The {product-title} installation program creates a resource group, however, the boot image can also be uploaded to a custom named resource group:
25+
+
26+
[source,terminal]
27+
----
28+
$ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS <1>
29+
----
30+
+
31+
<1> The `westus` object is an example region.
32+
+
33+
. Create a storage container using the storage account you generated:
34+
+
35+
[source,terminal]
36+
+
37+
----
38+
$ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}
39+
----
40+
. You must use the {product-title} installation program JSON file to extract the URL and `arch64` VHD name:
41+
.. Extract the `URL` field and set it to `RHCOS_VHD_ORIGIN_URL` as the file name by running the following command:
42+
+
43+
[source,terminal]
44+
----
45+
$ RHCOS_VHD_ORIGIN_URL=$(./openshift-install coreos print-stream-json | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".url')
46+
----
47+
.. Extract the `aarch64` VHD name and set it to `BLOB_NAME` as the file name by running the following command:
48+
+
49+
[source,terminal]
50+
----
51+
$ BLOB_NAME=rhcos-$(./openshift-install coreos print-stream-json | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".release')-azure.aarch64.vhd
52+
----
53+
. Generate a shared access signature (SAS) token. Use this token to upload the {op-system} VHD to your storage container with the following commands:
54+
+
55+
[source,terminal]
56+
----
57+
$ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`
58+
----
59+
+
60+
[source,terminal]
61+
----
62+
$ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`
63+
----
64+
. Copy the {op-system} VHD into the storage container:
65+
+
66+
[source, terminal]
67+
----
68+
$ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
69+
--source-uri "${RHCOS_VHD_ORIGIN_URL}" \
70+
--destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}
71+
----
72+
+
73+
You can check the status of the copying process with the following command:
74+
+
75+
[source,terminal]
76+
----
77+
$ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy
78+
----
79+
+
80+
.Example output
81+
[source,terminal]
82+
----
83+
{
84+
"completionTime": null,
85+
"destinationSnapshot": null,
86+
"id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
87+
"incrementalCopy": null,
88+
"progress": "17179869696/17179869696",
89+
"source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
90+
"status": "success", <1>
91+
"statusDescription": null
92+
}
93+
----
94+
+
95+
<1> If the status parameter displays the `success` object, the copying process is complete.
96+
97+
. Create an image gallery using the following command:
98+
+
99+
[source,terminal]
100+
----
101+
$ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}
102+
----
103+
Use the image gallery to create an image definition. In the following example command, `rhcos-arm64` is the name of the image definition.
104+
+
105+
[source,terminal]
106+
----
107+
$ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --publisher RedHat --offer arm --sku arm64 --os-type linux --architecture Arm64 --hyper-v-generation V2
108+
----
109+
. To get the URL of the VHD and set it to `RHCOS_VHD_URL` as the file name, run the following command:
110+
+
111+
[source,terminal]
112+
----
113+
$ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)
114+
----
115+
. Use the `RHCOS_VHD_URL` file, your storage account, resource group, and image gallery to create an image version. In the following example, `1.0.0` is the image version.
116+
+
117+
[source,terminal]
118+
----
119+
$ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}
120+
----
121+
. Your `arm64` boot image is now generated. You can access the ID of your image with the following command:
122+
+
123+
[source,terminal]
124+
----
125+
$ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-arm64 -e 1.0.0
126+
----
127+
The following example image ID is used in the `recourseID` parameter of the machine set:
128+
+
129+
.Example `resourceID`
130+
[source,terminal]
131+
----
132+
/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0
133+
----

‎modules/mixed-arch-modify-machine-set.adoc

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
//Module included in the following assembly
2+
//
3+
//post_installation_configuration/cluster-tasks.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="mixed-arch-modify-machine-set_{context}"]
7+
8+
= Adding a machine set to your cluster using the `arm64` boot image
9+
10+
To add `arm64` worker nodes to your heterogeneous cluster, you must create an Azure machine set that uses the `arm64` boot image. To create your own custom machine set on Azure, see "Creating a machine set on Azure".
11+
12+
.Prerequisites
13+
14+
* You installed the OpenShift CLI (`oc`).
15+
16+
.Procedure
17+
* Create a machine set and modify the `resourceID` and `vmSize` parameters with the following command. This machine set will control the `arm64` worker nodes in your cluster:
18+
+
19+
[source,terminal]
20+
----
21+
$ oc create -f arm64-machine-set-0.yaml
22+
----
23+
.Sample YAML machine set with `arm64` boot image
24+
+
25+
[source,yaml]
26+
----
27+
apiVersion: machine.openshift.io/v1beta1
28+
kind: MachineSet
29+
metadata:
30+
labels:
31+
machine.openshift.io/cluster-api-cluster: <infrastructure_id>
32+
machine.openshift.io/cluster-api-machine-role: worker
33+
machine.openshift.io/cluster-api-machine-type: worker
34+
name: <infrastructure_id>-arm64-machine-set-0
35+
namespace: openshift-machine-api
36+
spec:
37+
replicas: 2
38+
selector:
39+
matchLabels:
40+
machine.openshift.io/cluster-api-cluster: <infrastructure_id>
41+
machine.openshift.io/cluster-api-machineset: <infrastructure_id>-arm64-machine-set-0
42+
template:
43+
metadata:
44+
labels:
45+
machine.openshift.io/cluster-api-cluster: <infrastructure_id>
46+
machine.openshift.io/cluster-api-machine-role: worker
47+
machine.openshift.io/cluster-api-machine-type: worker
48+
machine.openshift.io/cluster-api-machineset: <infrastructure_id>-arm64-machine-set-0
49+
spec:
50+
lifecycleHooks: {}
51+
metadata: {}
52+
providerSpec:
53+
value:
54+
acceleratedNetworking: true
55+
apiVersion: machine.openshift.io/v1beta1
56+
credentialsSecret:
57+
name: azure-cloud-credentials
58+
namespace: openshift-machine-api
59+
image:
60+
offer: ""
61+
publisher: ""
62+
resourceID: /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0 <1>
63+
sku: ""
64+
version: ""
65+
kind: AzureMachineProviderSpec
66+
location: <region>
67+
managedIdentity: <infrastructure_id>-identity
68+
networkResourceGroup: <infrastructure_id>-rg
69+
osDisk:
70+
diskSettings: {}
71+
diskSizeGB: 128
72+
managedDisk:
73+
storageAccountType: Premium_LRS
74+
osType: Linux
75+
publicIP: false
76+
publicLoadBalancer: <infrastructure_id>
77+
resourceGroup: <infrastructure_id>-rg
78+
subnet: <infrastructure_id>-worker-subnet
79+
userDataSecret:
80+
name: worker-user-data
81+
vmSize: Standard_D4ps_v5 <2>
82+
vnet: <infrastructure_id>-vnet
83+
zone: "<zone>"
84+
----
85+
<1> Set the `resourceID` parameter to the `arm64` boot image.
86+
<2> Set the `vmSize` parameter to the instance type used in your installation. Some example instance types are `Standard_D4ps_v5` or `D8ps`.
87+
88+
.Verification
89+
. Verify that the new ARM64 machines are running by entering the following command:
90+
+
91+
[source,terminal]
92+
----
93+
$ oc get machineset -n openshift-machine-api
94+
----
95+
+
96+
.Example output
97+
[source,terminal]
98+
----
99+
NAME DESIRED CURRENT READY AVAILABLE AGE
100+
<infrastructure_id>-arm64-machine-set-0 2 2 2 2 10m
101+
----
102+
. You can check that the nodes are ready and scheduable with the following command:
103+
+
104+
[source, terminal]
105+
----
106+
$ oc get nodes
107+
----

‎modules/mixed-arch-upgrade-mirrors.adoc

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
//Module included in the following assemblies
2+
//
3+
//post_installation_configuration/cluster-tasks.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="mixed-arch-upgrade-mirrors_{context}"]
7+
8+
= Upgrading your heterogeneous cluster
9+
10+
You must perform an explicit upgrade command to upgrade your existing cluster to a heterogeneous cluster.
11+
12+
.Prerequisites
13+
14+
* You installed the OpenShift CLI (`oc`).
15+
16+
.Procedure
17+
* To manually upgrade your cluster, use the following command:
18+
[source, terminal]
19+
----
20+
$ oc adm upgrade --allow-explicit-upgrade --to-image <image-pullspec> <1>
21+
----
22+
<1> You can access the `image-pullspec` object from the link:https://mirror.openshift.com/pub/openshift-v4/multi/clients/ocp/4.11.0-multi-rc.6/[mixed-arch mirrors page] in the `release.txt` file.

‎post_installation_configuration/deploy-heterogeneous-configuration.adoc

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
:_content-type: ASSEMBLY
2+
:context: deploy-heterogeneous-configuration
3+
[id="post-install-deploy-heterogeneous-configuration"]
4+
= Configuring a heterogeneous cluster
5+
include::_attributes/common-attributes.adoc[]
6+
7+
toc::[]
8+
9+
A heterogeneous cluster is a cluster that supports worker machines with different architectures. You can deploy a heterogeneous cluster by creating an Azure installer-provisioned cluster using the heterogeneous installer binary. For Azure installation, see xref:../installing/installing_azure/installing-azure-customizations.adoc[Installing on Azure with customizations].
10+
11+
[WARNING]
12+
====
13+
The heterogeneous clusters Technology Preview feature has limited usability with installing, upgrading, and running payloads.
14+
====
15+
16+
The following procedures explain how to generate an `arm64` boot image and create an Azure machine set with the `arm64` boot image. This will add `arm64` worker nodes to your heterogeneous cluster and deploy the desired amount of ARM64 virtual machines (VM). This section also shows how to upgrade your existing cluster to a heterogeneous cluster. Heterogeneous clusters are only available on Azure installer-provisioned infrastructures with `x86_64` control planes.
17+
18+
:FeatureName: Heterogeneous clusters for {product-title} on Azure installer-provisioned infrastructure installations
19+
include::snippets/technology-preview.adoc[leveloffset=+1]
20+
21+
include::modules/mixed-arch-creating-arm64-bootimage.adoc[leveloffset=+1]
22+
23+
include::modules/mixed-arch-modify-machine-set.adoc[leveloffset=+1]
24+
25+
[role="_additional-resources"]
26+
.Additional resources
27+
* xref:../machine_management/creating_machinesets/creating-machineset-azure.adoc[Creating a machine set on Azure]
28+
29+
include::modules/mixed-arch-upgrade-mirrors.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)