Skip to content

Commit b665105

Browse files
maxwelldbmaxwelldb
authored
Merge pull request #32876 from maxwelldb/osp-sr-iov-ipi-osdocs2063
[ShiftStack] SR-IOV compute machines for IPI installations
2 parents 6a519eb + a6b2028 commit b665105

File tree

6 files changed

+320
-1
lines changed

6 files changed

+320
-1
lines changed

‎_topic_map.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -287,6 +287,8 @@ Topics:
287287
File: installing-openstack-installer-custom
288288
- Name: Installing a cluster on OpenStack with Kuryr
289289
File: installing-openstack-installer-kuryr
290+
- Name: Installing a cluster that supports SR-IOV compute machines on OpenStack
291+
File: installing-openstack-installer-sr-iov
290292
- Name: Installing a cluster on OpenStack on your own infrastructure
291293
File: installing-openstack-user
292294
- Name: Installing a cluster on OpenStack with Kuryr on your own infrastructure

‎installing/installing_openstack/installing-openstack-installer-sr-iov.adoc‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
[id="installing-openstack-installer-sr-iov"]
2+
= Installing a cluster on OpenStack that supports SR-IOV-connected compute machines
3+
include::modules/common-attributes.adoc[]
4+
:context: installing-openstack-installer-sr-iov
5+
6+
toc::[]
7+
8+
In {product-title} version {product-version}, you can install a cluster on {rh-openstack-first} that can use compute machines with single-root I/O virtualization (SR-IOV) technology.
9+
10+
== Prerequisites
11+
12+
* Review details about the
13+
xref:../../architecture/architecture-installation.adoc#architecture-installation[{product-title} installation and update]
14+
processes.
15+
** Verify that {product-title} {product-version} is compatible with your {rh-openstack} version by using the "Supported platforms for OpenShift clusters" section. You can also compare platform support across different versions by viewing the link:https://access.redhat.com/articles/4679401[{product-title} on {rh-openstack} support matrix].
16+
// Statement seems somewhat obsolete, though not sure about assembly placement.
17+
* Verify that your network configuration does not rely on a provider network. Provider networks are not supported.
18+
19+
* Have a storage service installed in {rh-openstack}, like block storage (Cinder) or object storage (Swift). Object storage is the recommended storage technology for {product-title} registry cluster deployment. For more information, see xref:../../scalability_and_performance/optimizing-storage.adoc#optimizing-storage[Optimizing storage].
20+
21+
* Have metadata service enabled in {rh-openstack}
22+
23+
include::modules/installation-osp-default-deployment.adoc[leveloffset=+1]
24+
include::modules/installation-osp-control-compute-machines.adoc[leveloffset=+2]
25+
include::modules/installation-osp-bootstrap-machine.adoc[leveloffset=+2]
26+
include::modules/cluster-entitlements.adoc[leveloffset=+1]
27+
include::modules/installation-osp-enabling-swift.adoc[leveloffset=+1]
28+
include::modules/installation-osp-verifying-external-network.adoc[leveloffset=+1]
29+
include::modules/installation-osp-describing-cloud-parameters.adoc[leveloffset=+1]
30+
include::modules/installation-obtaining-installer.adoc[leveloffset=+1]
31+
include::modules/installation-initializing.adoc[leveloffset=+1]
32+
include::modules/installation-configure-proxy.adoc[leveloffset=+2]
33+
include::modules/installation-configuration-parameters.adoc[leveloffset=+1]
34+
include::modules/installation-osp-custom-subnet.adoc[leveloffset=+2]
35+
include::modules/installation-osp-deploying-bare-metal-machines.adoc[leveloffset=+2]
36+
include::modules/installation-osp-config-yaml.adoc[leveloffset=+2]
37+
include::modules/ssh-agent-using.adoc[leveloffset=+1]
38+
include::modules/installation-osp-accessing-api.adoc[leveloffset=+1]
39+
include::modules/installation-osp-accessing-api-floating.adoc[leveloffset=+2]
40+
include::modules/installation-osp-accessing-api-no-floating.adoc[leveloffset=+2]
41+
include::modules/installation-osp-configuring-sr-iov.adoc[leveloffset=+1]
42+
include::modules/installation-launching-installer.adoc[leveloffset=+1]
43+
include::modules/installation-osp-verifying-cluster-status.adoc[leveloffset=+1]
44+
include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1]
45+
The cluster is operational. Before you can add SR-IOV compute machines though, you must perform additional tasks.
46+
47+
include::modules/networking-osp-preparing-for-sr-iov.adoc[leveloffset=+1]
48+
include::modules/networking-osp-enabling-metadata.adoc[leveloffset=+2]
49+
include::modules/networking-osp-enabling-vfio-noiommu.adoc[leveloffset=+2]
50+
51+
//Tweak copied statement, but same gist as UPI. "Day 1 is done. Now, day 2."
52+
The cluster is installed and prepared for SR-IOV configuration. Complete the post-installation SR-IOV tasks that are listed in the "Next steps" section.
53+
54+
== Next steps
55+
56+
* To complete SR-IOV configuration for your cluster:
57+
** xref:../../scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.adoc#installing-the-performance-addon-operator_cnf-master[Install the Performance Addon Operator].
58+
** xref:../../scalability_and_performance/what-huge-pages-do-and-how-they-are-consumed-by-apps.adoc#what-huge-pages-do_huge-pages[Configure the Performance Addon Operator with huge pages support].
59+
** xref:../../networking/hardware_networks/installing-sriov-operator.adoc#installing-sr-iov-operator_installing-sriov-operator[Install the SR-IOV Operator].
60+
** xref:../../networking/hardware_networks/configuring-sriov-device.adoc#nw-sriov-networknodepolicy-object_configuring-sriov-device[Configure your SR-IOV network device].
61+
** xref:../../machine_management/creating_machinesets/creating-machineset-osp.adoc#machineset-yaml-osp-sr-iov_creating-machineset-osp[Add an SR-IOV compute machine set].
62+
* xref:../../post_installation_configuration/cluster-tasks.adoc#available_cluster_customizations[Customize your cluster].
63+
* If necessary, you can
64+
xref:../../support/remote_health_monitoring/opting-out-of-remote-health-reporting.adoc#opting-out-remote-health-reporting_opting-out-remote-health-reporting[opt out of remote health reporting].
65+
* If you need to enable external access to node ports, xref:../../networking/configuring_ingress_cluster_traffic/configuring-ingress-cluster-traffic-nodeport.adoc#nw-using-nodeport_configuring-ingress-cluster-traffic-nodeport[configure ingress cluster traffic by using a node port].
66+
* If you did not configure {rh-openstack} to accept application traffic over floating IP addresses, xref:../../post_installation_configuration/network-configuration.adoc#installation-osp-configuring-api-floating-ip_post-install-network-configuration[configure {rh-openstack} access with floating IP addresses].

‎machine_management/creating_machinesets/creating-machineset-osp.adoc‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,15 @@ include::modules/machine-api-overview.adoc[leveloffset=+1]
1111

1212
include::modules/machineset-yaml-osp.adoc[leveloffset=+1]
1313

14+
include::modules/machineset-yaml-osp-sr-iov.adoc[leveloffset=+1]
15+
16+
.Additional resources
17+
18+
* xref:../../installing/installing_openstack/installing-openstack-installer-sr-iov.adoc#installing-openstack-installer-sr-iov[Installing a cluster on OpenStack that supports SR-IOV-connected compute machines]
19+
20+
include::modules/machineset-yaml-osp-sr-iov-port-security.adoc[leveloffset=+1]
21+
1422
include::modules/machineset-creating.adoc[leveloffset=+1]
1523

1624
// Mothballed - re-add when available
17-
// include::modules/machineset-osp-adding-bare-metal.adoc[leveloffset=+1]
25+
// include::modules/machineset-osp-adding-bare-metal.adoc[leveloffset=+1]

‎modules/installation-osp-control-compute-machines.adoc‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,14 @@
22
//
33
// * installing/installing_openstack/installing-openstack-installer-custom.adoc
44
// * installing/installing_openstack/installing-openstack-installer-kuryr.adoc
5+
// * installing/installing_openstack/installing-openstack-installer-sr-iov.adoc
56

67
ifeval::["{context}" == "installing-openstack-user-sr-iov"]
78
:osp-sr-iov:
89
endif::[]
10+
ifeval::["{context}" == "installing-openstack-installer-sr-iov"]
11+
:osp-sr-iov:
12+
endif::[]
913

1014
[id="installation-osp-control-compute-machines_{context}"]
1115
= Control plane and compute machines
@@ -41,3 +45,6 @@ endif::osp-sr-iov[]
4145
ifeval::["{context}" == "installing-openstack-user-sr-iov"]
4246
:!osp-sr-iov:
4347
endif::[]
48+
ifeval::["{context}" == "installing-openstack-installer-sr-iov"]
49+
:!osp-sr-iov:
50+
endif::[]

‎modules/machineset-yaml-osp-sr-iov-port-security.adoc‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * machine_management/creating_machinesets/creating-machineset-osp.adoc
4+
5+
[id="machineset-yaml-osp-sr-iov-port-security_{context}"]
6+
= Sample YAML for SR-IOV deployments where port security is disabled
7+
8+
To create single-root I/O virtualization (SR-IOV) ports on a network that has port security disabled, define a machine set that includes the ports as items in the `spec.template.spec.providerSpec.value.ports` list. This difference from the standard SR-IOV machine set is due to the automatic security group and allowed address pair configuration that occurs for ports that are created by using the network and subnet interfaces.
9+
10+
Ports that you define for machines subnets require:
11+
12+
* Allowed address pairs for the API and ingress virtual IP ports
13+
* The compute security group
14+
* Attachment to the machines network and subnet
15+
16+
[NOTE]
17+
====
18+
Only parameters that are specific to SR-IOV deployments where port security is disabled are described in this sample. To review a more general sample, see Sample YAML for a machine set custom resource that uses SR-IOV on {rh-openstack}".
19+
====
20+
21+
.An example machine set that uses SR-IOV networks and has port security disabled
22+
[source,yaml]
23+
----
24+
apiVersion: machine.openshift.io/v1beta1
25+
kind: MachineSet
26+
metadata:
27+
labels:
28+
machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
29+
machine.openshift.io/cluster-api-machine-role: <node_role>
30+
machine.openshift.io/cluster-api-machine-type: <node_role>
31+
name: <infrastructure_ID>-<node_role>
32+
namespace: openshift-machine-api
33+
spec:
34+
replicas: <number_of_replicas>
35+
selector:
36+
matchLabels:
37+
machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
38+
machine.openshift.io/cluster-api-machineset: <infrastructure_ID>-<node_role>
39+
template:
40+
metadata:
41+
labels:
42+
machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
43+
machine.openshift.io/cluster-api-machine-role: <node_role>
44+
machine.openshift.io/cluster-api-machine-type: <node_role>
45+
machine.openshift.io/cluster-api-machineset: <infrastructure_ID>-<node_role>
46+
spec:
47+
metadata: {}
48+
providerSpec:
49+
value:
50+
apiVersion: openstackproviderconfig.openshift.io/v1alpha1
51+
cloudName: openstack
52+
cloudsSecret:
53+
name: openstack-cloud-credentials
54+
namespace: openshift-machine-api
55+
flavor: <nova_flavor>
56+
image: <glance_image_name_or_location>
57+
kind: OpenstackProviderSpec
58+
ports:
59+
- allowedAddressPairs: <1>
60+
- ipAddress: <API_VIP_port_IP>
61+
- ipAddress: <ingress_VIP_port_IP>
62+
fixedIPs:
63+
- subnetID: <machines_subnet_UUID> <2>
64+
nameSuffix: nodes
65+
networkID: <machines_network_UUID> <2>
66+
securityGroups:
67+
- <compute_security_group_UUID> <3>
68+
- networkID: <SRIOV_network_UUID>
69+
nameSuffix: sriov
70+
fixedIPs:
71+
- subnetID: <SRIOV_subnet_UUID>
72+
tags:
73+
- sriov
74+
vnicType: direct
75+
portSecurity: False
76+
primarySubnet: <machines_subnet_UUID>
77+
serverMetadata:
78+
Name: <infrastructure_ID>-<node_role>
79+
openshiftClusterID: <infrastructure_ID>
80+
tags:
81+
- openshiftClusterID=<infrastructure_ID>
82+
trunk: false
83+
userDataSecret:
84+
name: worker-user-data
85+
configDrive: True
86+
----
87+
<1> Specify allowed address pairs for the API and ingress ports.
88+
<2> Specify the machines network and subnet.
89+
<3> Specify the compute machines security group.
90+
91+
[NOTE]
92+
====
93+
Trunking is enabled for ports that are created by entries in the networks and subnets lists. The name of ports that are created from these lists follow the pattern `<machine_name>-<nameSuffix>`. The `nameSuffix` field is required in port definitions.
94+
95+
Trunking is not enabled for ports that are defined in the ports list.
96+
97+
Optionally, you can add tags to ports as part of their `tags` lists.
98+
====
99+
100+
If your cluster uses Kuryr and the {rh-openstack} SR-IOV network has port security disabled, the primary port for compute machines must have:
101+
102+
* The value of the `spec.template.spec.providerSpec.value.networks.portSecurityEnabled` parameter set to `false`.
103+
104+
* For each subnet, the value of the `spec.template.spec.providerSpec.value.networks.subnets.portSecurityEnabled` parameter set to `false`.
105+
106+
* The value of `spec.template.spec.providerSpec.value.securityGroups` set to empty: `[]`.
107+
108+
.An example section of a machine set for a cluster on Kuryr that uses SR-IOV and has port security disabled
109+
[source,yaml]
110+
----
111+
...
112+
networks:
113+
- subnets:
114+
- uuid: <machines_subnet_UUID>
115+
portSecurityEnabled: false
116+
portSecurityEnabled: false
117+
securityGroups: []
118+
...
119+
----
120+
121+
In that case, you can apply the compute security group to the primary VM interface after the VM is created. For example, from a command line:
122+
[source,terminal]
123+
----
124+
$ openstack port set --enable-port-security --security-group <infrastructure_ID>-<node_role> <main_port_ID>
125+
----

‎modules/machineset-yaml-osp-sr-iov.adoc‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * machine_management/creating_machinesets/creating-machineset-osp.adoc
4+
5+
[id="machineset-yaml-osp-sr-iov_{context}"]
6+
= Sample YAML for a machine set custom resource that uses SR-IOV on {rh-openstack}
7+
8+
If you configured your cluster for single-root I/O virtualization (SR-IOV), you can create machine sets that use that technology.
9+
10+
This sample YAML defines a machine set that uses SR-IOV networks. The nodes that it creates are labeled with `node-role.openshift.io/<node_role>: ""`
11+
12+
In this sample, `infrastructure_ID` is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and `node_role` is the node label to add.
13+
14+
The sample assumes two SR-IOV networks that are named "radio" and "uplink". The networks are used in port definitions in the `spec.template.spec.providerSpec.value.ports` list.
15+
16+
[NOTE]
17+
====
18+
Only parameters that are specific to SR-IOV deployments are described in this sample. To review a more general sample, see "Sample YAML for a machine set custom resource on {rh-openstack}".
19+
====
20+
21+
.An example machine set that uses SR-IOV networks
22+
[source,yaml]
23+
----
24+
apiVersion: machine.openshift.io/v1beta1
25+
kind: MachineSet
26+
metadata:
27+
labels:
28+
machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
29+
machine.openshift.io/cluster-api-machine-role: <node_role>
30+
machine.openshift.io/cluster-api-machine-type: <node_role>
31+
name: <infrastructure_ID>-<node_role>
32+
namespace: openshift-machine-api
33+
spec:
34+
replicas: <number_of_replicas>
35+
selector:
36+
matchLabels:
37+
machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
38+
machine.openshift.io/cluster-api-machineset: <infrastructure_ID>-<node_role>
39+
template:
40+
metadata:
41+
labels:
42+
machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
43+
machine.openshift.io/cluster-api-machine-role: <node_role>
44+
machine.openshift.io/cluster-api-machine-type: <node_role>
45+
machine.openshift.io/cluster-api-machineset: <infrastructure_ID>-<node_role>
46+
spec:
47+
metadata:
48+
providerSpec:
49+
value:
50+
apiVersion: openstackproviderconfig.openshift.io/v1alpha1
51+
cloudName: openstack
52+
cloudsSecret:
53+
name: openstack-cloud-credentials
54+
namespace: openshift-machine-api
55+
flavor: <nova_flavor>
56+
image: <glance_image_name_or_location>
57+
serverGroupID: <optional_UUID_of_server_group>
58+
kind: OpenstackProviderSpec
59+
networks:
60+
- subnets:
61+
- UUID: <machines_subnet_UUID>
62+
ports:
63+
- networkID: <radio_network_UUID> <1>
64+
nameSuffix: radio
65+
fixedIPs:
66+
- subnetID: <radio_subnet_UUID> <2>
67+
tags:
68+
- sriov
69+
- radio
70+
vnicType: direct <3>
71+
portSecurity: false <4>
72+
- networkID: <uplink_network_UUID> <1>
73+
nameSuffix: uplink
74+
fixedIPs:
75+
- subnetID: <uplink_subnet_UUID> <2>
76+
tags:
77+
- sriov
78+
- uplink
79+
vnicType: direct <3>
80+
portSecurity: false <4>
81+
primarySubnet: <machines_subnet_UUID>
82+
securityGroups:
83+
- filter: {}
84+
name: <infrastructure_ID>-<node_role>
85+
serverMetadata:
86+
Name: <infrastructure_ID>-<node_role>
87+
openshiftClusterID: <infrastructure_ID>
88+
tags:
89+
- openshiftClusterID=<infrastructure_ID>
90+
trunk: true
91+
userDataSecret:
92+
name: <node_role>-user-data
93+
availabilityZone: <optional_openstack_availability_zone>
94+
configDrive: true <5>
95+
----
96+
<1> Enter a network UUID for each port.
97+
<2> Enter a subnet UUID for each port.
98+
<3> The value of the `vnicType` parameter must be `direct` for each port.
99+
<4> The value of the `portSecurity` parameter must be `false` for each port.
100+
+
101+
You cannot set security groups and allowed address pairs for ports when port security is disabled. Setting security groups on the instance applies the groups to all ports that are attached to it.
102+
<5> The value of the `configDrive` parameter must be `true`.
103+
104+
[NOTE]
105+
====
106+
Trunking is enabled for ports that are created by entries in the networks and subnets lists. The name of ports that are created from these lists follow the pattern `<machine_name>-<nameSuffix>`. The `nameSuffix` field is required in port definitions.
107+
108+
Trunking is not enabled for ports that are defined in the ports list.
109+
110+
Optionally, you can add tags to ports as part of their `tags` lists.
111+
====

0 commit comments

Comments
 (0)