Merge pull request #551 from jamepark4/add_mdev_install_srvc

Create example install-nvidia service

Migrating the work from CIFMW [1] to the architecture repo.  Instead of having CIFMW be responsible for updating the EDPMs and installing the required Nvidia driver, instead have the procedure handled by a composable service defined in the architecture repo.
[1] openstack-k8s-operators/ci-framework#2637

Reviewed-by: John Fulton <[email protected]>
Reviewed-by: jamepark4 <[email protected]>

Author: softwarefactory-project-zuul[bot]

automation/vars/nvidia-mdev.yaml

@@ -31,7 +31,7 @@ vas:
           - >-
             oc -n openstack wait
             osdpns openstack-edpm --for condition=SetupReady
-            --timeout=60m
+            --timeout=90m
         values:
           - name: edpm-nodeset-values
             src_file: values.yaml
@@ -47,16 +47,6 @@ vas:
           - name: edpm-deployment-values
             src_file: values.yaml
         build_output: deployment.yaml
-        post_stage_run:
-          - name: Run phase 1 playbook
-            type: playbook
-            # As a reminder, the job needs to set the nvidia driver URL
-            source: "../../playbooks/nvidia-mdev-phase1.yml"
-            inventory: "${HOME}/ci-framework-data/artifacts/zuul_inventory.yml"
-          - name: Run phase 2 playbook
-            type: playbook
-            source: "../../playbooks/nvidia-mdev-phase2.yml"
-            inventory: "${HOME}/ci-framework-data/artifacts/zuul_inventory.yml"
 
       - path: examples/va/nvidia-mdev/edpm-post-driver/nodeset
         wait_conditions:

examples/va/nvidia-mdev/README.md

@@ -45,7 +45,9 @@ virtualization and requires real baremetal hosts.
 #### Nova Mdev Configuration
 
 To deploy vGPU devices comprised of different types as well as the capacity to 
-live migrate, you would need the below configuration applied to Nova.
+live migrate, define what mdev types should be enabled and map the respective
+mdev types to their pci address(es). Example below using nivida-228 and
+nvidia-229 as the types.
 
 ```YAML
 ---
@@ -70,94 +72,32 @@ metadata:
   namespace: openstack
 ```
 
+#### Openstack Dataplane Composable Service
+
+An Openstack Dataplane service can used to customize how the GPU cards need to
+be installed on the EDPM nodes. An example of OSPDS service can be seen
+[here](../../../va/nvidia-mdev/edpm/nodeset/nova_sriov.yaml). With the OSDPS
+configured, the operator would need to make sure to include the service to the
+list of services for the Openstack Dataplane NodeSet.
+
+**Note:** The example listed is not an officially supported procedure for
+installing Nvidia GPU's in RHOSO and is meant to be purely an example of how
+to leverage OSDPS. Please reference Nvidia's documentation when creating a
+procedure to install GPU's.
+
 #### Provider.yaml
 
 In order to easily take advantage of multiple Mdev types in an environment when
 creating flavors, we can associate traits to specific resource providers. With
-provier.yaml we can map those traits and apply them as part of a deployment.
-
-```YAML
----
-apiVersion: v1
-data:
-  provider.yaml: |
-    meta:
-      schema_version: "1.0"
-    providers:
-      - identification:
-          name: edpm-compute-0.ctlplane.example.com_pci_0000_04_00_0
-        traits:
-          additional:
-            - CUSTOM_NVIDIA_229
-      - identification:
-          name: edpm-compute-0.ctlplane.example.com_pci_0000_82_00_0
-        traits:
-          additional:
-            - CUSTOM_NVIDIA_228
-      - identification:
-          name: edpm-compute-1.ctlplane.example.com_pci_0000_04_00_0
-        traits:
-          additional:
-            - CUSTOM_NVIDIA_229
-      - identification:
-          name: edpm-compute-1.ctlplane.example.com_pci_0000_82_00_0
-        traits:
-          additional:
-            - CUSTOM_NVIDIA_228
-kind: ConfigMap
-  name: compute-provider
-  namespace: openstack
----
-apiVersion: dataplane.openstack.org/v1beta1
-kind: OpenStackDataPlaneService
-metadata:
-  name: compute-provider
-  namespace: openstack
-spec:
-  addCertMounts: false
-  caCerts: combined-ca-bundle
-  dataSources:
-  - configMapRef:
-      name: compute-provider
-  - configMapRef:
-      name: cpu-pinning-nova
-  - configMapRef:
-      name: sriov-nova
-  - secretRef:
-      name: nova-cell1-compute-config
-  - secretRef:
-      name: nova-migration-ssh-key
-  edpmServiceType: nova
-  playbook: osp.edpm.nova
-  tlsCerts:
-    default:
-      contents:
-      - dnsnames
-      - ips
-      issuer: osp-rootca-issuer-internal
-      networks:
-      - ctlplane
----
-apiVersion: dataplane.openstack.org/v1beta1
-kind: OpenStackDataPlaneDeployment
-metadata:
-  name: edpm-deployment-post-driver
-  namespace: openstack
-spec:
-  ansibleExtraVars:
-    edpm_reboot_strategy: force
-  nodeSets:
-  - openstack-edpm
-  preserveJobs: true
-  servicesOverride:
-  - reboot-os
-  - compute-provider
-```
+provider.yaml we can map those traits and apply them as part of a deployment.
+An example definition can be found [here](edpm-post-driver/nodeset/values.yaml)
+that associates different custom traits to different RPs.
 
 ## Stages
-All stages must be executed in the order listed below. Everything is required unless otherwise indicated.
+All stages must be executed in the order listed below. Everything is required
+unless otherwise indicated.
 
 1. [Install the OpenStack K8S operators and their dependencies](../../common/)
 2. [Configuring networking and deploy the OpenStack control plane](control-plane.md)
 3. [Configure and deploy the initial dataplane](edpm-pre.md)
-4. [Update Dataplane to deploy necessary vGPU MDev requirements](edpm-post.md)
+4. [Update Dataplane to reboot EDPM nodes and optionally apply provider.yaml](edpm-post.md)

examples/va/nvidia-mdev/edpm-post.md

@@ -1,26 +1,9 @@
-# Download Nvidia drivers to EDPM Nodes and apply follow up deployment
+# Reboot EDPM Nodes and optionally apply provider.yaml
 
 ## Assumptions
 
 - Initial [dataplane](edpm-pre.md) deployment has finalized and is successful
 
-## Apply necessary Nvidia configurations to EDPM Nodes
-### The following commands should be executed on every EDPM Node expected to provide vGPU MDevs
-Log into EDPM Node and update or create a blacklist file in /etc/modeprobe
-```
-cat /etc/modprobe.d/blacklist-nouveau.conf
-blacklist nouveau
-options nouveau modeset=0
-```
-Download the relevant Nvidia driver for your hardware and install per Nvidia's
-instructions.
-
-Regenerate initramfs
-```
-dracut --force
-grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline
-```
-
 ## Create a post deployment to finalize Nvidia configuration
 Log out of EDPMs and return to architecture repo on the controller.
 
@@ -29,9 +12,9 @@ cd architecture/examples/va/nvidia-mdev/edpm-post-driver
 ```
 
 ### Optional: Create a provider.yaml
-Create a configmap for the provider.yaml to map CUSTOM_TRAITS to the relevant
-resource providers and their MDevs, then create the corresponding service that
-will apply the configMap.
+Create a configmap for the ```provider.yaml``` to map ```CUSTOM_TRAITS``` to
+the relevant resource providers and their MDevs, then create the corresponding
+service that will apply the configMap.
 
 Update the post [nodeset](edpm-post-driver/nodeset/values.yaml) values to how
 you wish to map resource provider to traits.
@@ -42,11 +25,15 @@ kustomize build nodeset > compute-provider-service.yaml
 oc apply -f compute-provider-service.yaml
 ```
 
-## Reboot EDPM Nodes and optionaly apply provider.yaml
-In order finish Nvidia Driver installation the EDPM Nodes will need a final
+## Update post deployment configration and apply
+In order to finish Nvidia Driver installation the EDPM Nodes will need a final
 reboot. This will require a new deployment that will run ```reboot-os``` on the
 relevant EDPM Nodes.
 
+If applying the ```provider.yaml``` configuration via OSPDS from the previous
+optional step, then include the service ```compute-provider``` to the list of
+services as well.
+
 Update [deployment](edpm-post-driver/deployment/values.yaml) values to suit
 your environment and to include provider.yaml if using.
 ```

examples/va/nvidia-mdev/edpm/nodeset/kustomization.yaml

@@ -72,3 +72,15 @@ replacements:
           - data.03-sriov-nova\.conf
         options:
           create: true
+  - source:
+      kind: ConfigMap
+      name: edpm-nodeset-values
+      fieldPath: data.nova.mdev.nvidia_mdev_driver_url
+    targets:
+      - select:
+          kind: ConfigMap
+          name: nvidia-url
+        fieldPaths:
+          - data.nvidia_mdev_driver_url
+        options:
+          create: true

examples/va/nvidia-mdev/edpm/nodeset/values.yaml

@@ -136,6 +136,7 @@ data:
       - neutron-sriov
       - libvirt
       - nova-custom-sriov
+      - install-nvidia
   nova:
     compute:
       conf: |
@@ -158,3 +159,5 @@ data:
         # CHANGEME
         [pci]
         device_spec = {"vendor_id":"8086", "product_id":"1572", "address": "0000:19:00.3", "physical_network":"sriov-phy4", "trusted":"true"}
+    mdev:
+      nvidia_mdev_driver_url: http://example.nvidia.com/path/to/driver/example-rhel-host-driver-xyz.rpm

va/nvidia-mdev/edpm/nodeset/nova_sriov.yaml

@@ -1,3 +1,6 @@
+# Note: The OSPDS shown below is not an official procedure for installing
+# Nvidia drivers, it is only an example of how to leverage OSPDS to customize
+# the installtion process.
 ---
 apiVersion: v1
 kind: ConfigMap
@@ -13,6 +16,13 @@ metadata:
 data:
   03-sriov-nova.conf: _replaced_
 ---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nvidia-url
+data:
+  nvidia_mdev_driver_url: _replaced_
+---
 apiVersion: dataplane.openstack.org/v1beta1
 kind: OpenStackDataPlaneService
 metadata:
@@ -39,3 +49,59 @@ spec:
         - ctlplane
       issuer: osp-rootca-issuer-internal
   caCerts: combined-ca-bundle
+---
+apiVersion: dataplane.openstack.org/v1beta1
+kind: OpenStackDataPlaneService
+metadata:
+  name: install-nvidia
+  namespace: openstack
+spec:
+  dataSources:
+    - configMapRef:
+        name: nvidia-url
+  playbookContents: |
+    - name: Install Nvidia Driver
+      hosts: all
+      tasks:
+        - name: Blacklist nouveau
+          become: true
+          ansible.builtin.copy:
+            dest: "/etc/modprobe.d/blacklist-nouveau.conf"
+            mode: "0644"
+            content: |-
+              blacklist nouveau
+              options nouveau modeset=0
+            force: false
+          register: _blacklist_nouveau
+        - name: Get the Nvidia Driver URL
+          delegate_to: localhost
+          ansible.builtin.set_fact:
+            nvidia_rpm_url: "{{ lookup('file', '/var/lib/openstack/configs/install-nvidia/nvidia_mdev_driver_url') | from_yaml }}"
+        - name: Gather the package facts
+          ansible.builtin.package_facts:
+            manager: auto
+        - name: Install nvidia driver RPM either from path or URL
+          become: true
+          ansible.builtin.dnf:
+            name: "{{ nvidia_rpm_url }}"
+            state: present
+            disable_gpg_check: true
+          when: nvidia_rpm_url not in ansible_facts.packages
+          register: _nvidia_driver_install
+        - name: Check if grub2-mkconfig has --update-bls-cmdline option
+          ansible.builtin.shell:
+            cmd: grub2-mkconfig --help | grep '\-\-update-bls-cmdline'
+          ignore_errors: true
+          register: check_update_bls_cmdline
+          changed_when: false
+        - name: Regenerate initramfs
+          become: true
+          ansible.builtin.command: "{{ item }}"
+          loop:
+            - 'dracut --force'
+            - >-
+              grub2-mkconfig -o /boot/grub2/grub.cfg
+              {{ '--update-bls-cmdline'
+              if check_update_bls_cmdline.rc == 0
+              else '' }}
+          when: _blacklist_nouveau.changed or _nvidia_driver_install.changed