driver/secheduler/docs for Adds Pick guest CPU architecture based on

host arch in libvirt driver support

This is split 2 of 3 for the architecture emulation feature.

This implements emulated multi-architecture support through qemu
within OpenStack Nova.

Additional config variable check to pull host architecture into
hw_architecture field for emulation checks to be made.

Adds a custom function that simply performs a check for
hw_emulation_architecture field being set, allowing for core code to
function as normal while enabling a simple check to enable emulated
architectures to follow the same path as all multi-arch support
already established for physical nodes but instead levaraging qemu
which allows for the overall emulation.

Added check for domain xml unit test to strip arch from the os tag,
as it is not required uefi checks, and only leveraged for emulation
checks.

Added additional test cases test_driver validating emulation
functionality with checking hw_emulation_architecture against the
os_arch/hw_architecture field. Added required os-traits and settings
for scheduler request_filter.

Added RISCV64 to architecture enum for better support in driver.

Implements: blueprint pick-guest-arch-based-on-host-arch-in-libvirt-driver
Closes-Bug: 1863728
Change-Id: Ia070a29186c6123cf51e1b17373c2dc69676ae7c
Signed-off-by: Jonathan Race <[email protected]>

Author: chateaulav

doc/source/admin/hw-emulation-architecture.rst

@@ -0,0 +1,133 @@
+..
+      Licensed under the Apache License, Version 2.0 (the "License"); you may
+      not use this file except in compliance with the License. You may obtain
+      a copy of the License at
+
+          http://www.apache.org/licenses/LICENSE-2.0
+
+      Unless required by applicable law or agreed to in writing, software
+      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+      License for the specific language governing permissions and limitations
+      under the License.
+
+============================================================================
+hw_emulation_architecture - Configuring QEMU instance emulation architecture
+============================================================================
+
+.. versionadded:: 25.0.0 (Yoga)
+
+   The libvirt driver now allows for handling of specific cpu architectures
+   when defined within the image metadata properties, to be emulated through
+   QEMU.
+
+   Added ``hw_emulation_architecture`` as an available image_meta property.
+
+.. note::
+
+   The following only applies to environments using libvirt compute hosts.
+   and should be considered experimental in its entirety, during its first
+   release as a feature.
+
+Introduction
+------------
+
+This capability is to fill a need with environments that do not have the
+capability to support the various cpu architectures that are present today
+with physical hardware. A small subset of architectures that are supported
+both within libvirt and QEMU have been selected as prime candidates for
+emulation support.
+
+While support has been added for the below base architectures, this does
+not guarantee that every subset or custom operating system that leverages
+one of these architectures will function.
+
+Configure
+---------
+
+-------------------
+QEMU Binary Support
+-------------------
+
+To ensure that libvirt and QEMU can properly handle the level of cpu
+emulation desired by the end-user, you are required to install the specific
+``qemu-system-XXX``, ``qemu-efi-arm``, ``qemu-efi-aarch64`` binaries on the
+compute nodes that will be providing support.
+
+---------------
+Console Support
+---------------
+
+Consideration need to be made in regards to which architectures you want to
+support, as there are limitations on support through spice, novnc, and
+serial. All testing and validation has been done to ensure that spice and
+serial connections function as expected.
+
+- ``AARCH64`` - Spice & Serial
+- ``S390X`` - Serial
+- ``PPC64LE`` - Spice & Serial
+- ``MIPSEL`` - untested
+
+--------------------------------
+Supported Emulated Architectures
+--------------------------------
+
+The supported emulated architectures require specific image meta
+properties to be set in order to trigger the proper settings to be
+configured by libvirtd.
+
+For end users the emulation architecture of an instance is controlled by the
+selection of an image with the ``hw_emulation_architecture`` image metadata
+property set.
+
+
+AARCH64
+~~~~~~~
+
+``Tested and Validated as functional``
+
+.. code-block:: shell
+
+    $ openstack image set --property hw_emulation_architecture=aarch64 $IMAGE
+    $ openstack image set --property hw_machine_type=virt $IMAGE
+    $ openstack image set --property hw_firmware_type=uefi $IMAGE
+
+S390x
+~~~~~
+
+``Tested and Validated as functional``
+
+.. code-block:: shell
+
+    $ openstack image set --property hw_emulation_architecture=s390x $IMAGE
+    $ openstack image set --property hw_machine_type=s390-ccw-virtio $IMAGE
+    $ openstack image set --property hw_video_model=virtio $IMAGE
+
+PPC64LE
+~~~~~~~
+
+``Tested and Validated as functional``
+
+.. code-block:: shell
+
+    $ openstack image set --property hw_emulation_architecture=ppc64le $IMAGE
+    $ openstack image set --property hw_machine_type=pseries $IMAGE
+
+
+MIPSEL
+~~~~~~
+
+``Testing and validation is ongoing to overcome PCI issues``
+
+.. note::
+
+   Support is currently impacted, one current method for support is manually
+   patching and compiling as defined in libvirt bug
+   `XML error: No PCI buses available`_.
+
+.. _`XML error: No PCI buses available`: https://bugzilla.redhat.com/show_bug.cgi?id=1432101
+
+.. code-block:: shell
+
+    $ openstack image set --property hw_emulation_architecture=mipsel $IMAGE
+    $ openstack image set --property hw_machine_type=virt $IMAGE

doc/source/admin/index.rst

@@ -227,3 +227,4 @@ Once you are running nova, the following information is extremely useful.
    upgrades
    node-down
    hw-machine-type
+   hw-emulation-architecture

nova/scheduler/request_filter.py

@@ -212,6 +212,8 @@ def transform_image_metadata(ctxt, request_spec):
         'hw_disk_bus': 'COMPUTE_STORAGE_BUS',
         'hw_video_model': 'COMPUTE_GRAPHICS_MODEL',
         'hw_vif_model': 'COMPUTE_NET_VIF_MODEL',
+        'hw_architecture': 'HW_ARCH',
+        'hw_emulation_architecture': 'COMPUTE_ARCH',
     }
 
     trait_names = []

nova/tests/functional/libvirt/test_uefi.py

@@ -14,6 +14,7 @@
 # under the License.
 
 import datetime
+import re
 
 from lxml import etree
 from oslo_log import log as logging
@@ -47,6 +48,8 @@ def test_create_server(self):
         orig_create = nova.virt.libvirt.guest.Guest.create
 
         def fake_create(cls, xml, host):
+            xml = re.sub('type arch.*machine',
+                'type machine', xml)
             tree = etree.fromstring(xml)
             self.assertXmlEqual(
                 """

nova/tests/unit/scheduler/test_request_filter.py

@@ -406,13 +406,15 @@ def test_compute_status_filter(self, mock_log):
         self.assertIn('took %.1f seconds', log_lines[1])
 
     @mock.patch.object(request_filter, 'LOG', new=mock.Mock())
-    def test_transform_image_metadata(self):
+    def test_transform_image_metadata_x86(self):
         self.flags(image_metadata_prefilter=True, group='scheduler')
         properties = objects.ImageMetaProps(
             hw_disk_bus=objects.fields.DiskBus.SATA,
             hw_cdrom_bus=objects.fields.DiskBus.IDE,
             hw_video_model=objects.fields.VideoModel.QXL,
-            hw_vif_model=network_model.VIF_MODEL_VIRTIO
+            hw_vif_model=network_model.VIF_MODEL_VIRTIO,
+            hw_architecture=objects.fields.Architecture.X86_64,
+            hw_emulation_architecture=objects.fields.Architecture.AARCH64
         )
         reqspec = objects.RequestSpec(
             image=objects.ImageMeta(properties=properties),
@@ -426,6 +428,36 @@ def test_transform_image_metadata(self):
             'COMPUTE_NET_VIF_MODEL_VIRTIO',
             'COMPUTE_STORAGE_BUS_IDE',
             'COMPUTE_STORAGE_BUS_SATA',
+            'HW_ARCH_X86_64',
+            'COMPUTE_ARCH_AARCH64',
+        }
+        self.assertEqual(expected, reqspec.root_required)
+
+    @mock.patch.object(request_filter, 'LOG', new=mock.Mock())
+    def test_transform_image_metadata_aarch64(self):
+        self.flags(image_metadata_prefilter=True, group='scheduler')
+        properties = objects.ImageMetaProps(
+            hw_disk_bus=objects.fields.DiskBus.SATA,
+            hw_cdrom_bus=objects.fields.DiskBus.IDE,
+            hw_video_model=objects.fields.VideoModel.QXL,
+            hw_vif_model=network_model.VIF_MODEL_VIRTIO,
+            hw_architecture=objects.fields.Architecture.AARCH64,
+            hw_emulation_architecture=objects.fields.Architecture.X86_64
+        )
+        reqspec = objects.RequestSpec(
+            image=objects.ImageMeta(properties=properties),
+            flavor=objects.Flavor(extra_specs={}),
+        )
+        self.assertTrue(
+            request_filter.transform_image_metadata(None, reqspec)
+        )
+        expected = {
+            'COMPUTE_GRAPHICS_MODEL_QXL',
+            'COMPUTE_NET_VIF_MODEL_VIRTIO',
+            'COMPUTE_STORAGE_BUS_IDE',
+            'COMPUTE_STORAGE_BUS_SATA',
+            'HW_ARCH_AARCH64',
+            'COMPUTE_ARCH_X86_64',
         }
         self.assertEqual(expected, reqspec.root_required)
 

nova/tests/unit/virt/libvirt/test_driver.py

@@ -5218,7 +5218,8 @@ def test_check_uefi_support_aarch64(self):
         self.mock_uname.return_value = fakelibvirt.os_uname(
             'Linux', '', '5.4.0-0-generic', '', fields.Architecture.AARCH64)
         drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True)
-        self.assertTrue(drvr._check_uefi_support(None))
+        image_meta = objects.ImageMeta.from_dict(self.test_image_meta)
+        self.assertTrue(drvr._check_uefi_support(image_meta))
 
     def test_get_guest_config_with_block_device(self):
         drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True)
@@ -5769,6 +5770,7 @@ def test_create_serial_console_devices_based_on_arch(
         self.flags(enabled=True, group='serial_console')
         drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True)
         instance = objects.Instance(**self.test_instance)
+        image_meta = objects.ImageMeta.from_dict(self.test_image_meta)
 
         expected = {
           fields.Architecture.X86_64: vconfig.LibvirtConfigGuestSerial,
@@ -5781,7 +5783,10 @@ def test_create_serial_console_devices_based_on_arch(
 
             guest = vconfig.LibvirtConfigGuest()
             drvr._create_consoles(
-                guest_cfg=guest, instance=instance, flavor={}, image_meta={})
+                guest_cfg=guest,
+                instance=instance,
+                flavor={},
+                image_meta=image_meta)
             self.assertEqual(1, len(guest.devices))
             console_device = guest.devices[0]
             self.assertIsInstance(console_device, device_type)
@@ -5993,9 +5998,13 @@ def _test_consoles(arch_to_mock, serial_enabled,
             self.flags(enabled=serial_enabled, group='serial_console')
             guest_cfg = vconfig.LibvirtConfigGuest()
             instance = objects.Instance(**self.test_instance)
+            image_meta = objects.ImageMeta.from_dict(self.test_image_meta)
 
             drvr._create_consoles(
-                guest_cfg, instance=instance, flavor=None, image_meta=None)
+                guest_cfg,
+                instance=instance,
+                flavor=None,
+                image_meta=image_meta)
 
             self.assertEqual(1, len(guest_cfg.devices))
             device = guest_cfg.devices[0]
@@ -7975,6 +7984,33 @@ def test_get_guest_cpu_config_qemu_custom_aarch64(self):
         self.assertIsInstance(conf.cpu, vconfig.LibvirtConfigGuestCPU)
         self.assertEqual(conf.cpu.mode, 'custom')
 
+    def test_get_x86_64_hw_emulated_architecture_aarch64(self):
+        drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True)
+        image_meta = objects.ImageMeta.from_dict({
+            "disk_format": "raw",
+            'properties': {
+                    'hw_architecture': 'x86_64',
+                    'hw_emulation_architecture': 'aarch64',
+                    'hw_machine_type': 'virt',
+                    'hw_firmware_type': 'uefi',
+                    }})
+
+        self.assertEqual(drvr._check_emulation_arch(image_meta),
+                        'aarch64')
+
+    def test_get_x86_64_hw_emulated_architecture_ppc64(self):
+        drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True)
+        image_meta = objects.ImageMeta.from_dict({
+            "disk_format": "raw",
+            'properties': {
+                    'hw_architecture': 'x86_64',
+                    'hw_emulation_architecture': 'ppc64le',
+                    'hw_machine_type': 'pseries',
+                    }})
+
+        self.assertEqual(drvr._check_emulation_arch(image_meta),
+                        'ppc64le')
+
     @mock.patch.object(libvirt_driver.LOG, 'warning')
     def test_get_guest_cpu_config_custom_with_extra_flags(self,
             mock_warn):

nova/virt/libvirt/config.py

@@ -2823,6 +2823,7 @@ def __init__(self, **kwargs):
         self.os_init_path = None
         self.os_boot_dev = []
         self.os_smbios = None
+        self.os_arch = None
         self.os_mach_type = None
         self.os_bootmenu = False
         self.devices = []
@@ -2865,6 +2866,8 @@ def _format_os(self, root):
             os.set("firmware", self.os_firmware)
 
         type_node = self._text_node("type", self.os_type)
+        if self.os_arch is not None:
+            type_node.set("arch", self.os_arch)
         if self.os_mach_type is not None:
             type_node.set("machine", self.os_mach_type)
         os.append(type_node)