Detect maximum number of SEV guests automatically

Libvirt has implemented the capability to expose maximum number of
SEV guests and SEV-ES guests in 8.0.0[1][2]. This allows nova to detect
maximum number of memory encrypted guests using that feature.
The detection is not used if the [libvirt] num_memory_encrypted_guests
option is set to preserve the current behavior.

Note that current nova supports only SEV and does not support SEV-ES,
so this implementation only uses the maximum number of SEV guests.
The maximum number of SEV-ES guests will be used in case we implement
support for SEV-ES.

[1] https://gitlab.com/libvirt/libvirt/-/commit/34cb8f6fcd6a56a7bbcef2f7402def1682509e16
[2] https://gitlab.com/libvirt/libvirt/-/commit/7826148a72c97367fc6aaa76397fe92d32169723

Implements: blueprint libvirt-detect-sev-max-guests
Change-Id: I502e1713add7e6a1eb11ecce0cc2b5eb6a14527a

Author: kajinamit

doc/source/admin/sev.rst

@@ -91,32 +91,31 @@ steps:
   needs to track how many slots are available and used in order to
   avoid attempting to exceed that limit in the hardware.
 
-  At the time of writing (September 2019), work is in progress to
-  allow QEMU and libvirt to expose the number of slots available on
-  SEV hardware; however until this is finished and released, it will
-  not be possible for Nova to programmatically detect the correct
-  value.
-
-  So this configuration option serves as a stop-gap, allowing the
-  cloud operator the option of providing this value manually.  It may
-  later be demoted to a fallback value for cases where the limit
-  cannot be detected programmatically, or even removed altogether when
-  Nova's minimum QEMU version guarantees that it can always be
-  detected.
+  Since version 8.0.0, libvirt exposes maximun mumber of SEV guests
+  which can run concurrently in its host, so the limit is automatically
+  detected using this feature.
+
+  However in case an older version of libvirt is used, it is not possible for
+  Nova to programmatically detect the correct value and Nova imposes no limit.
+  So this configuration option serves as a stop-gap, allowing the cloud
+  operator the option of providing this value manually.
+
+  This option also allows the cloud operator to set the limit lower than
+  the actual hard limit.
 
   .. note::
 
-     When deciding whether to use the default of ``None`` or manually
-     impose a limit, operators should carefully weigh the benefits
-     vs. the risk.  The benefits of using the default are a) immediate
-     convenience since nothing needs to be done now, and b) convenience
-     later when upgrading compute hosts to future versions of Nova,
-     since again nothing will need to be done for the correct limit to
-     be automatically imposed.  However the risk is that until
-     auto-detection is implemented, users may be able to attempt to
-     launch guests with encrypted memory on hosts which have already
-     reached the maximum number of guests simultaneously running with
-     encrypted memory.  This risk may be mitigated by other limitations
+     If libvirt older than 8.0.0 is used, operators should carefully weigh
+     the benefits vs. the risk when deciding whether to use the default of
+     ``None`` or manually impose a limit.
+     The benefits of using the default are a) immediate convenience since
+     nothing needs to be done now, and b) convenience later when upgrading
+     compute hosts to future versions of libvirt, since again nothing will
+     need to be done for the correct limit to be automatically imposed.
+     However the risk is that until auto-detection is implemented, users may
+     be able to attempt to launch guests with encrypted memory on hosts which
+     have already reached the maximum number of guests simultaneously running
+     with encrypted memory.  This risk may be mitigated by other limitations
      which operators can impose, for example if the smallest RAM
      footprint of any flavor imposes a maximum number of simultaneously
      running guests which is less than or equal to the SEV limit.
@@ -221,16 +220,6 @@ features:
   include using ``hw_disk_bus=scsi`` with
   ``hw_scsi_model=virtio-scsi`` , or ``hw_disk_bus=sata``.
 
-- QEMU and libvirt cannot yet expose the number of slots available for
-  encrypted guests in the memory controller on SEV hardware.  Until
-  this is implemented, it is not possible for Nova to programmatically
-  detect the correct value.  As a short-term workaround, operators can
-  optionally manually specify the upper limit of SEV guests for each
-  compute host, via the new
-  :oslo.config:option:`libvirt.num_memory_encrypted_guests`
-  configuration option :ref:`described above
-  <num_memory_encrypted_guests>`.
-
 Permanent limitations
 ~~~~~~~~~~~~~~~~~~~~~
 

nova/conf/libvirt.py

@@ -854,14 +854,15 @@
 future.  If the machine does not support memory encryption, the option
 will be ignored and inventory will be set to 0.
 
-If the machine does support memory encryption, *for now* a value of
-``None`` means an effectively unlimited inventory, i.e. no limit will
-be imposed by Nova on the number of SEV guests which can be launched,
-even though the underlying hardware will enforce its own limit.
-However it is expected that in the future, auto-detection of the
-inventory from the hardware will become possible, at which point
-``None`` will cause auto-detection to automatically impose the correct
-limit.
+If the machine does support memory encryption and this option is not set,
+the driver detects maximum number of SEV guests from the libvirt API which
+is available since v8.0.0. Setting this option overrides the detected limit,
+unless the given value is not larger than the detected limit.
+
+On the other hand, if an older version of libvirt is used, ``None`` means
+an effectively unlimited inventory, i.e. no limit will be imposed by Nova
+on the number of SEV guests which can be launched, even though the underlying
+hardware will enforce its own limit.
 
 .. note::
 

nova/tests/fixtures/libvirt.py

@@ -1990,6 +1990,16 @@ def getCPUModelNames(self, arch):
     _domain_capability_features_with_SEV_unsupported = \
         _domain_capability_features_with_SEV.replace('yes', 'no')
 
+    _domain_capability_features_with_SEV_max_guests = '''  <features>
+    <gic supported='no'/>
+    <sev supported='yes'>
+      <cbitpos>47</cbitpos>
+      <reducedPhysBits>1</reducedPhysBits>
+      <maxGuests>100</maxGuests>
+      <maxESGuests>15</maxESGuests>
+    </sev>
+  </features>'''
+
     def getCapabilities(self):
         """Return spoofed capabilities."""
         numa_topology = self.host_info.numa_topology

nova/tests/unit/virt/libvirt/test_driver.py

@@ -29368,7 +29368,7 @@ def test_get_mem_encrypted_slots_unsupported(self):
 
 @mock.patch.object(vc, '_domain_capability_features',
                    new=vc._domain_capability_features_with_SEV)
-class TestLibvirtSEVSupported(TestLibvirtSEV):
+class TestLibvirtSEVSupportedNoMaxGuests(TestLibvirtSEV):
     """Libvirt driver tests for when AMD SEV support is present."""
     @test.patch_exists(SEV_KERNEL_PARAM_FILE, True)
     @test.patch_open(SEV_KERNEL_PARAM_FILE, "1\n")
@@ -29389,6 +29389,36 @@ def test_get_mem_encrypted_slots_config_zero_supported(self):
         self.assertEqual(0, self.driver._get_memory_encrypted_slots())
 
 
+@mock.patch.object(vc, '_domain_capability_features',
+                   new=vc._domain_capability_features_with_SEV_max_guests)
+class TestLibvirtSEVSupportedMaxGuests(TestLibvirtSEV):
+    """Libvirt driver tests for when AMD SEV support is present."""
+    @test.patch_exists(SEV_KERNEL_PARAM_FILE, True)
+    @test.patch_open(SEV_KERNEL_PARAM_FILE, "1\n")
+    @mock.patch.object(libvirt_driver.LOG, 'warning')
+    def test_get_mem_encrypted_slots_no_override(self, mock_log):
+        self.assertEqual(100, self.driver._get_memory_encrypted_slots())
+        mock_log.assert_not_called()
+
+    @test.patch_exists(SEV_KERNEL_PARAM_FILE, True)
+    @test.patch_open(SEV_KERNEL_PARAM_FILE, "1\n")
+    @mock.patch.object(libvirt_driver.LOG, 'warning')
+    def test_get_mem_encrypted_slots_overlide_more(self, mock_log):
+        self.flags(num_memory_encrypted_guests=120, group='libvirt')
+        self.assertEqual(100, self.driver._get_memory_encrypted_slots())
+        mock_log.assert_called_with(
+            'Host is configured with libvirt.num_memory_encrypted_guests '
+            'set to %d, but supports only %d.', 120, 100)
+
+    @test.patch_exists(SEV_KERNEL_PARAM_FILE, True)
+    @test.patch_open(SEV_KERNEL_PARAM_FILE, "1\n")
+    @mock.patch.object(libvirt_driver.LOG, 'warning')
+    def test_get_mem_encrypted_slots_override_less(self, mock_log):
+        self.flags(num_memory_encrypted_guests=80, group='libvirt')
+        self.assertEqual(80, self.driver._get_memory_encrypted_slots())
+        mock_log.assert_not_called()
+
+
 class LibvirtPMEMNamespaceTests(test.NoDBTestCase):
 
     def setUp(self):

nova/tests/unit/virt/libvirt/test_host.py

@@ -890,6 +890,8 @@ def _test_get_domain_capabilities_sev(self, supported):
         if supported:
             self.assertEqual(47, sev.cbitpos)
             self.assertEqual(1, sev.reduced_phys_bits)
+            self.assertIsNone(sev.max_guests)
+            self.assertIsNone(sev.max_es_guests)
 
     @mock.patch.object(
         fakelibvirt.virConnect, '_domain_capability_features', new=
@@ -904,6 +906,22 @@ def test_get_domain_capabilities_sev_unsupported(self):
     def test_get_domain_capabilities_sev_supported(self):
         self._test_get_domain_capabilities_sev(True)
 
+    @mock.patch.object(
+        fakelibvirt.virConnect, '_domain_capability_features', new=
+        fakelibvirt.virConnect._domain_capability_features_with_SEV_max_guests)
+    def test_get_domain_capabilities_sev_max_guests(self):
+        caps = self._test_get_domain_capabilities()
+        self.assertEqual(vconfig.LibvirtConfigDomainCaps, type(caps))
+        features = caps.features
+        self.assertEqual(1, len(features))
+        sev = features[0]
+        self.assertEqual(vconfig.LibvirtConfigDomainCapsFeatureSev, type(sev))
+        self.assertTrue(sev.supported)
+        self.assertEqual(47, sev.cbitpos)
+        self.assertEqual(1, sev.reduced_phys_bits)
+        self.assertEqual(100, sev.max_guests)
+        self.assertEqual(15, sev.max_es_guests)
+
     @mock.patch.object(fakelibvirt.virConnect, "getHostname")
     def test_get_hostname_caching(self, mock_hostname):
         mock_hostname.return_value = "foo"

nova/virt/libvirt/config.py

@@ -310,6 +310,8 @@ def __init__(self, **kwargs):
         self.supported = False
         self.cbitpos = None
         self.reduced_phys_bits = None
+        self.max_guests = None
+        self.max_es_guests = None
 
     def parse_dom(self, xmldoc):
         super(LibvirtConfigDomainCapsFeatureSev, self).parse_dom(xmldoc)
@@ -322,6 +324,10 @@ def parse_dom(self, xmldoc):
                 self.reduced_phys_bits = int(c.text)
             elif c.tag == 'cbitpos':
                 self.cbitpos = int(c.text)
+            elif c.tag == 'maxGuests':
+                self.max_guests = int(c.text)
+            elif c.tag == 'maxESGuests':
+                self.max_es_guests = int(c.text)
 
 
 class LibvirtConfigDomainCapsOS(LibvirtConfigObject):

nova/virt/libvirt/driver.py

@@ -9009,33 +9009,30 @@ def _update_provider_tree_for_vpmems(self, provider_tree, nodename,
                 resources[rc].add(resource_obj)
 
     def _get_memory_encrypted_slots(self):
-        slots = CONF.libvirt.num_memory_encrypted_guests
+        conf_slots = CONF.libvirt.num_memory_encrypted_guests
+
         if not self._host.supports_amd_sev:
-            if slots and slots > 0:
+            if conf_slots and conf_slots > 0:
                 LOG.warning("Host is configured with "
                             "libvirt.num_memory_encrypted_guests set to "
-                            "%d, but is not SEV-capable.", slots)
+                            "%d, but is not SEV-capable.", conf_slots)
             return 0
 
-        # NOTE(aspiers): Auto-detection of the number of available
-        # slots for AMD SEV is not yet possible, so honor the
-        # configured value, or impose no limit if this is not
-        # specified.  This does incur a risk that if operators don't
-        # read the instructions and configure the maximum correctly,
-        # the maximum could be exceeded resulting in SEV guests
-        # failing at launch-time.  However at least SEV guests will
-        # launch until the maximum, and when auto-detection code is
-        # added later, an upgrade will magically fix the issue.
-        #
-        # Note also that the configured value can be 0 on an
-        # SEV-capable host, since there might conceivably be good
-        # reasons for the operator to want to disable SEV even when
-        # it's available (e.g. due to performance impact, or
-        # implementation bugs which may surface later).
-        if slots is not None:
-            return slots
-        else:
-            return db_const.MAX_INT
+        slots = db_const.MAX_INT
+
+        # NOTE(tkajinam): Current nova supports SEV only so we ignore SEV-ES
+        if self._host.max_sev_guests is not None:
+            slots = self._host.max_sev_guests
+
+        if conf_slots is not None:
+            if conf_slots > slots:
+                LOG.warning("Host is configured with "
+                            "libvirt.num_memory_encrypted_guests set to %d, "
+                            "but supports only %d.", conf_slots, slots)
+            slots = min(slots, conf_slots)
+
+        LOG.debug("Available memory encrypted slots: %d", slots)
+        return slots
 
     @property
     def static_traits(self) -> ty.Dict[str, bool]:

nova/virt/libvirt/host.py

@@ -162,6 +162,8 @@ def __init__(self, uri, read_only=False,
         # kernel, QEMU, and/or libvirt. These are determined on demand and
         # memoized by various properties below
         self._supports_amd_sev: ty.Optional[bool] = None
+        self._max_sev_guests: ty.Optional[int] = None
+        self._max_sev_es_guests: ty.Optional[int] = None
         self._supports_uefi: ty.Optional[bool] = None
         self._supports_secure_boot: ty.Optional[bool] = None
 
@@ -1836,11 +1838,29 @@ def supports_amd_sev(self) -> bool:
                     if feature_is_sev and feature.supported:
                         LOG.info("AMD SEV support detected")
                         self._supports_amd_sev = True
+                        self._max_sev_guests = feature.max_guests
+                        self._max_sev_es_guests = feature.max_es_guests
                         return self._supports_amd_sev
 
         LOG.debug("No AMD SEV support detected for any (arch, machine_type)")
         return self._supports_amd_sev
 
+    @property
+    def max_sev_guests(self) -> ty.Optional[int]:
+        """Determine maximum number of guests with AMD SEV.
+        """
+        if not self.supports_amd_sev:
+            return None
+        return self._max_sev_guests
+
+    @property
+    def max_sev_es_guests(self) -> ty.Optional[int]:
+        """Determine maximum number of guests with AMD SEV-ES.
+        """
+        if not self.supports_amd_sev:
+            return None
+        return self._max_sev_es_guests
+
     @property
     def supports_remote_managed_ports(self) -> bool:
         """Determine if the host supports remote managed ports.

releasenotes/notes/max-sev-guests-e3da9adb3f75fbb8.yaml

@@ -0,0 +1,6 @@
+---
+features:
+  - |
+    Now the libvirt driver is capable to detect maximum number of guests with
+    memory encrypted which can run concurrently in its compute host using
+    the new fields in libvirt API available since version 8.0.0.