Split PCI pools per PF

Each PCI device and each PF is a separate RP in Placement and the
scheduler allocate them specifically so the PCI filtering and claiming
also needs to handle these devices individually. Nova pooled PCI devices
together if they had the same device_spec and same device type and numa
node. Now this is changed that only pool VFs from the same parent PF.
Fortunately nova already handled consuming devices for a single
InstancePCIRequest from multiple PCI pools, so this change does not
affect the device consumption code path.

The test_live_migrate_server_with_neutron test needed to be changed.
Originally this test used a compute with the following config:
* PF 81.00.0
** VFs 81.00.[1-4]
* PF 81.01.0
** VFs 81.01.[1-4]
* PF 82.00.0

And booted a VM that needed one VF and one PF. This request has two
widely different solutions:
1) allocate the VF from under 81.00 and therefore consume 81.00.0 and
   allocate the 82.00.0 PF
   This was what the test asserted to happen.
2) allocate the VF from under 81.00 and therefore consume 81.00.0 and
   allocate the 81.00.0 PF and therefore consume all the VFs under it
   This results in a different amount of free devices than #1)

AFAIK nova does not have any implemented preference for consuming PFs
without VFs. The test just worked by chance (some internal device and
pool ordering made it that way). However when the PCI pools are split
nova started choosing solution #2) making the test fail. As both
solution is equally good from nova's scheduling contract perspective I
don't consider this as a behavior change. Therefore the test is updated
not to create a situation where two different scheduling solutions are
possible.

blueprint: pci-device-tracking-in-placement
Change-Id: I4b67cca3807fbda9e9b07b220a28e331def57624

Author: gibizer

nova/pci/stats.py

@@ -139,10 +139,19 @@ def _create_pool_keys_from_dev(
             return None
         tags = devspec.get_tags()
         pool = {k: getattr(dev, k) for k in self.pool_keys}
+
         if tags:
             pool.update(
                 {k: v for k, v in tags.items() if k not in self.ignored_tags}
             )
+        # NOTE(gibi): since PCI in placement maps a PCI dev or a PF to a
+        # single RP and the scheduler allocates from a specific RP we need
+        # to split the pools by PCI or PF address. We can still keep
+        # the VFs from the same parent PF in a single pool though as they
+        # are equivalent from placement perspective.
+        address = dev.parent_addr or dev.address
+        pool['address'] = address
+
         # NOTE(gibi): parent_ifname acts like a tag during pci claim but
         # not provided as part of the whitelist spec as it is auto detected
         # by the virt driver.

nova/tests/functional/libvirt/test_pci_sriov_servers.py

@@ -879,7 +879,7 @@ def test_live_migrate_server_with_neutron(self):
 
         # start two compute services with differing PCI device inventory
         source_pci_info = fakelibvirt.HostPCIDevicesInfo(
-            num_pfs=2, num_vfs=8, numa_node=0)
+            num_pfs=1, num_vfs=4, numa_node=0)
         # add an extra PF without VF to be used by direct-physical ports
         source_pci_info.add_device(
             dev_type='PF',
@@ -932,7 +932,7 @@ def test_live_migrate_server_with_neutron(self):
         # our source host should have marked two PCI devices as used, the VF
         # and the parent PF, while the future destination is currently unused
         self.assertEqual('test_compute0', server['OS-EXT-SRV-ATTR:host'])
-        self.assertPCIDeviceCounts('test_compute0', total=11, free=8)
+        self.assertPCIDeviceCounts('test_compute0', total=6, free=3)
         self.assertPCIDeviceCounts('test_compute1', total=4, free=4)
 
         # the instance should be on host NUMA node 0, since that's where our
@@ -956,7 +956,7 @@ def test_live_migrate_server_with_neutron(self):
                 # TODO(stephenfin): Stop relying on a side-effect of how nova
                 # chooses from multiple PCI devices (apparently the last
                 # matching one)
-                'pci_slot': '0000:81:01.4',
+                'pci_slot': '0000:81:00.4',
                 'physical_network': 'physnet4',
             },
             port['binding:profile'],
@@ -980,7 +980,7 @@ def test_live_migrate_server_with_neutron(self):
 
         # we should now have transitioned our usage to the destination, freeing
         # up the source in the process
-        self.assertPCIDeviceCounts('test_compute0', total=11, free=11)
+        self.assertPCIDeviceCounts('test_compute0', total=6, free=6)
         self.assertPCIDeviceCounts('test_compute1', total=4, free=1)
 
         # the instance should now be on host NUMA node 1, since that's where

nova/tests/unit/compute/test_resource_tracker.py

@@ -2401,7 +2401,10 @@ def test_claim_with_pci(self, migr_mock, pci_stats_mock,
                         vendor_id='0001',
                         product_id='0002',
                         numa_node=0,
-                        tags={'dev_type': 'type-PCI'},
+                        tags={
+                            'dev_type': 'type-PCI',
+                            'address': '0000:81:00.0'
+                        },
                         count=0
                     )
                 ]