Skip to content

Commit 099a6f6

Browse files
gibizergibizer
committed
Optimize numa_fit_instance_to_host
The numa_fit_instance_to_host algorithm tries all the possible host cell permutations to fit the instance cells. So in worst case scenario it does n! / (n-k)! _numa_fit_instance_cell calls (n=len(host_cells) k=len(instance_cells)) to find if the instance can be fit to the host. With 16 NUMA nodes host and 8 NUMA node guests this means 500 million calls to _numa_fit_instance_cell. This takes excessive time. However going through these permutations there are many repetitive host_cell, instance_cell pairs to try to fit. E.g. host_cells=[H1, H2, H2] instance_cells=[G1, G2] Produces pairings: * H1 <- G1 and H2 <- G2 * H1 <- G1 and H3 <- G2 ... Here G1 is checked to fit H1 twice. But if it does not fit in the first time then we know that it will not fit in the second time either. So we can cache the result of the first check and use that cache for the later permutations. This patch adds two caches to the algo. A fit_cache to hold host_cell.id, instance_cell.id pairs that we know fit, and a no_fit_cache for those pairs that we already know that doesn't fit. This change significantly boost the performance of the algorithm. The reproduction provided in the bug 1978372 took 6 minutes on my local machine to run without the optimization. With the optimization it run in 3 seconds. This change increase the memory usage of the algorithm with the two caches. Those caches are sets of integer two tuples. And the total size of the cache is the total number of possible host_cell, instance_cell pairs which is len(host_cell) * len(instance_cells). So form the above example (16 host, 8 instance NUMA) it is 128 pairs of integers in the cache. That will not cause a significant memory increase. Closes-Bug: #1978372 Change-Id: Ibcf27d741429a239d13f0404348c61e2668b4ce4
1 parent d869163 commit 099a6f6

File tree

3 files changed

+125
-27
lines changed

3 files changed

+125
-27
lines changed

‎nova/tests/unit/virt/test_hardware.py‎

Lines changed: 84 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -3836,9 +3836,16 @@ def test_host_numa_fit_instance_to_host_single_cell(self):
38363836
siblings=[set([2]), set([3])])
38373837
])
38383838
inst_topo = objects.InstanceNUMATopology(
3839-
cells=[objects.InstanceNUMACell(
3840-
cpuset=set(), pcpuset=set([0, 1]), memory=2048,
3841-
cpu_policy=fields.CPUAllocationPolicy.DEDICATED)])
3839+
cells=[
3840+
objects.InstanceNUMACell(
3841+
id=0,
3842+
cpuset=set(),
3843+
pcpuset=set([0, 1]),
3844+
memory=2048,
3845+
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
3846+
)
3847+
]
3848+
)
38423849

38433850
inst_topo = hw.numa_fit_instance_to_host(host_topo, inst_topo)
38443851

@@ -3867,9 +3874,16 @@ def test_host_numa_fit_instance_to_host_single_cell_w_usage(self):
38673874
siblings=[set([2]), set([3])])
38683875
])
38693876
inst_topo = objects.InstanceNUMATopology(
3870-
cells=[objects.InstanceNUMACell(
3871-
cpuset=set(), pcpuset=set([0, 1]), memory=2048,
3872-
cpu_policy=fields.CPUAllocationPolicy.DEDICATED)])
3877+
cells=[
3878+
objects.InstanceNUMACell(
3879+
id=0,
3880+
cpuset=set(),
3881+
pcpuset=set([0, 1]),
3882+
memory=2048,
3883+
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
3884+
)
3885+
]
3886+
)
38733887

38743888
inst_topo = hw.numa_fit_instance_to_host(host_topo, inst_topo)
38753889

@@ -3898,9 +3912,16 @@ def test_host_numa_fit_instance_to_host_single_cell_fail(self):
38983912
siblings=[set([2]), set([3])])
38993913
])
39003914
inst_topo = objects.InstanceNUMATopology(
3901-
cells=[objects.InstanceNUMACell(
3902-
cpuset=set(), pcpuset=set([0, 1]), memory=2048,
3903-
cpu_policy=fields.CPUAllocationPolicy.DEDICATED)])
3915+
cells=[
3916+
objects.InstanceNUMACell(
3917+
id=0,
3918+
cpuset=set(),
3919+
pcpuset=set([0, 1]),
3920+
memory=2048,
3921+
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
3922+
)
3923+
]
3924+
)
39043925

39053926
inst_topo = hw.numa_fit_instance_to_host(host_topo, inst_topo)
39063927
self.assertIsNone(inst_topo)
@@ -3927,12 +3948,24 @@ def test_host_numa_fit_instance_to_host_fit(self):
39273948
siblings=[set([4]), set([5]), set([6]), set([7])])
39283949
])
39293950
inst_topo = objects.InstanceNUMATopology(
3930-
cells=[objects.InstanceNUMACell(
3931-
cpuset=set(), pcpuset=set([0, 1]), memory=2048,
3932-
cpu_policy=fields.CPUAllocationPolicy.DEDICATED),
3933-
objects.InstanceNUMACell(
3934-
cpuset=set(), pcpuset=set([2, 3]), memory=2048,
3935-
cpu_policy=fields.CPUAllocationPolicy.DEDICATED)])
3951+
cells=[
3952+
objects.InstanceNUMACell(
3953+
id=0,
3954+
cpuset=set(),
3955+
pcpuset=set([0, 1]),
3956+
memory=2048,
3957+
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
3958+
),
3959+
objects.InstanceNUMACell(
3960+
id=1,
3961+
cpuset=set(),
3962+
pcpuset=set([2, 3]),
3963+
memory=2048,
3964+
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
3965+
),
3966+
]
3967+
)
3968+
39363969
inst_topo = hw.numa_fit_instance_to_host(host_topo, inst_topo)
39373970

39383971
for cell in inst_topo.cells:
@@ -3970,12 +4003,24 @@ def test_host_numa_fit_instance_to_host_barely_fit(self):
39704003
])
39714004

39724005
inst_topo = objects.InstanceNUMATopology(
3973-
cells=[objects.InstanceNUMACell(
3974-
cpuset=set(), pcpuset=set([0, 1]), memory=2048,
3975-
cpu_policy=fields.CPUAllocationPolicy.DEDICATED),
3976-
objects.InstanceNUMACell(
3977-
cpuset=set(), pcpuset=set([2, 3]), memory=2048,
3978-
cpu_policy=fields.CPUAllocationPolicy.DEDICATED)])
4006+
cells=[
4007+
objects.InstanceNUMACell(
4008+
id=0,
4009+
cpuset=set(),
4010+
pcpuset=set([0, 1]),
4011+
memory=2048,
4012+
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
4013+
),
4014+
objects.InstanceNUMACell(
4015+
id=1,
4016+
cpuset=set(),
4017+
pcpuset=set([2, 3]),
4018+
memory=2048,
4019+
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
4020+
),
4021+
]
4022+
)
4023+
39794024
inst_topo = hw.numa_fit_instance_to_host(host_topo, inst_topo)
39804025

39814026
for cell in inst_topo.cells:
@@ -4003,12 +4048,24 @@ def test_host_numa_fit_instance_to_host_fail_capacity(self):
40034048
siblings=[set([4]), set([5]), set([6]), set([7])])
40044049
])
40054050
inst_topo = objects.InstanceNUMATopology(
4006-
cells=[objects.InstanceNUMACell(
4007-
cpuset=set(), pcpuset=set([0, 1]), memory=2048,
4008-
cpu_policy=fields.CPUAllocationPolicy.DEDICATED),
4009-
objects.InstanceNUMACell(
4010-
cpuset=set(), pcpuset=set([2, 3]), memory=2048,
4011-
cpu_policy=fields.CPUAllocationPolicy.DEDICATED)])
4051+
cells=[
4052+
objects.InstanceNUMACell(
4053+
id=0,
4054+
cpuset=set(),
4055+
pcpuset=set([0, 1]),
4056+
memory=2048,
4057+
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
4058+
),
4059+
objects.InstanceNUMACell(
4060+
id=1,
4061+
cpuset=set(),
4062+
pcpuset=set([2, 3]),
4063+
memory=2048,
4064+
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
4065+
),
4066+
]
4067+
)
4068+
40124069
inst_topo = hw.numa_fit_instance_to_host(host_topo, inst_topo)
40134070
self.assertIsNone(inst_topo)
40144071

‎nova/virt/hardware.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2357,12 +2357,37 @@ def numa_fit_instance_to_host(
23572357
host_cells,
23582358
key=lambda cell: total_pci_in_cell.get(cell.id, 0))
23592359

2360+
# a set of host_cell.id, instance_cell.id pairs where we already checked
2361+
# that the instance cell does not fit
2362+
not_fit_cache = set()
2363+
# a set of host_cell.id, instance_cell.id pairs where we already checked
2364+
# that the instance cell does fit
2365+
fit_cache = set()
23602366
for host_cell_perm in itertools.permutations(
23612367
host_cells, len(instance_topology)):
23622368
chosen_instance_cells: ty.List['objects.InstanceNUMACell'] = []
23632369
chosen_host_cells: ty.List['objects.NUMACell'] = []
23642370
for host_cell, instance_cell in zip(
23652371
host_cell_perm, instance_topology.cells):
2372+
2373+
cell_pair = (host_cell.id, instance_cell.id)
2374+
2375+
# if we already checked this pair, and they did not fit then no
2376+
# need to check again just move to the next permutation
2377+
if cell_pair in not_fit_cache:
2378+
break
2379+
2380+
# if we already checked this pair, and they fit before that they
2381+
# will fit now too. So no need to check again. Just continue with
2382+
# the next cell pair in the permutation
2383+
if cell_pair in fit_cache:
2384+
chosen_host_cells.append(host_cell)
2385+
# Normally this would have done by _numa_fit_instance_cell
2386+
# but we optimized that out here based on the cache
2387+
instance_cell.id = host_cell.id
2388+
chosen_instance_cells.append(instance_cell)
2389+
continue
2390+
23662391
try:
23672392
cpuset_reserved = 0
23682393
if (instance_topology.emulator_threads_isolated and
@@ -2379,11 +2404,18 @@ def numa_fit_instance_to_host(
23792404
# This exception will been raised if instance cell's
23802405
# custom pagesize is not supported with host cell in
23812406
# _numa_cell_supports_pagesize_request function.
2407+
2408+
# cache the result
2409+
not_fit_cache.add(cell_pair)
23822410
break
23832411
if got_cell is None:
2412+
# cache the result
2413+
not_fit_cache.add(cell_pair)
23842414
break
23852415
chosen_host_cells.append(host_cell)
23862416
chosen_instance_cells.append(got_cell)
2417+
# cache the result
2418+
fit_cache.add(cell_pair)
23872419

23882420
if len(chosen_instance_cells) != len(host_cell_perm):
23892421
continue

‎releasenotes/notes/bug-1978372-optimized-numa-fitting-algorithm-5d5b922b0bdbf818.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
fixes:
3+
- |
4+
The algorithm that is used to see if a multi NUMA guest fits to
5+
a multi NUMA host has been optimized to speed up the decision
6+
on hosts with high number of NUMA nodes ( > 8). For details see
7+
`bug 1978372`_
8+
9+
.. _bug 1978372: https://bugs.launchpad.net/nova/+bug/1978372

0 commit comments

Comments
 (0)