contest: loadavg: avoid false negatives when waiting for idle

kunit is sensitive to overload. We currently ungate it too early,
we seem to kick off before other workers start. Wait a few cycles
to confirm the machine is actually idle.

Signed-off-by: Jakub Kicinski <[email protected]>

Author: kuba-moo

contest/remote/lib/loadavg.py

@@ -4,12 +4,21 @@
 import time
 
 
-def wait_loadavg(target, check_ival=30):
+def wait_loadavg(target, check_ival=30, stable_cnt=4):
+    """
+    Wait for loadavg to drop but be careful at the start, the load
+    may have not ramped up, yet, so if we ungate early whoever is waiting
+    will experience the overload.
+    """
     while target is not None:
         load, _, _ = os.getloadavg()
 
         if load <= target:
-            break
+            if stable_cnt == 0:
+                break
+            stable_cnt -= 1
+        else:
+            stable_cnt = 0
 
-        print(f"Waiting for loadavg to decrease: {load} > {target}")
+        print(f"Waiting for loadavg to decrease: {load} > {target} ({stable_cnt})")
         time.sleep(check_ival)