Increase timeout to 80s. Backoff poll frequency.

Timeout was 60s, increase to 80s. This may help on heavily loaded
storage environments.

We see ~60 debug messages for each timed out wait. This is more than
needed and puts a bit more load on the control plane than needed.
So change the approach: Start with 0.5s polling freq but increase
wait time by 0.1s each time, slowly backing off. So we produce only
35 debug lines and API calls before running into the timeout.

Sidenote: Exponential backoff is a well-known approach to deal
with congestion. We kept it simple (linear back-off), as our timeout
is not huge.

Signed-off-by: Kurt Garloff <[email protected]>

Author: garloff

Tests/iaas/volume-backup/volume-backup-tester.py

@@ -29,7 +29,7 @@
 
 # timeout in seconds for resource availability checks
 # (e.g. a volume becoming available)
-WAIT_TIMEOUT = 60
+WAIT_TIMEOUT = 80
 
 
 def wait_for_resource(
@@ -39,10 +39,12 @@ def wait_for_resource(
     timeout=WAIT_TIMEOUT,
 ) -> None:
     seconds_waited = 0
+    wait_delay = 0.5
     resource = get_func(resource_id)
     while resource is None or resource.status not in expected_status:
-        time.sleep(1.0)
-        seconds_waited += 1
+        time.sleep(wait_delay)
+        seconds_waited += wait_delay
+        wait_delay += 0.1
         if seconds_waited >= timeout:
             raise RuntimeError(
                 f"Timed out after {seconds_waited} s: waiting for resource {resource_id} "