testscript: improve wait_for_ssh() network diagnosis and logging

phip1611 · phip1611 · commit 9f1ebad72c08 · 2026-01-15T18:41:44.000+01:00
This commit uses the new helpers in wait_for_ssh() to improve network
diagnosis and logging in the case of an error. We will be able to better
understand if a network issue is caused by the VM host or the VM.

More specifically, we will specifically see if a network issue is
caused by the host losing an IP on one of its interfaces for whatever
reason.

wait_for_ssh() now:
- checks if the host shares the same IPv4 net with the IP
- if the VM is pingable
- then tries to SSH into it

A possible missing IP in VM host error might be reported like this:

```
======================================================================
ERROR: test_hotplug (builtins.LibvirtTests.test_hotplug)
Tests device hot plugging with multiple devices of different types:
----------------------------------------------------------------------
Traceback (most recent call last):
  File "&lt;string&gt;", line 391, in test_hotplug
  File "&lt;string&gt;", line 2000, in wait_for_ssh
  File "&lt;string&gt;", line 2285, in assert_ip_in_local_192_168_net24
RuntimeError: The VM host is not in the same network as IP 192.168.5.2! It may has lost a IP?!
networks: [IPv4Network('192.168.1.0/24'), IPv4Network('192.168.2.0/24'), IPv4Network('192.168.3.0/24'), IPv4Network('192.168.4.0/24'), IPv4Network('192.168.100.0/24')]
`ip a`:1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    altname enx525400123456
    inet 10.0.2.15/24 metric 1024 brd 10.0.2.255 scope global dynamic eth0
       valid_lft 86360sec preferred_lft 86360sec
    inet6 fec0::b396:b54d:c532:85b8/64 scope site temporary dynamic
       valid_lft 86363sec preferred_lft 14363sec
    inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute
       valid_lft 86363sec preferred_lft 14363sec
    inet6 fe80::5054:ff:fe12:3456/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: eth1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:01:02 brd ff:ff:ff:ff:ff:ff
    altname enp0s9
    altname ens9
    altname enx525400120102
    inet 192.168.100.1/24 brd 192.168.100.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe12:102/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
4: br0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 22:0b:45:80:ae:68 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet 192.168.2.1/24 brd 192.168.2.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::200b:45ff:fe80:ae68/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
5: br4: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ea:27:4a:87:ff:89 brd ff:ff:ff:ff:ff:ff
    inet 192.168.4.1/24 brd 192.168.4.255 scope global br4
       valid_lft forever preferred_lft forever
    inet6 fe80::e827:4aff:fe87:ff89/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
6: br3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc htb state UP group default qlen 1000
    link/ether 52:54:00:be:b0:2c brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.1/24 brd 192.168.3.255 scope global br3
       valid_lft forever preferred_lft forever
7: vtap0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:e5:b8:01 brd ff:ff:ff:ff:ff:ff
8: vtap1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:e5:b8:02 brd ff:ff:ff:ff:ff:ff
9: tap3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br3 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:e5:b8:03 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fee5:b803/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
10: tap4: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br4 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:e5:b8:04 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fee5:b804/64 scope link proto kernel_ll
       valid_lft forever preferred_lft foreve
```

On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster &lt;philipp.schuster@cyberus-technology.de&gt;
diff --git a/tests/testscript.py b/tests/testscript.py
@@ -1983,25 +1983,34 @@ def wait_for_ping(machine: Machine, ip="192.168.1.2"):
 
 def wait_for_ssh(machine: Machine, user="root", password="root", ip="192.168.1.2"):
     """
-    Waits for SSH to become available to connect into the Cloud Hypervisor VM
-    hosted on the corresponding machine.
+    Waits for the VM to become accessible via SSH.
 
-    Effectively we use it to wait until the Cloud Hypervisor VM's network is up
-    and available.
+    It first checks whether the VM responds to ping, and then attempts to
+    establish an SSH connection using the provided credentials.
 
     :param machine: VM host
     :param user: user for SSH login
     :param password: password for SSH login
     :param ip: SSH host to log into
     """
     retries = 100
+
+    # Sometimes we experienced test runs where the host lost IPs. We therefore
+    # check that early and always for better debuggability.
+    assert_ip_in_local_192_168_net24(machine, ip)
+    wait_for_ping(machine, ip)
+
+    print(f"Waiting for ssh connection into VM with IP {ip} ...")
     for i in range(retries):
-        print(f"Wait for ssh {i}/{retries}")
+        print(f"Wait for ssh ({i + 1}/{retries}) ...")
         status, _ = ssh(machine, "echo hello", user, password, ip)
         if status == 0:
             return
         time.sleep(0.1)
-    raise RuntimeError(f"Could not establish SSH connection to {ip}")
+
+    raise RuntimeError(
+        f"Could not establish SSH connection to {ip} after {retries} attempts"
+    )
 
 
 def ssh(machine: Machine, cmd, user="root", password="root", ip="192.168.1.2"):
