fix: do not error during pytest teardown after test failure

During pytest teardown, we explicitly kill all microvms that a test
might have left behind. We do this by reading Firecracker's pid file and
then SIGKILL-ing the process. However, if a test fails, we also try to
upload its entire jail to S3 as buildkite artifacts, for post-mortem
analysis. Here, we are moving instead of copying files, to save on disk
space. This is a problem though, as the pid file is inside the jail, and
this moving happens _before_ the call to .kill() (and has to happen
before, as killing a VM cleans up its jail). This leads to duplicate
pytest errors every time a test fails: One for the actual failure, and
then a second failure from Microvm.kill() due to the pid file not being
found.

Fix this using a heuristic of when to copy vs move jail contents: We
only ran into problem with snapshot files, which are always at lesat
128MB in size (minimal microvm size), and hence only move file greater
than this threshold.

Signed-off-by: Patrick Roy <[email protected]>

Author: roypat

tests/conftest.py

@@ -350,7 +350,16 @@ def microvm_factory(request, record_property, results_dir, netns_factory):
                 if not os.path.isfile(src):
                     continue
                 dst = uvm_data / item
-                shutil.move(src, dst)
+                # This optimization of moving instead of copying it mainly intended for
+                # snapshot files, which in tests that generate snapshots in a loop
+                # can quickly fill up /srv. However, small files are fine to duplicate,
+                # and in fact mustn't be moved before the call to uvm_factory.kill(),
+                # as they are needed during teardown (e.g. firecracker.pid file is
+                # needed by Microvm.kill()).
+                if src.stat().st_size >= 128 * 1024 * 1024:
+                    shutil.move(src, dst)
+                else:
+                    shutil.copy(src, dst)
                 console_data = uvm.console_data
                 if console_data:
                     uvm_data.joinpath("guest-console.log").write_text(console_data)