test: ab: Add function for A/B-Tests across precompiled binaries

Currently, we have two types of pre-PR A/B-Tests: Those that depend on
the repository as a whole (e.g. `cargo audit`, which checks properties
of Cargo.toml), and those that depend on the pre-compiled firecracker
binaries.

At the moment, these tests can use the same infrastructure, because our
buildkite shared build step will pre-compile binaries and then upload
the entire repository as an artifact, and so both types of A/B-tests
will find what they're looking for in this artifact and end up being
happy. However, this model makes it very difficult to manually run
A/B-tests locally (one will need to setup clones of the git repository
at specific locations and then pre-compile firecracker with specific
flags inside them). This patch series has the goal to simplify these
manual A/B-tests, but the cost is that the pre-PR A/B tests will require
different infrastructure.

Thus, prepare by providing different functions for each of these types
of A/B-test to use.

Signed-off-by: Patrick Roy <[email protected]>

Author: roypat

tests/framework/ab_test.py

@@ -29,10 +29,15 @@
 import scipy
 
 from framework import utils
+from framework.defs import FC_WORKSPACE_DIR
 from framework.microvm import Microvm
 from framework.utils import CommandReturn
 from framework.with_filelock import with_filelock
-from host_tools.cargo_build import get_binary, get_firecracker_binaries
+from host_tools.cargo_build import (
+    DEFAULT_TARGET_DIR,
+    get_binary,
+    get_firecracker_binaries,
+)
 
 # Locally, this will always compare against main, even if we try to merge into, say, a feature branch.
 # We might want to do a more sophisticated way to determine a "parent" branch here.
@@ -96,6 +101,26 @@ def git_ab_test(
     return result_a, result_b, comparison
 
 
+DEFAULT_B_DIRECTORY = FC_WORKSPACE_DIR / "build" / "cargo_target" / DEFAULT_TARGET_DIR
+
+
+def binary_ab_test(
+    test_runner: Callable[[Path, bool], T],
+    comparator: Callable[[T, T], U] = default_comparator,
+    *,
+    a_directory: Path,
+    b_directory: Path = DEFAULT_B_DIRECTORY,
+):
+    """
+    Similar to `git_ab_test`, but instead of locally checking out different revisions, it operates on
+    directories containing firecracker/jailer binaries
+    """
+    result_a = test_runner(a_directory, True)
+    result_b = test_runner(b_directory, False)
+
+    return result_a, result_b, comparator(result_a, result_b)
+
+
 def is_pr() -> bool:
     """Returns `True` iff we are executing in the context of a build kite run on a pull request"""
     return os.environ.get("BUILDKITE_PULL_REQUEST", "false") != "false"