Addressing comments for the mapper tool.

Signed-off-by: L Lakshmanan <s2760012@ed.ac.uk>

Author: KarthikL1729

.github/workflows/tools-tests.yaml

@@ -35,3 +35,47 @@ jobs:
     - name: Run tests
       working-directory: ${{ matrix.module }}
       run: go test -cover -race
+
+  mapper_e2e:
+    name: Mapper E2E Test
+    runs-on: ubuntu-20.04
+    strategy:
+      fail-fast: false
+      matrix:
+        module: [ tools/mapper, ]
+
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v3
+        with:
+          lfs: 'true'
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.9'
+
+      - uses: actions/cache@v4
+        with:
+          path: ${{ env.pythonLocation }}
+          key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('requirements.txt') }}
+
+      - name: Install requirements
+        run: pip install -r ./requirements.txt
+
+      - name: Profile load check
+        run: |
+          tar -xzvf tools/mapper/profile.tar.gz -C tools/mapper/
+          python3 -c "import json; json.load(open('tools/mapper/profile.json'))"
+  
+      - name: Test traces load check
+        run: |
+          tar -xzvf tools/mapper/test_files/trace.tar.gz -C tools/mapper/test_files/
+          python3 tools/mapper/trace_load_test.py -t tools/mapper/test_files/extremes/
+
+      - name: Extreme mapping tests
+        run: |
+          python3 mapper.py -t tools/mapper/test_files/extremes -p tools/mapper/profile.json
+
+      - name: Run mapper tool on sample trace
+        run: |
+          python3 mapper.py -t tools/mapper/test_files/20 -p tools/mapper/profile.json

docs/.gitattributes

@@ -7,7 +7,7 @@ The `profile.json` JSON output file is generated by the [`profiler` tool](https:
 ### Usage
 
 ```bash
-usage: mapper.py [-h] -t TRACE_DIRECTORYPATH -p PROFILE_FILEPATH [-o OUTPUT_FILEPATH] [-u UNIQUE_ASSIGNMENT]
+usage: mapper.py [-h] -t TRACE_DIRECTORYPATH -p PROFILE_FILEPATH
 
 Arguments:
   -h, --help            show this help message and exit
@@ -60,7 +60,7 @@ The [`sampler`](https://github.com/vhive-serverless/invitro/tree/main/sampler) t
 
 For every function in the trace, the closest function in the [`vSwarm`](https://github.com/vhive-serverless/vSwarm/tree/main/) benchmark suite is set as its proxy (50-percentile memory and 50-percentile duration are considered to find the highest correlation). The 50th percentile is used to ensure that the mapping is not only corresponding to the peak values of the workload, but is also leading to a representative proxy function. Currently the tool utilizes only _Serving Functions_ that are _NOT Pipelined_ as proxy functions.
 
-Currently, vSwarm does not have full coverage of the Azure trace functions. To make sure that we do not have a high error in the mapping of functions, we set a hard threshold of 40% as the maximum absolute error (from the actual trace function duration) that a proxy function can have to be mapped to a trace function. If we do not find an eligible function for the mapping from vSwarm, we fall back to using standard InVitro sample functions in their place for those functions alone.
+Currently, vSwarm does not have full coverage of the Azure trace functions. To make sure that we do not have a high error in the mapping of functions, we set a hard threshold of 40% as the maximum absolute error (from the actual trace function duration) that a proxy function can have to be mapped to a trace function. If we do not find an eligible function for the mapping from vSwarm, we fall back to using standard InVitro trace functions in their place for those functions alone.
 
 This mapping requires the profiles of the benchmark functions for it to be used as a proxy. The tool utilizes the `profile.json` JSON output file generated by the [`profiler` tool](https://github.com/vhive-serverless/vSwarm/tree/load-generator/tools/profiler#profiler) to obtain the profile of the benchmark suite functions. The User can configure the path of the JSON file through the `-p` (or `--profile-filepath`) flag (by default, it is `profile.json`, which needs to be unzipped).
 
@@ -121,15 +121,18 @@ The dropped functions plot is as shown below:
 
 Beyond this, we also display the following error metrics whenever the mapper is run with a trace:
 
-- Average memory error
-- Average duration error
-- Average absolute memory error
-- Average absolute duration error
-- Average relative memory error
-- Average relative duration error
-- Average absolute relative memory error
-- Average absolute relative duration error
-- Functions in the trace with 0 duration
-- Number of mapped functions with higher than 40% duration error (replaced by InVitro sample functions)
+
+| Metric | Value |
+| --- | --- |
+| Average memory error | -7.638341413565764 MB per invocation |
+| Average duration error | 4174.5554028958695 ms per invocation |
+| Average absolute memory error | 24.24782794856284 MB per invocation |
+| Average absolute duration error | 4414.451828203135 ms per invocation |
+| Average relative memory error | -0.8412999109296387 |
+| Average relative duration error | 0.004934168605729668 |
+| Average absolute relative memory error | 1.0028566557523266 |
+| Average absolute relative duration error | 0.20141343497568448 |
+| Functions with 0 duration | 1596 |
+| Number of mapped functions with higher than 40% duration error (replaced by InVitro trace functions) | 5258 |
 
 ---

docs/WD.png

@@ -1,9 +1,5 @@
-import numpy as np
-import scipy.optimize as sp
 import math
-
 from log_config import *
-from typing import Tuple
 
 def get_error(trace_function, proxy_function) -> float:
     """
@@ -44,7 +40,7 @@ def get_error(trace_function, proxy_function) -> float:
 
 def get_closest_proxy_function(
     trace_functions: dict, proxy_functions: dict
-) -> Tuple[dict, int]:
+) -> dict:
     """
     Obtains the closest proxy function for every trace function
     
@@ -57,73 +53,32 @@ def get_closest_proxy_function(
     - `int`: 0 if no error. -1 if error
     """
 
-    try:
-        proxy_list = []
-        for function_name in proxy_functions:
-            proxy_list.append(proxy_functions[function_name])
-            proxy_functions[function_name]["index"] = len(proxy_list) - 1
-
-        for function_name in trace_functions:
-            min_error = math.inf
-            min_error_index = -1
-            for i in range(0, len(proxy_list)):
-                error = get_error(trace_functions[function_name], proxy_list[i])
-                if error < min_error:
-                    min_error = error
-                    min_error_index = i
-
-            if min_error == math.inf:
-                log.warning(f"Proxy function for function {function_name} not found")
-                continue
-
-            trace_functions[function_name]["proxy-function"] = proxy_list[
-                min_error_index
-            ]["name"]
-            trace_functions[function_name]["proxy-correlation"] = get_error(
-                trace_functions[function_name], proxy_list[min_error_index]
-            )
-            log.debug(
-                f"Found proxy function for {function_name}: {trace_functions[function_name]['proxy-function']} with correlation: {trace_functions[function_name]['proxy-correlation']}"
-            )
-
-        for function_name in proxy_functions:
-            del proxy_functions[function_name]["index"]
-
-        return trace_functions, 0
-
-    except Exception as e:
-        log.error(f"Finding closest proxy function failed. Error: {e}")
-        return trace_functions, -1
-
-
-def get_proxy_function(
-    trace_functions: dict, proxy_functions: dict
-) -> Tuple[dict, int]:
-    """
-    Obtains the closest proxy function for every trace function
-    
-    Parameters:
-    - `trace_functions` (dict): Dictionary containing information regarding trace functions
-    - `proxy_functions` (dict): Dictionary containing information regarding proxy functions
+    proxy_list = []
+    for function_name in proxy_functions:
+        proxy_list.append(proxy_functions[function_name])
+        proxy_functions[function_name]["index"] = len(proxy_list) - 1
+
+    for function_name in trace_functions:
+        min_error = math.inf
+        min_error_index = -1
+        for i in range(0, len(proxy_list)):
+            error = get_error(trace_functions[function_name], proxy_list[i])
+            if error < min_error:
+                min_error = error
+                min_error_index = i
+
+        if min_error == math.inf:
+            log.warning(f"Proxy function for function {function_name} not found. Using InVitro trace function.")
+            trace_functions[function_name]["proxy-function"] = "trace-func-go"
+            continue
+
+        trace_functions[function_name]["proxy-function"] = proxy_list[
+            min_error_index
+        ]["name"]
+
+    for function_name in proxy_functions:
+        del proxy_functions[function_name]["index"]
 
-    Returns:
-    - `dict`: Dictionary containing information regarding trace functions with the associated proxy functions
-    - `int`: 0 if no error. -1 if error
-    """
-
-    log.info(
-        f"Lower the correlation value, the proxy function is a better proxy of the trace function"
-    )
-
-    log.info(
-        f"Getting closest proxy function for every trace function."
-    )
-    trace_functions, err = get_closest_proxy_function(
-        trace_functions=trace_functions, proxy_functions=proxy_functions
-    )
-
-    if err == -1:
-        log.critical(f"Mapping between trace function and proxy function not obtained")
-        return trace_functions, -1
+    log.info("Proxy functions found for all trace functions.")
 
-    return trace_functions, 0
+    return trace_functions

docs/WD_dropped_data.png

@@ -4,12 +4,12 @@
 import re
 import argparse
 import pandas as pd
-import matplotlib.pyplot as plt
 from find_proxy_function import *
 
 from log_config import *
 
 INVOCATION_COLUMN = 4
+VSWARM_MAX_DUR = 27000
 
 def load_trace(trace_directorypath):
     duration_info = {}
@@ -27,6 +27,9 @@ def load_trace(trace_directorypath):
                 f"Durations file {duration_filepath} cannot be read. Error: {e}"
             )
             return None, -1
+    else:
+        log.critical(f"Durations file {duration_filepath} not found")
+        return None, -1
 
     # Read the memory file
     memory_filepath = trace_directorypath + "/memory.csv"
@@ -39,6 +42,9 @@ def load_trace(trace_directorypath):
                 f"Memory file {memory_filepath} cannot be read. Error: {e}"
             )
             return None, -1
+    else:
+        log.critical(f"Memory file {memory_filepath} not found")
+        return None, -1
 
     # Rename all columns in the dataframe with a lambda (for example: percentile_Average_1 -> 1-percentile) if x matches a regex
     duration_info = duration_info.rename(columns=lambda x: re.sub(r'percentile_(\w+)_(\d+)', r'\2-' + "percentile", x))
@@ -59,52 +65,9 @@ def load_trace(trace_directorypath):
 
     return trace_functions, 0
 
-def generate_plot(trace_directorypath, profile_filepath, output_filepath, invoke=True):
-    trace_functions, err = load_trace(trace_directorypath)
-    if err == -1:
-        log.critical(f"Load Generation failed")
-        return
-    elif err == 0:
-        log.info(f"Trace loaded")
-
-    ## Check whether the profile file for proxy functions exists or not
-    if os.path.exists(profile_filepath):
-        log.info(
-            f"Profile file for proxy functions {profile_filepath} exists. Accessing information"
-        )
-        try:
-            with open(profile_filepath, "r") as jf:
-                proxy_functions = json.load(jf)
-        except Exception as e:
-            log.critical(
-                f"Profile file for proxy functions {profile_filepath} cannot be read. Error: {e}"
-            )
-            log.critical(f"Load Generation failed")
-            return
-    else:
-        log.critical(f"Profile file for proxy functions {profile_filepath} not found")
-        log.critical(f"Load Generation failed")
-        return
-    
-    if os.path.exists(output_filepath):
-        log.info(
-            f"Mapper output file for trace functions {output_filepath} exists. Accessing information"
-        )
-        try:
-            with open(output_filepath, "r") as jf:
-                mapped_traces = json.load(jf)
-        except Exception as e:
-            log.critical(
-                f"Mapper output file for trace functions {output_filepath} cannot be read. Error: {e}"
-            )
-            log.critical(f"Load Generation failed")
-            return
-    else:
-        log.critical(f"Mapper output file for trace functions {output_filepath} not found")
-        log.critical(f"Load Generation failed")
-        return
+def generate_plot(trace_directorypath, trace_functions, proxy_functions, mapped_trace, invocation_statistics=True):
 
-    if invoke:
+    if invocation_statistics:
         invocations = pd.read_csv(trace_directorypath+"/invocations.csv")
         inv_df = {}
         for i in range(len(invocations)):
@@ -115,12 +78,12 @@ def generate_plot(trace_directorypath, profile_filepath, output_filepath, invoke
         dropped_functions, dropped_invocations, total_invocations = 0, 0, 0
         for trace in trace_functions:
             duration = trace_functions[trace]["duration"]["50-percentile"]
-            if duration > 27000:
+            if duration > VSWARM_MAX_DUR:
                 dropped_functions += 1
                 dropped_invocations += inv_df[trace]
                 continue
             total_invocations += inv_df[trace]
-            proxy_name = mapped_traces[trace]["proxy-function"]
+            proxy_name = mapped_trace[trace]["proxy-function"]
             profile_duration = proxy_functions[proxy_name]["duration"]["50-percentile"]
             trace_durations.append(duration)
             mapped_durations.append(profile_duration)
@@ -130,9 +93,9 @@ def generate_plot(trace_directorypath, profile_filepath, output_filepath, invoke
         mapped_durations = []
         for trace in trace_functions:
             duration = trace_functions[trace]["duration"]["50-percentile"]
-            if duration > 27000:
+            if duration > VSWARM_MAX_DUR:
                 continue
-            proxy_name = mapped_traces[trace]["proxy-function"]
+            proxy_name = mapped_trace[trace]["proxy-function"]
             profile_duration = proxy_functions[proxy_name]["duration"]["50-percentile"]
             trace_durations.append(duration)
             mapped_durations.append(profile_duration)
@@ -161,7 +124,7 @@ def main():
     output_filepath = trace_directorypath + "/mapper_output.json"
     trace_functions, err = load_trace(trace_directorypath)
     if err == -1:
-        log.critical(f"Load Generation failed")
+        log.critical(f"Trace loading failed")
         return
     elif err == 0:
         log.info(f"Trace loaded")
@@ -186,15 +149,12 @@ def main():
         return
 
     # Getting a proxy function for every trace function
-    trace_functions, err = get_proxy_function(
+    trace_functions = get_closest_proxy_function(
         trace_functions=trace_functions,
         proxy_functions=proxy_functions,
     )
-    if err == -1:
-        log.critical(f"Load Generation failed")
-        return
-    elif err == 0:
-        log.info(f"Proxy functions obtained")
+    
+    log.info(f"Proxy functions obtained")
 
     # Writing the proxy functions to a file
 
@@ -240,12 +200,10 @@ def main():
         trace_dur = trace_functions[function]["duration"]["50-percentile"]
         proxy_dur = proxy_functions[mapper_output[function]["proxy-function"]]["duration"]["50-percentile"]
         proxy_mem = proxy_functions[mapper_output[function]["proxy-function"]]["memory"]["50-percentile"]
-        log.warning(f"Memory error for function {function} is {abs(trace_mem - proxy_mem)} MB per invocation")
         mem_error += trace_mem - proxy_mem
         rel_mem_error += (trace_mem - proxy_mem)/trace_mem
         abs_mem_error += abs(trace_mem - proxy_mem)
         abs_rel_mem_error += abs((trace_mem - proxy_mem)/trace_mem)
-        log.warning(f"Duration error for function {function} is {abs(trace_dur - proxy_dur)}ms per invocation")
         dur_error += trace_dur - proxy_dur
         abs_dur_error += abs(trace_dur - proxy_dur)
         if trace_dur == 0: