Merge pull request #4 from powerapi-ng/add-perf-benchmarking

Adds perf alone, hwpc alone and perf+hwpc event scrapping over synthetic runs of stress-ng

Author: Inkedstinct

.env.example

@@ -0,0 +1,4 @@
+G5K_USERNAME="TO_BE_DEFINED"
+G5K_PASSWORD="TO_BE_DEFINED"
+DOCKER_HUB_USERNAME="TO_BE_DEFINED"
+DOCKER_HUB_TOKEN="TO_BE_DEFINED"

.gitignore

@@ -0,0 +1,17 @@
+/logs.d/**/*
+.env
+/Cargo.lock
+/tasks.d/**/*
+/jobs.d/**/*
+/results.d/**/*
+/scripts.d/**/*
+/inventories.d/**/*
+/backup/**/*
+/test_results/**/*
+/jobs.yaml
+/menage.sh
+
+
+# Added by cargo
+
+/target

Cargo.toml

@@ -0,0 +1,29 @@
+[package]
+name = "benchmarking"
+version = "0.1.0"
+edition = "2021"
+
+[dependencies]
+log = "0.4"
+env_logger = "0.10"
+chrono = "0.4.38"
+reqwest = { version = "0.12.9", features = ["json"] }
+serde = { version = "1.0.214", features = ["derive"] }
+thiserror = "1.0.65"
+serde_json = "1.0.132"
+openssl = { version = "0.10.68", features = ["vendored"] }
+dotenv = "0.15.0"
+tokio = { version = "1.41.0", features = ["full"] }
+mockito = "1.5.0"
+ssh2 = "0.9.4"
+serde_yaml = "0.9.34"
+openssh = { version = "0.11.2", features = ["native-mux"] }
+openssh-sftp-client = { version = "0.15.1", features = ["openssh"] }
+rand = "0.8.5"
+regex = "1.11.1"
+askama = { version = "0.12.1", features = ["serde", "serde-json", "serde-yaml", "serde_json", "serde_yaml"] }
+derive_more = { version = "1.0.0", features = ["display"] }
+flate2 = { version = "1.0.34", features = ["zlib"] }
+tar = "0.4.43"
+bytes = "1.8.0"
+subprocess = "0.2.9"

HYPOTHESES.md

@@ -0,0 +1,34 @@
+# Hypotheses
+
+This document tracks hypotheses to be validated in the benchmarking process, organized by **Tested** and **To Be Tested** sections. Each hypothesis includes the rationale and expected outcome.
+
+## To Be Tested
+
+1. **Statistical Sufficiency of 30 Iterations**
+    - **Hypothesis**: Running 30 iterations per measurement will yield sufficient data for statistical analysis of energy results, ensuring reliability in evaluating variability.
+    - **Expected Outcome**: Consistent patterns emerge within 30 iterations, showing clear trends in energy measurements without requiring additional runs.
+    - **Rationale**: 30 iterations are often a threshold for normal distribution in statistical sampling.
+
+2. **Validity of `NB_CPU_OPS` Values `[10^7, 10^8, 10^9]`**
+    - **Hypothesis**: Using `NB_CPU_OPS` values of `[10^7, 10^8, 10^9]` allows all measurements to complete within the allocated G5K reservation time.
+    - **Expected Outcome**: These values will yield consistent, complete data sets without running into time constraints.
+    - **Rationale**: Choosing `NB_CPU_OPS` values that finish within the reservation time maximizes efficiency and resource usage while keeping results comprehensive.
+
+3. **Post-Mortem Mode Accuracy for SmartWatts**
+    - **Hypothesis**: Running SmartWatts in post-mortem mode will still produce accurate energy measurement results, suitable for evaluating the precision of PowerAPI tools.
+    - **Expected Outcome**: SmartWatts post-mortem results closely match real-time energy data.
+    - **Rationale**: Save another set of stress-ng would be intesting. No proof of difference in precision have been demonstrated between CSV and MongoDB output mode for HWPC Sensor.
+
+4. **Feasibility of Storing Results on NFS**
+    - **Hypothesis**: The average size of each aggregated results file will be under 3.32 MB, allowing for efficient storage within the 25GB per site NFS limit.
+    - **Expected Outcome**: Average file size per node stays within the 3.32 MB limit, avoiding storage issues.
+    - **Rationale**: Storage constraints on NFS necessitate a size estimation for scalability.
+
+5. **Parallel Execution of HWPC Sensor & Perf**
+    - **Hypothesis**: HWPC Sensor and `perf` measurements can run concurrently on the same `stress-ng` process without interference or measurement degradation.
+    - **Expected Outcome**: Both HWPC and `perf` output valid data when run in parallel, with no notable interference.
+    - **Rationale**: Efficient benchmarking may require concurrent measurements to maximize data collection without extending reservation times.
+
+---
+
+## Tested Hypotheses

README.md

@@ -1,3 +1,83 @@
+# PowerAPI Framework Benchmarks
+
+Here’s how the **"What it does"** section could look based on the content you provided:
+
+---
+
+## What It Does
+
+This repository contains the source code for generating and running benchmarks for the **PowerAPI Framework**. These benchmarks are designed to adapt to the configuration and architecture of underlying nodes in the **Grid5000** infrastructure. 
+
+### Key Processes:
+
+1. **Gather Node Information**: The benchmarks start by scraping the Grid5000 API to collect details about all available nodes.
+2. **Reuse Existing Job List (Optional)**: If a `jobs.yaml` file exists, the tool can leverage it to initialize the job list.
+3. **Generate Bash Scripts**: For each filtered node, a custom bash script is generated using templates located in the `/templates` directory.
+4. **Submit Jobs via OAR**: The generated scripts are submitted to corresponding nodes through SSH using **OAR**, ensuring that no more than `N` jobs are simultaneously active.
+5. **Monitor and Collect Results**:
+    - The status of each submitted job is tracked until it completes (either successfully or in a failed state).
+    - Upon completion, **rsync** is used to retrieve the results files locally. If the retrieval fails, the job’s state is marked as `UnknownState` for manual review.
+6. **Store Results**: Once all filtered nodes have completed their benchmark jobs, the benchmarking process concludes, and all result files are stored in the `/results.d` directory.
+
+This automated workflow ensures efficient and scalable benchmarking tailored to the dynamic nature of the Grid5000 environment.
+
+
+## Why it exists.
+
+This benchmarks aim at measuring the variability introduced by the PowerAPI framework over the RAPL interface.
+
+## Who it’s for.
+Currently, this work remains internal for PowerAPI staff to study the variability of the PowerAPI framework along its development.
+
+
+## Installation
+
+To use this repository, you need to clone it locally, ensure you have Cargo installed, and then compile and run the project. Follow the steps below:
+
+### Prerequisites
+
+Before proceeding, make sure your system meets the following requirements:
+
+- **Rust and Cargo**: Install Rust (which includes Cargo, Rust’s package manager and build system). If Rust is not installed, follow the instructions below:
+    1. Download and install Rust by running:
+     ```bash
+     curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+     ```
+    2. Follow the on-screen instructions to complete the installation.
+    3. Add Cargo to your PATH (usually done by Rust installer automatically). Restart your terminal if necessary.
+
+- **Dependencies**:
+  - A working installation of **OAR** on your Grid5000 node (or appropriate access).
+  - SSH access configured to interact with the Grid5000 nodes (for example, `ssh rennes` ran locally shall connect you to the rennes' frontend).
+
+### Clone the Repository
+
+Clone this repository to your local machine:
+
+```bash
+git clone https://github.com/powerapi-ng/benchmarking.git
+cd benchmarking
+```
+
+### Build the Project
+
+Compile the project using Cargo:
+
+```bash
+cargo build --release
+```
+
+This will produce an optimized executable located in the `target/release/` directory.
+
+### Run the Project
+
+Execute the compiled program:
+
+```bash
+./target/release/benchmarking
+```
+
+
 # Tips G5k
 
 - To execute a script on a given list of servers (chifflot) during 4 hours max: 
@@ -20,4 +100,3 @@ g5k-setup-docker -t
 
 - To check reservations at Lille site (with authentification): 
 
-https://intranet.grid5000.fr/oar/Lille/monika.cgi