Add regression performance scripts

Author: bakpaul

.github/workflows/ci.yml

@@ -0,0 +1,64 @@
+name: Launch the performance test
+
+
+permissions:
+  contents: write
+on:
+  workflow_dispatch:
+    inputs:
+        branch:
+          description: 'Branch to test'
+          required: true
+          default: 'master'
+          type: string
+        commit_hash:
+          description: 'Commit hash on the branch, if not specified thent he latest is taken'
+          required: false
+          type: string
+  repository_dispatch:
+
+
+jobs:
+  run:
+    runs-on: self-hosted
+
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          path: ./PerformanceRegression
+      
+      - name: Setup environment
+        shell: bash
+        run: |
+          if [[ "${{ github.event_name}}" == "workflow_dispatch" ]]; then 
+            echo "TEST_BRANCH=${{ inputs.branch }}" | tee -a $GITHUB_ENV
+            echo "TEST_COMMIT_HASH=${{ inputs.commit_hash }}" | tee -a $GITHUB_ENV
+          else
+            echo "TEST_BRANCH=${{ github.event.client_payload.branch }}" | tee -a $GITHUB_ENV
+            echo "TEST_COMMIT_HASH=${{ github.event.client_payload.commit_hash }}" | tee -a $GITHUB_ENV
+          fi
+
+      - name: Run performance tests
+        id: run_perf_tests
+        shell: bash
+        run: |
+          export RO_GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}
+          bash ./PerformanceRegression/main.sh $(pwd) $TEST_BRANCH $TEST_COMMIT_HASH
+
+      - name: Upload artifacts
+        uses: actions/[email protected]
+        with:
+          name: ${{ inputs.branch }}_${{ inputs.commit_hash }}
+          path: logs
+
+      - name: Push results
+        if: steps.run_perf_tests.outcome == 'success'
+        run: |
+          cd PerformanceRegression
+          git config user.name github-actions
+          git config user.email [email protected]
+          git add old_results/*
+          git commit -m "Add performance stats for $TEST_BRANCH $TEST_COMMIT_HASH"
+          git pull -r
+          git push
+      

.gitignore

@@ -0,0 +1,3 @@
+.idea/
+
+*__pycache__/

README.md

@@ -0,0 +1,70 @@
+# Performance regression testing for SOFA
+
+This repository is meant for testing computatioin speed of SOFA Framework. It is constituted of two parts :
+1. Scripts to generate computation statistics
+2. Script to display the results in a dashboard
+
+The repository serves also as an archive because all te generated results are saved in the folder `old_results` by the action
+
+This plugin has been built to work on Ubuntu, but it might work properly on other OS, it just hasn't been tested.
+
+## Use the action
+
+This script is meant to be launched on a physical self-hosted runner to make sure the running conditions are reproducible. To launch the statistic generation, a workflow-dispatch is available.
+Inputs are simple:
+1. Sofa branch name (master by default)
+2. Commit id (empty by default) --> if none given the tip of the branch is taken
+
+The action is meant to work only with master and releases branches.
+The action will then perform the statistic geenration and push them on the repository.
+
+
+
+## Use on your own computer
+
+### Statistics generation
+
+#### Dependencies
+If you want to run the statistic generation on your own computer, those are the dependencies :
+- **SOFA:** Dependencies to build SOFA without openGl, Qt, and no plugin. See https://sofa-framework.github.io/doc/getting-started/build/linux/
+- **Tracy:** a compiled version of the tracy-cvsexporter and tracy-capture. Make sure they are the same version of the one used by SOFA and directly accessible (add their directory to your path). If the one from SOFA changes, it might be good to dump the version in the patches (see branch-dependency/git-patches) to be able to still build and test old releases.
+- **ccache:** By default ccache is used, it is recommended to have it installed
+- **Python3:** Version 3.12 + PyGithub
+- **inxi:** to dump hardware info
+
+
+#### How to
+The statistics are run through the `main.sh` script :
+
+*brief:* This script is used to launch the full pipeline of performance testing for one specific commit or branch.
+It is parametrized by three files :
+- perf.scenes : List of  "${scene name} ${starting time step to record} ${nax time step} ${number of scene launch}
+- default.timers : timers to be used for statistics for scenes that uses a DefaultAnimationLoop. It allows to redefine the timer name using the syntax TimeNameInStats=TimerNameInSOFA. You can also use simple addition and subtraction of timer names on the right-hand side (e.g : "CollisionDetection=CollisionEndEvent-CollisionBeginEvent" or "ProjectAndMechaMap=ProjectAndPropagateDx+ProjectAndPropagateXAndV"). Subtraction uses timestamp and addition time spawn. This has been added to be able to have a unified notation between those timers an the ones from FreeMotionAnimaitonLoop scenes
+- freemotion.timers : Same than the last one but for scenes with FreeMotionAnimationLoop
+
+*inputs:*
+- workdir     (required): directory used to store all working files. Where folders build, sofa, and output will be created.
+- sofa-branch (required): branch to test
+- commit-hash (optional): specific commit to test, if none is given, the tip of the branch will be tested.
+
+*env:*
+- RO_GITHUB_TOKEN: a read only github token
+
+*outputs:*
+- log files in the folder $WORK_DIR/logs
+- computed statistic : in a file named after the commit hash with the commit date and time (or the release version if only a branch is given and its name follows the release naming convention vXX.XX). e.g. sha_yyyy-mm-dd_hh-mm-ss.csv. The statistics are formated in a matrix which lines are the scenes names defined in perf.scenes and column are the results of the cartesian product between the timer names present in the files `default.timers` and `freemotion.timers` and all the type of statistics defined in generate_statistics/methods.py:computeSingleData.
+- hardware info : hardware information of the computer who generated the results. Same naming convention except that no date or time is added.
+
+Note that the statistic file and hard info file are copied in the repository in folder old_results to be ready to commit them
+
+### Dashboard
+If you just want to display the archived results you can only launch the dashboard script in `dashboard/dashboard.py`.
+There are two ways of launching it :
+1. Using directly python3 to run the script. This will result in a dashoboard being accessible on 127.0.0.1:8050
+2. Using docker compose. You'll have two solution, either using the pre generated image by ging in the folder `dashboard/docker/` and running `docker compose up redis` or by building it yourself buy running `docker compose build && docker compose up local`
+
+#### Dependencies
+Dependencies depends on how you run it:
+1. If runing through python3, you'll need to install all the dependencies in the file `dashboard/docker/requirements.txt` by calling `python3 -m pip install -r requirements.txt`, you can skip `gunicorn`.
+2. If runing through docker, you'll need docker desktop and docker compose. See https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
+