Performance history canary (#1209)

This PR adds a "canary" build to the performance regression CI of
OpenJDK. The "canary" is a chosen revision of mmtk-core and mmtk-openjdk
that is tested alongside each merged PR. The performance of the "canary"
should not change unless there is an environment change or there is a
noise. Spotting a change in the "canary" performance can help us
identify environment changes that are unintended or otherwise unnoticed,
and also identify the noise level.

Currently, we choose a specific release version as the version of the
"canary". Using a release version has the advantage of being easy to
specify the exact revision of both the mmtk-core and the mmtk-openjdk
repository. We may also switch to some methods of automatically select
the canary version in the future.

There are other minor changes made.

- We slightly change the directory structure. We create two directory,
namely `latest` and `canary`. In each of the directories, we check out
`mmtk-core` and `mmtk-openjdk` of the latest and the canary versions,
respectively.
- We use the `ci-replace-mmtk-dep.py` script to replace the revision of
the `mmtk-core` dependency in `mmtk-openjdk`. As a result, we no longer
need to use `sed`, and no longer need to copy the `mmtk-core` directory
into `mmtk-openjdk/repos`.
-   We no longer set the `RUSTUP_TOOLCHAIN` environment variable because
1. The latest and the canary version may not use the same toolchain,
and,
2. the right toolchain will be chosen when running the `cargo` command
according to the `rust-toolchain` file in the directory.
- The scripts in https://github.com/mmtk/ci-perf-kit are changed to take
the canary into consideration, too.

Author: wks

.github/workflows/perf-regression-ci.yml

@@ -97,45 +97,81 @@ jobs:
   openjdk-perf-regression:
     runs-on: [self-hosted, Linux, freq-scaling-off]
     timeout-minutes: 1440
+    env:
+      # This version will be used as the canary version, and will be used to checkout both
+      # `mmtk-core` and `mmtk-openjdk`.
+      #
+      # A "canary" is a chosen version that is tested alongside each merged pull request.  The
+      # performance of the canary should not change unless
+      #
+      # 1.  There is an environment change, such as changes of the operating system, the hardware,
+      #     the firmware, or the methodology we use for testing, or
+      # 2.  There are non-deterministic factors (i.e. noises) affecting each execution.
+      #
+      # Running the canary alongside regular regression tests help us identify unnoticed environment
+      # changes and the level of noise.
+      #
+      # Currently, we choose one release version as the canary, and will use it until we can no
+      # longer run it for any reason, such as the toolchain for compiling that version is no longer
+      # available.  Then we may change to another release version and mark the change of canary on
+      # the timeline, or introduce a mechanism to dynamically choose the canary version.
+      CANARY_VERSION: "v0.28.0"
     steps:
-      - name: Checkout MMTk Core
+      # checkout latest versions
+      - name: Checkout MMTk Core (latest)
         uses: actions/checkout@v4
         with:
-          path: mmtk-core
-      - name: Checkout OpenJDK Binding
+          path: latest/mmtk-core
+      - name: Checkout OpenJDK Binding (latest)
         uses: actions/checkout@v4
         with:
           repository: mmtk/mmtk-openjdk
-          path: mmtk-openjdk
-      - name: Checkout OpenJDK
-        working-directory: mmtk-openjdk
+          path: latest/mmtk-openjdk
+      - name: Checkout OpenJDK (latest)
+        working-directory: latest/mmtk-openjdk
         run: |
           ./.github/scripts/ci-checkout.sh
       # checkout perf-kit
       - name: Checkout Perf Kit
         uses: actions/checkout@v4
         with:
           repository: mmtk/ci-perf-kit
-          ref: "0.8.0"
+          ref: "0.8.1"
           path: ci-perf-kit
-          token: ${{ secrets.CI_ACCESS_TOKEN }}
           submodules: true
-      # setup
-      - name: Overwrite MMTk core in openjdk binding
-        run: cp -r mmtk-core mmtk-openjdk/repos/
-      - name: Setup Rust Toolchain
-        run: echo "RUSTUP_TOOLCHAIN=`cat mmtk-core/rust-toolchain`" >> $GITHUB_ENV
-      # cleanup previosu build
-      - name: Cleanup previous build
+      # checkout canary versions.
+      - name: Checkout MMTk Core (canary)
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ env.CANARY_VERSION }}
+          path: canary/mmtk-core
+      - name: Checkout OpenJDK Binding (canary)
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ env.CANARY_VERSION }}
+          repository: mmtk/mmtk-openjdk
+          path: canary/mmtk-openjdk
+      - name: Checkout OpenJDK (canary)
+        working-directory: canary/mmtk-openjdk
         run: |
-          rm -rf mmtk-openjdk/repos/openjdk/scratch
-          rm -rf mmtk-openjdk/repos/openjdk/build
-      - name: Setup
+          ./.github/scripts/ci-checkout.sh
+      # setup
+      - name: Setup directory structures
         run: |
-          ./ci-perf-kit/scripts/history-run-setup.sh
-          sed -i 's/^mmtk[[:space:]]=/#ci:mmtk=/g' mmtk-openjdk/mmtk/Cargo.toml
-          sed -i 's/^#[[:space:]]mmtk/mmtk/g' mmtk-openjdk/mmtk/Cargo.toml
-      - id: branch
+          for BASE_DIR in ./latest ./canary; do
+            pushd $BASE_DIR
+              # replace dependency
+              # Note that ci-replace-mmtk-dep.sh will apply `realpath()` to the `--mmtk-core-path` option.
+              # so we specify the relative path from the PWD to the mmtk-core repo.
+              ./mmtk-core/.github/scripts/ci-replace-mmtk-dep.sh mmtk-openjdk/mmtk/Cargo.toml \
+                --mmtk-core-path mmtk-core
+              # cleanup previous build
+              rm -rf mmtk-openjdk/repos/openjdk/scratch
+              rm -rf mmtk-openjdk/repos/openjdk/build
+            popd
+          done
+      - name: Setup branch name
+        id: branch
         # we cannot use env vars in action input (the deploy step). So put the env var to this step's outputs.
         run: echo "branch_name=$(echo ${GITHUB_REF#refs/heads/} | sed 's/\//_/g')" >> $GITHUB_OUTPUT
       # run
@@ -145,7 +181,10 @@ jobs:
           export RESULT_REPO_BRANCH=${{ env.RESULT_REPO_BRANCH }}
           export RESULT_REPO_ACCESS_TOKEN=${{ secrets.CI_ACCESS_TOKEN }}
           export FROM_DATE=2020-07-10
-          ./ci-perf-kit/scripts/openjdk-history-run.sh ./mmtk-openjdk ./reports/${{ steps.branch.outputs.branch_name }}
+          ./ci-perf-kit/scripts/openjdk-history-run.sh \
+            ./latest/mmtk-openjdk \
+            ./canary/mmtk-openjdk \
+            ./reports/${{ steps.branch.outputs.branch_name }}
       # deploy
       - name: Deploy to Github Page
         if: ${{ env.DEPLOY == 'true' }}