Skip to content

Commit 8d7c78e

Browse files
committed
Add regression check automation
This commit adds benchmark_regression_test.yml workflow that runs a regression test by comparing the benchmark results of the current branch with the previous benchmark results of the code in main branch. Signed-off-by: Arjun Raja Yogidas <[email protected]>
1 parent 93c9050 commit 8d7c78e

File tree

1 file changed

+68
-0
lines changed

1 file changed

+68
-0
lines changed
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
name: Benchmark Regression Check
2+
3+
on:
4+
pull_request:
5+
branches: [ main ]
6+
paths:
7+
- '**.go'
8+
- 'go.*'
9+
- 'cmd/go.*'
10+
- 'Makefile'
11+
- 'Dockerfile'
12+
- 'integration/**'
13+
- 'scripts/**'
14+
- '.github/workflows/**'
15+
16+
jobs:
17+
benchmark-and-fetch-previous-results_and_compare:
18+
name: Run Benchmark on current Branch
19+
runs-on: ubuntu-20.04
20+
steps:
21+
- uses: actions/checkout@v3
22+
- uses: actions/setup-go@v4
23+
with:
24+
go-version: '1.18.10'
25+
- run: make
26+
- name: Run benchmark
27+
run: make benchmarks-perf-test
28+
- name: Make previous directory
29+
run: mkdir -v ${{ github.workspace }}/previous
30+
- name: Download previous run artifact
31+
id: download-artifact
32+
uses: dawidd6/action-download-artifact@v2
33+
with:
34+
github_token: ${{secrets.GITHUB_TOKEN}}
35+
name: benchmark-result-artifact
36+
name_is_regexp: true
37+
path: ${{ github.workspace }}/previous
38+
repo: ${{ github.repository }}
39+
check_artifacts: false
40+
search_artifacts: true
41+
skip_unpack: false
42+
if_no_artifact_found: fail
43+
workflow: benchmark_visualization.yml
44+
- name: Perform Comparison and log results
45+
id: run-compare
46+
run: |
47+
sudo chmod +x ${{ github.workspace }}/scripts/check_regression.sh
48+
if sudo ${{ github.workspace }}/scripts/check_regression.sh ${{ github.workspace }}/previous/benchmark-result-artifact/results.json ${{github.workspace}}/benchmark/performanceTest/output/results.json; then
49+
echo "Comparison successful. All P90 values are within the acceptable range."
50+
else
51+
echo "Comparison failed. Current P90 values exceed 110% of the corresponding past values."
52+
echo "regression-detected=true" >> $GITHUB_OUTPUT
53+
fi
54+
- name: Stop the workflow if regression is detected
55+
if: steps.run-compare.outputs.regression-detected == 'true'
56+
uses: actions/github-script@v6
57+
with:
58+
github-token: ${{ secrets.GITHUB_TOKEN }}
59+
script: |
60+
const comment = `
61+
:warning: **Regression Detected** :warning:
62+
63+
The benchmark comparison indicates that there has been a performance regression.
64+
Please investigate and address the issue.
65+
To Investigate check logs of the previous job above.
66+
`;
67+
68+
core.setFailed(comment);

0 commit comments

Comments
 (0)