Skip to content

[CI] Remove dependency on bot user for uploading benchmark CI results #20333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: sycl
Choose a base branch
from
+19 −11
Open

[CI] Remove dependency on bot user for uploading benchmark CI results #20333

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
df8990c
test pushing to branch
ianayl Oct 3, 2025
7ce4420
give write permissions
ianayl Oct 8, 2025
866b64e
amend perms
ianayl Oct 8, 2025
b518e18
Revert "amend perms"
ianayl Oct 8, 2025
2109b73
Revert "give write permissions"
ianayl Oct 8, 2025
6e73cf7
test perms
ianayl Oct 8, 2025
e20975f
typo
ianayl Oct 8, 2025
3bea805
more perms
ianayl Oct 8, 2025
418bd1d
test fail
ianayl Oct 9, 2025
1539867
set to correct repo
ianayl Oct 9, 2025
8e1f1b8
switch branch
ianayl Oct 9, 2025
66c4b5c
try removing manually passing secrets
ianayl Oct 10, 2025
baf64ef
fix typo
ianayl Oct 14, 2025
ac56a43
make write permissions job-level
ianayl Oct 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/sycl-docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ jobs:
touch .nojekyll
# Update benchmarking dashboard configuration
cat << EOF > benchmarks/config.js
remoteDataUrl = 'https://raw.githubusercontent.com/intel/llvm-ci-perf-results/refs/heads/unify-ci/';
remoteDataUrl = 'https://raw.githubusercontent.com/intel/llvm/refs/heads/sycl-benchmark-ci-results/';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my note on this change - while it seems convenient, I can see on my PC, that repo intel/llvm-ci-perf-results is ~600MB - this will be extra MBs added to intel/llvm repo, which is already quite big - just to consider

Copy link
Contributor

@uditagarwal97 uditagarwal97 Oct 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ianayl would it be possible to compress the benchmarking data before pushing it to intel/llvm ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory yes, but only the archived data. I think we'll need to have a conversation on this: it might even be reasonable to delete horribly outdated data.

We could also try moving archived data to intel/llvm-ci-perf-results, but alas that'll mean we need to keep an updated bot user token again for the repository.

Do github repositories have a max storage quota?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do github repositories have a max storage quota?

https://docs.github.com/en/repositories/creating-and-managing-repositories/repository-limits#repository-size

we'll need to have a conversation on this

I think we should have that conversation before merging this PR. Currently, the size of intel/llvm-ci-perf-results is ~600MB, but, if unchecked the size will increase further.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory yes, but only the archived data. I think we'll need to have a conversation on this: it might even be reasonable to delete horribly outdated data.

Compressing results files gives ~190MB out of 320M, so I think it is better to just remove outdated data. We have data starting at March'25.

How many months of data do wee need to keep. 6? In this case we will have data staring from 15th April today. The rest will be in git history, hopefully well compressed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also try moving archived data to intel/llvm-ci-perf-results, but alas that'll mean we need to keep an updated bot user token again for the repository.

If we go this way, we can archive e.g. once every second week, or something, and make a PR instead of pushing

defaultCompareNames = ["Baseline_PVC_L0"];
EOF
# Upload the generated docs as an artifact and deploy to GitHub Pages.
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/sycl-linux-run-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ on:
- "run-only"

permissions:
contents: read
contents: write
packages: read

jobs:
Expand Down Expand Up @@ -361,4 +361,3 @@ jobs:
build_ref: ${{ inputs.repo_ref }}
env:
RUNNER_TAG: ${{ inputs.runner }}
GITHUB_TOKEN: ${{ secrets.LLVM_SYCL_BENCHMARK_TOKEN }}
3 changes: 3 additions & 0 deletions .github/workflows/sycl-ur-perf-benchmarking.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,9 @@ jobs:
run_benchmarks_build:
name: Run Benchmarks on Build
needs: [ build_sycl, sanitize_inputs ]
permissions:
contents: write
packages: read
strategy:
matrix:
include:
Expand Down
21 changes: 12 additions & 9 deletions devops/actions/run-tests/benchmark/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,9 @@ runs:
python3 ./devops/scripts/benchmarks/presets.py query "$PRESET"
[ "$?" -ne 0 ] && exit 1 # Stop workflow if invalid preset
echo "PRESET=$PRESET" >> $GITHUB_ENV

# Set branch containing benchmark CI results:
echo "BENCHMARK_RESULTS_BRANCH=sycl-benchmark-ci-results" >> $GITHUB_ENV
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps just set it as an env (in the job scope, or top-level)...?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there another way to set an env variable? I thought this is how you do it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way you did it I usually use when I want to dynamically set something within the workflow.

If more of a static var is required, I usually use:

env:
  BUILD_DIR : "${{github.workspace}}/build"
  INSTALL_DIR: "${{github.workspace}}/build/install"

e.g. in UMF workflow:
https://github.com/oneapi-src/unified-memory-framework/blob/main/.github/workflows/nightly.yml#L14-L16

it will work in both "job" and "global" scope

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this defining the env variables for a single step only? i.e. variables defined this way do not propagate across steps

I use the variable again in actions/checkout call here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as I mentioned, you can set it up in "job" and/or "global" scope - see my example above set up for global scope

I think, setting it up for step scope is only possible if you use with clause (e.g. for running reusable workflow or an action)

- name: Compute CPU core range to run benchmarks on
shell: bash
run: |
Expand Down Expand Up @@ -134,9 +137,10 @@ runs:

cd -
- name: Checkout results repo
shell: bash
run: |
git clone -b unify-ci https://github.com/intel/llvm-ci-perf-results
uses: actions/checkout@v5
with:
ref: ${{ env.BENCHMARK_RESULTS_BRANCH }}
path: llvm-ci-perf-results
- name: Run compute-benchmarks
env:
# Need to append "_<device>_<backend>" to save name in order to follow
Expand Down Expand Up @@ -237,9 +241,8 @@ runs:
shell: bash
run: |
cd "./llvm-ci-perf-results"
git config user.name "SYCL Benchmarking Bot"
git config user.email "[email protected]"
results_branch="unify-ci"
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"

if git diff --quiet && git diff --cached --quiet; then
echo "No new results added, skipping push."
Expand All @@ -252,7 +255,7 @@ runs:
git commit -m "[GHA] Upload compute-benchmarks results from https://github.com/intel/llvm/actions/runs/${{ github.run_id }}"
results_file="$(git diff HEAD~1 --name-only -- results/ | head -n 1)"

if git push "https://[email protected]/intel/llvm-ci-perf-results.git" "$results_branch"; then
if git push; then
echo "Push succeeded"
break
fi
Expand All @@ -262,8 +265,8 @@ runs:
cached_result="$(mktemp -d)/$(basename $results_file)"
mv "$results_file" "$cached_result"

git reset --hard "origin/$results_branch"
git pull origin "$results_branch"
git reset --hard "origin/$BENCHMARK_CI_RESULTS"
git pull

mv "$cached_result" "$results_file"
fi
Expand Down