[CI][UR] Bring up UR benchmarking results dashboard in intel.github.io #17475
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| name: Run Benchmarks | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: '0 1 * * *' # 2 hrs earlier than sycl-nightly.yml | ||
| workflow_call: | ||
| inputs: | ||
| commit_hash: | ||
| type: string | ||
| required: false | ||
| default: '' | ||
| upload_results: | ||
| type: string # true/false: workflow_dispatch does not support booleans | ||
| required: true | ||
| runner: | ||
| type: string | ||
| required: true | ||
| backend: | ||
| type: string | ||
| required: true | ||
| reset_intel_gpu: | ||
| type: string # true/false: workflow_dispatch does not support booleans | ||
| required: true | ||
| default: true | ||
|
|
||
| workflow_dispatch: | ||
| inputs: | ||
| commit_hash: | ||
| description: Commit hash to build intel/llvm from | ||
| type: string | ||
| required: false | ||
| default: '' | ||
| upload_results: | ||
| description: 'Save and upload results' | ||
| type: choice | ||
| options: | ||
| - false | ||
| - true | ||
| default: true | ||
| runner: | ||
| type: choice | ||
| options: | ||
| - '["PVC_PERF"]' | ||
| backend: | ||
| description: Backend to use | ||
| type: choice | ||
| options: | ||
| - 'level_zero:gpu' | ||
| # TODO L0 V2 support | ||
| reset_intel_gpu: | ||
| description: Reset Intel GPUs | ||
| type: choice | ||
| options: | ||
| - false | ||
| - true | ||
| default: true | ||
|
|
||
| permissions: read-all | ||
|
|
||
| jobs: | ||
| build_sycl: | ||
| name: Build SYCL from PR | ||
| if: inputs.commit_hash != '' | ||
|
There was a problem hiding this comment. techincally passing empty // FYI, trigger type can be used to establish if nightly or not There was a problem hiding this comment. Good catch! think I resolved this via the latest commit: I've set default values on commit_hash to always be a string // trigger type can be used to determine if a job was scheduled, but I've chosen to use commit_hash here incase the user themselves wants to use the nightly build of sycl manually There was a problem hiding this comment. hmm tbh, I'm not sure if setting There was a problem hiding this comment. maybe I'm missing something, but can we easily read the hash commit from PR? I know we used There was a problem hiding this comment. The sycl-linux-build.yml job in intel/llvm doesn't support compiling commits from other branches, thus we cannot access commits from PR branches as of now: I have chosen to not stir the nest here, but this will probably be a future change that involes changing sycl-linux-build.yml There was a problem hiding this comment. hmm... so what should I pass here to check my PR? or is it intended as a post-merge check? There was a problem hiding this comment. Admittedly I dont think that'd be possible right now 😅 We will figure something out though, given that this PR is no longer urgent, I'll make a PR to change sycl-linux-build.yml so that we can actually accomplish this |
||
| uses: ./.github/workflows/sycl-linux-build.yml | ||
| with: | ||
| build_ref: ${{ inputs.commit_hash }} | ||
| build_cache_root: "/__w/" | ||
| build_artifact_suffix: "default" | ||
| build_cache_suffix: "default" | ||
| # Docker image has last nightly pre-installed and added to the PATH | ||
| build_image: "ghcr.io/intel/llvm/sycl_ubuntu2404_nightly:latest" | ||
| cc: clang | ||
| cxx: clang++ | ||
| changes: '[]' | ||
|
|
||
| run_benchmarks_build: | ||
| name: Run Benchmarks (on PR Build) | ||
| needs: [ build_sycl ] | ||
| if: inputs.commit_hash != '' | ||
| strategy: | ||
| matrix: | ||
| # Set default values if not specified: | ||
| include: | ||
| - runner: ${{ inputs.runner || '["PVC_PERF"]' }} | ||
| backend: ${{ inputs.backend || 'level_zero:gpu' }} | ||
| reset_intel_gpu: ${{ inputs.reset_intel_gpu || 'true' }} | ||
| ref: ${{ inputs.commit_hash }} | ||
| uses: ./.github/workflows/sycl-linux-run-tests.yml | ||
| secrets: inherit | ||
| with: | ||
| # TODO support other benchmarks | ||
| name: Run compute-benchmarks (${{ matrix.runner }}, ${{ matrix.backend }}) | ||
| runner: ${{ matrix.runner }} | ||
| image: ghcr.io/intel/llvm/sycl_ubuntu2404_nightly:latest | ||
| image_options: -u 1001 --device=/dev/dri -v /dev/dri/by-path:/dev/dri/by-path --privileged --cap-add SYS_ADMIN | ||
| target_devices: ${{ matrix.backend }} | ||
| reset_intel_gpu: ${{ matrix.reset_intel_gpu }} | ||
| tests_selector: benchmark_v2 | ||
| benchmark_upload_results: ${{ inputs.upload_results }} | ||
| benchmark_build_hash: ${{ inputs.commit_hash }} | ||
| repo_ref: ${{ matrix.ref }} | ||
| devops_ref: ${{ github.ref }} | ||
| sycl_toolchain_artifact: sycl_linux_default | ||
| sycl_toolchain_archive: ${{ needs.build_sycl.outputs.artifact_archive_name }} | ||
| sycl_toolchain_decompress_command: ${{ needs.build_sycl.outputs.artifact_decompress_command }} | ||
|
|
||
| run_benchmarks_nightly: | ||
| name: Run Benchmarks (on Nightly Build) | ||
| if: inputs.commit_hash == '' | ||
| strategy: | ||
| matrix: | ||
| # Set default values if not specified: | ||
| include: | ||
| - runner: ${{ inputs.runner || '["PVC_PERF"]' }} | ||
| backend: ${{ inputs.backend || 'level_zero:gpu' }} | ||
| reset_intel_gpu: ${{ inputs.reset_intel_gpu || 'true' }} | ||
| uses: ./.github/workflows/sycl-linux-run-tests.yml | ||
| secrets: inherit | ||
| with: | ||
| # TODO support other benchmarks | ||
| name: Run compute-benchmarks (${{ matrix.runner }}, ${{ matrix.backend }}) | ||
| runner: ${{ matrix.runner }} | ||
| image: ghcr.io/intel/llvm/sycl_ubuntu2404_nightly:latest | ||
| image_options: -u 1001 --device=/dev/dri -v /dev/dri/by-path:/dev/dri/by-path --privileged --cap-add SYS_ADMIN | ||
| target_devices: ${{ matrix.backend }} | ||
| reset_intel_gpu: ${{ matrix.reset_intel_gpu }} | ||
| tests_selector: benchmark_v2 | ||
| benchmark_upload_results: ${{ inputs.upload_results }} | ||
| repo_ref: ${{ github.ref }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| name: 'Run benchmarks' | ||
|
|
||
| # This action assumes the following prerequisites: | ||
| # | ||
| # - SYCL is placed in ./toolchain -- TODO change this | ||
| # - /devops has been checked out in ./devops. | ||
| # - env.GITHUB_TOKEN was properly set, because according to Github, that's | ||
| # apparently the recommended way to pass a secret into a github action: | ||
|
|
||
| # https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions#accessing-your-secrets | ||
| # | ||
| # - env.RUNNER_TAG set to the runner tag used to run this workflow: Currently, | ||
| # only specific runners are fully supported. | ||
|
|
||
| inputs: | ||
| target_devices: | ||
| type: string | ||
| required: True | ||
| upload_results: | ||
| type: string | ||
| required: True | ||
| build_hash: | ||
| type: string | ||
| required: False | ||
| default: '' | ||
|
|
||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Check specified runner type / target backend | ||
| shell: bash | ||
| env: | ||
| TARGET_DEVICE: ${{ inputs.target_devices }} | ||
| RUNNER_NAME: ${{ runner.name }} | ||
lukaszstolarczuk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| run: | | ||
| case "$RUNNER_TAG" in | ||
| '["PVC_PERF"]' ) ;; | ||
| *) | ||
| echo "#" | ||
| echo "# WARNING: Only specific tuned runners are fully supported." | ||
| echo "# This workflow is not guaranteed to work with other runners." | ||
| echo "#" ;; | ||
| esac | ||
| # Ensure runner name has nothing injected | ||
| # TODO: in terms of security, is this overkill? | ||
| if [ -z "$(printf '%s' "$RUNNER_NAME" | grep -oE '^[a-zA-Z0-9_-]+$')" ]; then | ||
| echo "Bad runner name, please ensure runner name is [a-zA-Z0-9_-]." | ||
| exit 1 | ||
| fi | ||
| echo "RUNNER_NAME=$RUNNER_NAME" >> $GITHUB_ENV | ||
| # input.target_devices is not directly used, as this allows code injection | ||
| case "$TARGET_DEVICE" in | ||
| level_zero:*) ;; | ||
| *) | ||
| echo "#" | ||
| echo "# WARNING: Only level_zero backend is fully supported." | ||
| echo "# This workflow is not guaranteed to work with other backends." | ||
| echo "#" ;; | ||
| esac | ||
| echo "ONEAPI_DEVICE_SELECTOR=$TARGET_DEVICE" >> $GITHUB_ENV | ||
| - name: Compute CPU core range to run benchmarks on | ||
| shell: bash | ||
| run: | | ||
| # Compute the core range for the first NUMA node; second node is used by | ||
| # UMF. Skip the first 4 cores as the kernel is likely to schedule more | ||
| # work on these. | ||
| CORES="$(lscpu | awk ' | ||
| /NUMA node0 CPU|On-line CPU/ {line=$0} | ||
| END { | ||
| split(line, a, " ") | ||
| split(a[4], b, ",") | ||
| sub(/^0/, "4", b[1]) | ||
| print b[1] | ||
| }')" | ||
| echo "CPU core range to use: $CORES" | ||
| echo "CORES=$CORES" >> $GITHUB_ENV | ||
| ZE_AFFINITY_MASK=0 | ||
| echo "ZE_AFFINITY_MASK=$ZE_AFFINITY_MASK" >> $GITHUB_ENV | ||
| - name: Checkout results repo | ||
| shell: bash | ||
| run: | | ||
| git clone -b unify-ci https://github.com/intel/llvm-ci-perf-results | ||
ianayl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - name: Run compute-benchmarks | ||
| env: | ||
| BUILD_HASH: ${{ inputs.build_hash }} | ||
| shell: bash | ||
| run: | | ||
| # TODO generate summary + display helpful message here | ||
| export CMPLR_ROOT=./toolchain | ||
| echo "-----" | ||
| sycl-ls | ||
| echo "-----" | ||
| pip install --user --break-system-packages -r ./devops/scripts/benchmarks/requirements.txt | ||
| echo "-----" | ||
| mkdir -p "./llvm-ci-perf-results/$RUNNER_NAME" | ||
| # TODO accomodate for different GPUs and backends | ||
| SAVE_NAME="Baseline_PVC_L0" | ||
| if [ -n "$BUILD_HASH" ]; then | ||
| SAVE_NAME="Commit_PVC_$BUILD_HASH" | ||
| fi | ||
| taskset -c "$CORES" ./devops/scripts/benchmarks/main.py \ | ||
| "$(realpath ./llvm_test_workdir)" \ | ||
| --sycl "$(realpath ./toolchain)" \ | ||
| --save "$SAVE_NAME" \ | ||
| --output-html remote \ | ||
| --results-dir "./llvm-ci-perf-results/$RUNNER_NAME" \ | ||
| --output-dir "./llvm-ci-perf-results/$RUNNER_NAME" \ | ||
| --preset Minimal | ||
ianayl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| echo "-----" | ||
| - name: Push compute-benchmarks results | ||
| if: inputs.upload_results == 'true' && always() | ||
| shell: bash | ||
| run: | | ||
| # TODO redo configuration | ||
| # $(python ./devops/scripts/benchmarking/load_config.py ./devops constants) | ||
| cd "./llvm-ci-perf-results" | ||
| git config user.name "SYCL Benchmarking Bot" | ||
| git config user.email "[email protected]" | ||
| git pull | ||
| git add . | ||
| # Make sure changes have been made | ||
| if git diff --quiet && git diff --cached --quiet; then | ||
| echo "No new results added, skipping push." | ||
| else | ||
| git commit -m "[GHA] Upload compute-benchmarks results from https://github.com/intel/llvm/actions/runs/${{ github.run_id }}" | ||
| git push "https://[email protected]/intel/llvm-ci-perf-results.git" unify-ci | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.