[CI][Bench] Create summary reports for benchmarking CI run results

[ianayl]

Much easier to figure out what's wrong with the benchmarking CI runs when it tells you what's wrong immediately: https://github.com/intel/llvm/actions/runs/16789472825

[ianayl]

Test runs, feel free to get an idea of what the summaries look like here:

• regression: https://github.com/intel/llvm/actions/runs/16789472825
• no regression: https://github.com/intel/llvm/actions/runs/16887540034

[sarnex]

yaml lgtm

[ianayl]

@intel/llvm-reviewers-benchmarking Friendly ping

[uditagarwal97]

Sorry, could you clarify what you meant by non CI-failing regressions?

[ianayl]

Hey thanks for taking a look Udit,
The benchmark CI now also runs UR benchmarks, L0 benchmarks, etc.; regressions in e.g. L0 should not cause the nightly benchmarking CI for SYCL to fail, thus they are filtered out and categorized differently.

[uditagarwal97]

I see. In that case, should we rename it to "non-SYCL regressions"?

[uditagarwal97]

Also, we don't list "regressions" in the summary where delta is less than the noise threshold, correct?

[ianayl]

I see. In that case, should we rename it to "non-SYCL regressions"?

I feel like that would be less confusing, but I am also aware that other projects use these series of benchmarking scripts as well (i.e. UMF), so I was hesitant to hardcode "non-SYCL" into the titles/descriptions. In hindsight this should perhaps be a customizable option.

Also, we don't list "regressions" in the summary where delta is less than the noise threshold, correct?

That is correct. Noise is ignored.

[ianayl]

Amended PR based on reviewer comments. New test runs:

• success (no regression): https://github.com/intel/llvm/actions/runs/16947777946
• regression, but with no user-defined regression filter type: https://github.com/intel/llvm/actions/runs/16970106654
• regression, but with user-defined regression filter type (in this case it's SYCL): https://github.com/intel/llvm/actions/runs/16974027983

[Copilot]

Pull Request Overview

Adds summary reports for benchmarking CI runs to improve visibility into benchmark failures. The change creates a GitHub CI summary file that provides a formatted overview of benchmark results, including improvements, filtered regressions, and concerning regressions.

• Adds new command-line arguments for producing GitHub summary reports
• Enhances the comparison script to generate markdown summaries alongside console output
• Updates the benchmark action to create and display the summary in GitHub CI

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
devops/scripts/benchmarks/compare.py	Adds GitHub summary generation functionality with new CLI arguments and markdown formatting for benchmark results
devops/actions/run-tests/benchmark/action.yml	Updates benchmark action to generate summary file and display it in GitHub CI step summary

[uditagarwal97]

Amended PR based on reviewer comments. New test runs:

• success (no regression): https://github.com/intel/llvm/actions/runs/16947777946
• regression, but with no user-defined regression filter type: https://github.com/intel/llvm/actions/runs/16970106654
• regression, but with user-defined regression filter type (in this case it's SYCL): https://github.com/intel/llvm/actions/runs/16974027983

The summary looks good now. 👍🏻

[uditagarwal97]

LGTM. Thanks.

[ianayl]

@intel/llvm-gatekeepers PR is ready for merge, thanks!

CI failure is unrelated and tracked in #19354 (Currently looking into this right now)