[Feature]: Solve the Perf regression test large variance

[MrGeva]

🚀 The feature, motivation and pitch

The test_perf.py regression test shows large variance when run in CI. I see that it runs on two different device types: H100 NVL and H100 PCIE. according to that the perf varies and also the clock frequencies. Need to ability to define threshold per device type if possible, otherwise need to use only one device type by applying a filter in the ci definitions (devops said that they use two device kinds due to machine shortage). see the attached excel chart analyzing the perf variance in correlation to the clock freqs implying difference in perf is due to two different freq levels due to two different device types being used.

test_report_test_perf_metric_token_throughput_llama_v3.1_8b_instruct-bench-_autodeploy-float16-maxbs_512-maxnt_2.csv

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.