Use iteration count instead of time for parameters warmup and rep of do_bench* functions for benchmarks

[anmyachev]

Closes #2255

Partially reusing the changes, which were removed in #2142 (namely the part related to using iteration count instead of time) solves the problem of not having enough data for "CV" column.

Old comments

CI status:

• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10882781069 (100ms)
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10883238613 (150ms)
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10884030513 (150ms, without IPEX)
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10901498420 (200ms)
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10910606716 (300ms)
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10920458857 (300ms, without IPEX)
  • https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10942449220 (again with up to date main branch)
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10994968680 (warmup=10)
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/11092788850 (again with latest changes)

UPD: For some reason this greatly affects the mean time. However, if I reduce warmup, the mean does not deteriorate as much.

[anmyachev]

@ESI-SYD @chengjunlu geomean diff will most likely be less, I will write the exact figures here after https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/11106696646/job/30855590713 is finished:

• Triton ADV: -5%
• Triton DFT: -2%
• Xetla: -8%

Are you aware of this effect where as the number of runs increases, the average time gets noticeably worse? I don't know what to do with this slowdown, but I still think the idea of ​​running multiple times (>3, only in this case "*-CV" column will not be NaN) is good (from the point of view of calculating the average).

cc @whitneywhtsang @etiotto

[etiotto]

@ESI-SYD @chengjunlu geomean diff will most likely be less, I will write the exact figures here after https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/11106696646/job/30855590713 is finished:

• Triton ADV: -5%
• Triton DFT: -2%
• Xetla: -8%

Are you aware of this effect where as the number of runs increases, the average time gets noticeably worse? I don't know what to do with this slowdown, but I still think the idea of ​​running multiple times (>3, only in this case "*-CV" column will not be NaN) is good (from the point of view of calculating the average).

cc @whitneywhtsang @etiotto

I think that when the warmup runs "too many times" the GPU may start heating up and then throttle the frequency down, so when the timed run start the performance is reduced. That means we are better off not increasing the rep/warmup to the point we see performance degradations in the benchmarks.

[etiotto]

I do not think we should increase the number of repetition too much. Going from 10 too 600 repetitions is a huge increase.

The kernel timing distribution should be a normal (gaussian) curve. We only need to run the benchmark enough times to approximate a gaussian "bell" curve. From https://www.scribbr.com/statistics/central-limit-theorem/#:~:text=By%20convention%2C%20we%20consider%20a,if%20the%20population%20is%20normal. looks like 30 is the number of reps we should use.

[etiotto]

From 10 to 600 times? Way too many repetitions. It is going to slow down the time it takes to run the benchmarks too much.

[anmyachev]

From 10 to 600 times? Way too many repetitions. It is going to slow down the time it takes to run the benchmarks too much.

This value is measured in milliseconds and is needed for some test combinations where one run takes more than 100 ms.

[whitneywhtsang]

If we revert #2142, then rep is the number of iterations, then the problem of NaNs in CV is gone?

[anmyachev]

If we revert #2142, then rep is the number of iterations, then the problem of NaNs in CV is gone?

@whitneywhtsang Most likely yes. However, I made a change to make do_bench function more similar to the one used in upstream triton. If this is not necessary, I can revert some of the changes.

[whitneywhtsang]

If we revert #2142, then rep is the number of iterations, then the problem of NaNs in CV is gone?

@whitneywhtsang Most likely yes. However, I made a change to make do_bench function more similar to the one used in upstream triton. If this is not necessary, I can revert some of the changes.

I also see the benefit of being more similar to upstream triton, but rep meaning the number of iterations is more intuitive IMO.

[etiotto]

If we revert #2142, then rep is the number of iterations, then the problem of NaNs in CV is gone?

@whitneywhtsang Most likely yes. However, I made a change to make do_bench function more similar to the one used in upstream triton. If this is not necessary, I can revert some of the changes.

I also see the benefit of being more similar to upstream triton, but rep meaning the number of iterations is more intuitive IMO.

I agree. To me "rep" should really mean the number of repetition of the kernel after warmup. I think we can diverge from upstream here, and then try to upstream our changes.

[anmyachev]

There is no point in calculating the number of iterations through the expected time of one iteration, since the required number of iterations is requested by the user.

[anmyachev]

There is no point in calculating the number of iterations through the expected time of one iteration, since the required number of iterations is requested by the user.

[anmyachev]

I translate the parameters into those that upstream (triton_do_bench) understands.

[anmyachev]

If we revert #2142, then rep is the number of iterations, then the problem of NaNs in CV is gone?

@whitneywhtsang Most likely yes. However, I made a change to make do_bench function more similar to the one used in upstream triton. If this is not necessary, I can revert some of the changes.

I also see the benefit of being more similar to upstream triton, but rep meaning the number of iterations is more intuitive IMO.

I agree. To me "rep" should really mean the number of repetition of the kernel after warmup. I think we can diverge from upstream here, and then try to upstream our changes.

@etiotto @whitneywhtsang ready for review

[whitneywhtsang]

Please update PR description.