Why do the benchmarks use wall-clock time?

[Raimo33]

Using wall-clock time seems problematic because it includes OS scheduling delays, interruptions from other processes, and other system noise, which are irrelevant to the actual computational cost of the library’s operations. Wouldn’t measuring CPU time give a far more precise and reliable metric of performance?

I'm running benchmarks on my machine and I see oscillations of +50% on some functions (with turboboost disabled, with CPU frequency fixed at 3 Ghz). It seems currently totally unreliable.

Yes, I can pin the process to a cpu core, but I can't prevent that core from handling other programs. Also, even if I somehow manage to have a core running only my benchmark and nothing else, I would still have caching issues because modern CPUs have shared caches between cores.

I think something like CLOCK_PROCESS_CPUTIME_ID would be way better

[real-or-random]

I'm running benchmarks on my machine and I see oscillations of +50% on some functions

This seems unusual.

What functions? Note that some functions are variable-time. How many benchmark iters are you using when you see this large variance?

[Raimo33]

ecmult_wnaf with 20'000 iterations (default). I had around 25 other processes running in the background (VScode + Browser). pinned to core 0. disabled turboboost.

other significant ones are hash_sha256 and group_to_affine_var