add delay between runs + result variation using different order

[devara-gheist]

1. How to add delays between runs? The purpose is to let the processor rest a little bit before starting the next run, to avoid thermal throttling.

Here's the code I'm using, note the programs are compiled binaries in multiple different languages:

import { run, bench, summary } from 'mitata';

const cPath = "../c/n_body_pthread_2";
const zigPath = "../zig/nBody_MT";
const rsPath = "../rs/n_body_MT7";
const pyPath = "../pyCodon/n_body_MT_py";

function delay(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

function runBench() {
    summary(() => {
        bench("PyCodon n_body MT", () => {
            Bun.spawnSync([pyPath]);
        });
        delay(10000);

        bench("Rust n_body MT", () => {
            Bun.spawnSync([rsPath]);
        });
        delay(10000);

        bench("Zig n_body MT", () => {
            Bun.spawnSync([zigPath]);
        });
        delay(10000);

        bench("C n_body MT", () => {
            Bun.spawnSync([cPath]);
        });
    })

    run();
}

runBench();

However monitoring the CPU via btop I see no delay at all between runs.

2. Different order of benchmarks yields wide variation of result.
   Please take a look at the series of images below:

• Order-1
  

• Order-2
  

Benchmark runs early in the order gets better result, while those in last orders gets worse result. How to avoid this? This is a consistent behaviour I monitored when testing the benchmark for 10x.

Also this is the main reason I was looking for a way to introduce delay to avoid thermal throttling.

3. Results are worse compared to using Hyperfine
   Below is the benchmark summary I did for the same set of programs via Hyperfine:
   

Note they are within 10-20 minutes between running bench using Mitata vs Hyperfine, of which the system processes remains the same. But on the bright side the percentage of which one is faster by how many % is close between Mitata and Hyperfine summaries.

[Update]: below is the latest bench I did with Mitata and Hyperfine.

Notice the numbers are closer because I "rest" my system for some time..but also note Mitata's numbers are always a bit higher than Hyperfine's.

Anyway I actually like Mitata especially with the options to add graphs to the benchmark result. I'm looking forward to use Mitata more once I understand more about it. Cheers!

[observ33r]

Static benchmarks are compiled and executed with await run(); at once. If you want to do something before bench runs are executed, use function generator..

import { setTimeout as sleep } from 'timers/promises'

summary(() => {
    bench('#1', async function* () {
        await sleep(10000);
        yield () => Bun.spawnSync(...)
    });
    /*...*/
});

await run();