variation of results depending on the order of execution

[EdamAme-x]

It would be fair to use the Runtime API to measure CPU/Memory Usage and run each benchmark when each has a sufficiently fair excess, but is there already such a feature?

[evanwashere]

as long as manual garbage collection is enabled, mitata will always remove its own garbage that could cause GC spikes and affect benchmark numbers, but otherwise there is ~zero (at most picoseconds) overhead from mitata

most usual outside causes are:

• results are within noise margin
• cpu clock frequency changes during benchmarking
• code outside mitata is causing gc spikes (lots of allocations)
• something else on system is occupying cpu time and starving the benchmark (quickest way to check this is to look at estimated clock frequency above cpu name)

if you could provide snippet to replicate this I could look into it deeper

[evanwashere]

I added new function to help tune gc in case if gc spikes are causing your problems

// will run gc before each iteration
bench(...).gc('inner')