t-test is too slow to be enabled by default

[jdmarshall]

Discussion in RafaelGSS/bench-node#148 is leaning toward me not being able to run this as default. Each benchmark takes at least 15 seconds because the repeat count is 30 x 0.5s.

This is due both to the way t-test is using opsSecPerRun instead of histogram.samples to calculate its confidence interval, and the way iterations are calculated which leads to 10 samples taking approximately the maxTime even on no-op tests.

[RafaelGSS]

I don't understand this issue. We never planned to enable it by default.

Benchmarks aren't meant to be very fast. First, it should be assertive and reliable.

[jdmarshall]

These are the existing benchmarks for prom-client, which is now using faceoff+bench-node to replace benchmark-regression+benchmark.js (the former is dormant, the second is declared EOL):

https://github.com/siimon/prom-client/tree/master/benchmarks

These benchmarks run in about 4.5 minutes. There's another PR out that add more metrics.

If I turned on ttest on these to stop reporting inconclusive results, the runtime would at present jump to two and a half hours.

I'm willing to press them on slower but more accurate tests especially if the false positives rate is reduced. But 145 minutes is not going to be acceptable.

I believe if you use 60 histogram samples per t-test, instead of 30 iterations per test, that runtime will look more like fifteen to twenty minutes. Also likely more accurate since you're double-averaging the numbers at present.

[jdmarshall]

RafaelGSS/bench-node#155 filed, on the grounds that it is inaccurate to use opsSerPerRun as input to t-test. Everything I read suggests that it's the sampleData instead that is the kind of data that should be fed into t-test.

This also represents a lot more short samples and therefore should better reveal jitter caused by the OS and v8.