vitest has too much memory usage in the face of a large number of benches

[D-Sketon]

I'm a contributor of es-toolkit. When I try to run about 200+ bench.ts using yarn vitest bench, the memory footprint is too high. I allocated 26000mb of memory for the node and it still crashes.

Is there a potential memory leak in vitest, or is there something wrong with the benchmarks I wrote?
benchmark: https://github.com/toss/es-toolkit/tree/main/benchmarks/performance

[hi-ogawa]

I tested your yarn bench and indeed the memory usage is too much.
Probably this is because we keep all the samples array in memory and in your case, this adds up a lot since each bench is very fast and taking many samples (see "samples" column in your table).

There are a few aspects to this issue, For one, Vitest uses tinybench underneath and it currently needs accumulating all samples to compute statistics tinylibs/tinybench#61, so keeping samples are necessary at least each bench function level. You can configure { time, iterations } options https://vitest.dev/api/#bench to adjust sample size, but it looks like this is not available as a global config, but maybe we could expose a config like that.

For second, keeping samples during each bench function is not so much of a problem, but a bigger issue is that Vitest is holding all the samples of all benchmarks until the end for reporting purposes.

Not sure what would be the solution, but we probably need to do something about this. Please go ahead creating an issue.

[AndreyWV]

Disable fileParallelism helped for me and speed up tests extremely
750 files ran in 70s before - and 30s after param change.
Tests with coverage ran 10minutes in 80 files and 1m all files after param change.
Instruction https://vitest.dev/guide/improving-performance