Return peak memory used in the stats

[aukejw]

LMStudio already reports memory usage for models in the UI. Is it possible to return peak memory usage in the stats? It would be useful to know whether inference would fit on local machines.

Main reason I'm asking is that I'm benchmarking LMStudio vs MLX, and counting LMStudio-related process gives a distorted view of reality:

Thanks!

[JaDenis]

I'm very interested in your findings, could you share?

[aukejw]

Benchmark results for a few local models are here: https://aukejw.github.io/mlx_transformers_benchmark/

Currently I'm assuming all processes with the name LM studio are relevant, and for mlx I use its peak memory function. I don't run benchmarks if loading the model doesn't fit alongside system processes.

Both are approximations, it's still hard to say whether a model will run on Apple silicon without using swap if other processes are running already (eg you're using the browser and running inference).

Also see this issue:
aukejw/mlx_transformers_benchmark#1