v3.0

With this major release, we're splitting the ML.ENERGY benchmark from the ML.ENERGY leaderboard into its own project. The benchmark itself was revamped so that it's much easier to run.

Features

• Server-agnostic LLM/MLLM benchmarking (assumes OpenAI Completions & Chat Completions API)
• LLM/MLLM benchmark supports both Docker and Singularity container runtime
• LLM/MLLM steady state tracking without requiring server image modifications
• Diffusion benchmark upgraded to use xDiT as the runtime, with Ulysses and Ring Attention degree sweeping
• Automated Slurm/Pegasus job generation from configs & sweep range specifications (under configs/)
• A range of automated sanity checks on benchmarking result data

More details on using the benchmark can be found under docs/.

This v3.0 release was built on top of the v2.0 benchmark. Major contributors of v3.0 are @jaywonchung, @majunze2001, and @ruofan-wu. Thanks!