👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Thank you for your work! We ran validation using the same build environment as last time, but the results seemed a bit unusual, which could be due to a problem with the model inference process. Here are the reproduction steps and outcomes for your reference.

We compiled and installed your PR inside the vllm-openai:v0.10.0 Docker image via pip install --no-build-isolation /tmp/vllm_patched/.

The deployment command was the same as in the previous PR, just with --enforce-eager removed:

python3 -m vllm.entrypoints.api_server --model /data/xxx/model/MiniMax-Text-01/ --tensor-parallel-size 8 --trust-remote-code --quantization experts_int8 --max_model_len 8192 --dtype bfloat16 --no-enable-prefix-caching

Here are the test results:

For gsm8k:

python3 bench_other.py --num-questions 500 --num-shots 5 --backend vllm --port 8000 --host http://127.0.0.1
# ...
Accuracy: 0.010
Invalid: 0.018
Latency: 202.115 s

For mmlu:

python3 bench_other.py --nsub 200 --backend vllm --port 8000 --host http://127.0.0.1
# ...
Total latency: 1405.775
Average accuracy: 0.801

Upon checking the model's output for GSM8K, we noticed a significant number of extra newlines and garbled characters, indicating an abnormal output format.

On the other hand, for the MMLU benchmark, which only requires a single letter as the answer, the accuracy is only slightly lower than normal. We suspect this simpler output format might be masking some underlying issues that are more apparent in the GSM8K results.

@rogeryoungh Thanks for the eval. There must be some bug that isn't being hit when I run with the tiny model. I will take another look at it.

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @tdoublep.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

I was able to reproduce the bad lm_eval results and dug into what is going on here.

The problem is related to implementation of the rotary embedding for this model (it is not compatible with torch compile). I've replaced it with the call to get_rope that the other models use. If there was any particular reason why a custom implementation (e.g., MiniMaxText01RotaryEmbedding) was needed, please let me know. I took a quick look through the code and couldn't see one.

I deploy the model as follows:

VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve MiniMaxAI/MiniMax-Text-01 \
	--tensor-parallel-size 8 \
	--trust-remote-code \
	--quantization experts_int8  \
	--max_model_len 4096 \
	--dtype bfloat16 \
	--gpu-memory-utilization 0.95 \
	--no-enable-prefix-caching

and then I run eval with:

lm_eval   --model local-completions   \
	--model_args base_url=http://localhost:8000/v1/completions,tokenizer=MiniMaxAI/MiniMax-Text-01 \
	--tasks gsm8k  \
	--batch_size 128 \
	--num_fewshot 5 \
	--limit 500

which now produces the expected output:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.898|±  |0.0135|
|     |       |strict-match    |     5|exact_match|↑  |0.892|±  |0.0139|

@rogeryoungh Could you give it another try on your end?

After pulling in latest changes from main, we can also now deploy with the default FlashAttention backend (instead of FlashInfer):

VLLM_USE_V1=1 vllm serve MiniMaxAI/MiniMax-Text-01 \
	--tensor-parallel-size 8 \
	--trust-remote-code \
	--quantization experts_int8  \
	--max_model_len 4096 \
	--dtype bfloat16 \
	--gpu-memory-utilization 0.95 \
	--no-enable-prefix-caching

The gsm8k eval above now produces:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.904|±  |0.0132|
|     |       |strict-match    |     5|exact_match|↑  |0.900|±  |0.0134|

Great work! I have verified the changes, and the implementation now works as expected. On GSM8k the accuracy is 0.908, and on MMLU the average accuracy is 0.847.

Deployment command:

VLLM_ATTENTION_BACKEND=FLASHINFER VLLM_USE_V1=1 python3 -m vllm.entrypoints.api_server \
  --model /data/xxx/model/MiniMax-Text-01/ \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --quantization experts_int8 \
  --max_model_len 4096 \
  --dtype bfloat16 \
  --no-enable-prefix-caching

GSM8k test:

Accuracy: 0.908
Invalid: 0.000
Latency: 203.588 s

MMLU test:

Total latency: 1489.795
Average accuracy: 0.847

I was able to reproduce the bad lm_eval results and dug into what is going on here.

The problem is related to implementation of the rotary embedding for this model (it is not compatible with torch compile). I've replaced it with the call to get_rope that the other models use. If there was any particular reason why a custom implementation (e.g., MiniMaxText01RotaryEmbedding) was needed, please let me know. I took a quick look through the code and couldn't see one.

Thanks a lot for the fix! The custom implementation didn’t have any special reason — it was just how it was written back then. Really appreciate your improvement.

I also retested with the default FlashAttention backend. On GSM8k, the accuracy was 0.904, and on MMLU the average accuracy was 0.847. Everything looks good now.