MMLU evaluation fails with Qwen3

[shaafsalman]

I'm trying to evaluate Qwen/Qwen3-30B-A3 on MMLU. I have my model deployed on vLLM on my local machine.

When I run the task normally like:
yaml{description: "mmlu:model=Qwen/Qwen3-VL-30B-A3B-Instruct,eval_split=test,subject=clinical_knowledge", groups: ["mmlu_clinical_knowledge"], priority: 1}

It either gives:

KeyError: 'choices'
File "/helm/clients/openai_client.py", line 354, in _make_chat_request
for raw_completion in response["choices"]:
KeyError: 'choices'
Or it completes the evaluation but does not show anything in raw predicted text .

So I added a parameter:
yaml{description: "mmlu:model=Qwen/Qwen3-VL-30B-A3B-Instruct,eval_split=test,subject=clinical_knowledge,increase_max_tokens=2048", groups: ["mmlu_clinical_knowledge"], priority: 4}
And ran the evals. It did run the evaluations and the answer is present in the generated text, but it was not able to detect it.

Do you have any solution to make it work?

these below are the inference with the latter config

[yifanmai]

Hi @shaafsalman, you can try using the following run entry instead:

mmlu:model=Qwen/Qwen3-VL-30B-A3B-Instruct,eval_split=test,subject=clinical_knowledge,output_format_instructions=mmlu

The run expander output_format_instructions=mmlu will add the string Answer with only a single letter. to the prompt, which works for most instruction-tuned models. If this still doesn't work, you may have to do some prompt engineering to find a prompt that works for your model (see this code for examples). Unfortunately, model behavior varies a lot between instruction-tuned models, so we do not have a universal prompt that works for all models.

[shaafsalman]

Thanks , I'll try adding output_format_instructions=mmlu.