mmlu-pro review is much worse

[ABCDabcde1098234]

Your work is great, but I have a question. I used your code to evaluate mmlu-pro, using the same model, and https://github.com/TIGER-AI-Lab/MMLU-Pro There is a significant difference in the evaluation scores, may I ask what is going on? Looking forward to your reply.

[MXueguang]

Hi @Dyw08,

Sorry about the late reply. I missed the issue.
I am not sure if I get your question correctly, do you mean with the same model, using the code provided in this repo and using the code provided in the original MMLU-Pro, there is a significant difference?

If that's the case, it's most likely the original MMLU-Pro evaluation is evaluated with 5 shot prompt, while in the evaluation code here, it directly answer without fewshot example.