Can't reproduce the OSWorld-G results

[jdubkim]

Hi thank you for releasing an amazing model!
I tried the model and it works really well.
While I was able to reproduce the results for ScreenSpot-v2, the results for OSWorld-G was only 30.1%.
I was wondering if there's something I did wrong or if something has changed in the code.
Thank you so much!

The command I tried:

In evaluation folder,

python qwen25_vllm_osworld_g_jedi.py --annotation_path ../benchmark/OSWorld-G.json --image_dir ../benchmark/images/ --model_path xlangai/Jedi-7B-1080p

[MillanK0817]

Hi,

Thank you for your feedback on this issue. We've received similar reports from other users, and we believe the problem stems from the chat template configuration in vLLM.
Please try running the model again using the Transformers. For vLLM hosting, you may try this template, which you can find here: https://huggingface.co/xlangai/Jedi-7B-1080p/blob/main/chat_template.json.
Please let us know if you still could not run it properly.