About the performance on ScreenSpot

[zhangxgu]

Hi, thanks for your great work!
[stage2]:
According to the scripts in this repo, we train the stage2 and evaluate it on Screenspot but only get 45.6%, which is also lower than the 77.4% in the paper.
Besides, when we try the provided Aguvis-7B-720P on Screenspot, it get 79.4%, which is also lower than the 84.4% in the paper.
I wonder if the author can provide the evaluation prompt for Screenspot.
[stage1]:
Another question is that have you ever tried to evaluate the middle checkpoints like 1000-step or 2000-step on ScreenSpot during stage1 sft? According to our experiments, it seems when training 1000-step, the model lead to a very bad performance (e.g. <30%) on ScreenSpot.

many thx!

[zhangxgu]

We have trained the stage1 with 32 A100& accumulate=2 for 1 epoch (~7500step) following this repo. However, the performance on ScreenSpot is only 36%, which is still lower than the 81.8% in paper.
We wonder if you can give us some advices for stage1 training.

[Timothyxxx]

Hi, thanks for the question! I can make sure the results can be replicated, as I tried multiple times and the same as other teams. And the aguvis 7b 720p is valid as well.

I highly suspect that something wrong happened to your evaluation code. Please be careful with the all and os aguvis introduced and prompt the correctly.