Using Vlm to generate overall information has some problems

[Hotzerara]

According to the paper, this VLM is trained in two stages. However, I noticed that only one checkpoint is available on Hugging Face. In 1_vlm_demo.py, this single checkpoint appears to be used for both Stage 1 and Stage 2.
I attempted to run 1_vlm_demo.py and checked the Stage 1 output in the /demo folder. The performance seems suboptimal, with many outputs failing to align with the prompt templates.
Is this result expected? Additionally, is it possible to obtain the fine-tuned model for Stage 1 separately?

[ziangcao0312]

Our VLM is not a two-stage model; it is designed for multi-turn conversation. We have validated the demo results, and others have successfully run our code. We kindly ask you to review it again.

[Hotzerara]

I prefer to only use VLM to generate the overall_information text for the image output and utilize the text information to do some additional tasks. Therefore, I only focused on the output of the first round of dialogue. The final output of the 3D assets was indeed normal. However, here I noticed that the output results of overall_information often have some incorrect situations compared to expectations, as shown in some of the output.txt files I provided below.
2_output.txt
3_output.txt

[ziangcao0312]

If the output is already abnormal at that step, the final result is unlikely to be correct. Were these results obtained by running the demo on your own images, or on the images we provided? The model may fail when given noisy in-the-wild images.

[Hotzerara]

I revisited the code and found that the issue likely stems from changing the attention calculation. I used eager mode to capture the attention matrix during inference, causing the output instability. Prior to this, did you test running inference using eager mode? Thanks for the help.