Results cannot be reproduced

[Jiahao000]

Hi @Hyperwjf,

Thanks for sharing your great work! I am trying to reproduce the results based on LLaVA-1.5-7B on POPE reported in the paper. The reproduced results are as follows:

vanilla (reproduced):

LogicCheckGPT (reproduced):

However, the results reported in the paper (Table 1) are:

It seems that both vanilla and LogicCheckGPT cannot be fully reproduced. I used the official code and ran ./logiccheckgpt/check_llava.py following the instructions of LLaVA-1.5 here without changing anything else.

Am I missing something?

Thanks for your help!

[Hyperwjf]

Hi Jiahao,

Thank you for your interest in our work! We have previously focused on organizing the specific implementation code for mPLUG-Owl. Recently, due to the requirements of issue #3, we provided a preliminary implementation of LLaVA v1.5. However, we haven't had the opportunity to thoroughly review the details yet. As we are currently busy with several deadlines, I haven't been able to check the details. We plan to revisit the code in a few weeks when we have more time. We will inform you once the review is complete. Thanks for your understanding!

[Jiahao000]

Thanks for your reply. Looking forward to your review!