Checkpoints explanation

[arshdeep-liat-ai]

Could you mention a brief description of the various checkpoints especially just for inference task:
downstream_commentary_none_open.pth vs downstream_commentary_all_open.pth
I was able to run the classification task but for the commentary generation, the results I am getting are repetitive words (hallucination). As I can see in the other issues as well, it would be great if the commentary generation inference instructions could be more clear.

[jyrao]

Thanks for the comments! downstream_commentary_all_open.pth file requires opening "visual_encoder" and "llm_decoder" hyperparameters when you are inferencing commentary generation tasks.

Otherwise for downstream_commentary_all_open.pth you can just leave them as defaults. Maybe #3 helps with that but I'm not sure if you are meeting the same hallucination with that.

[arshdeep-liat-ai]

Thanks for the response. I did try the hyperparameters variations mentioned in the github issues but the results for commentary inference are hallucinating.

Output on one of the videos: Diego is a good one. The use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use of the use

[jyrao]

That's weird though, may I know your entire terminal instruction with all the parameters? I saw such hallucinating before when user use "Meta-LLama3" rather than "Meta-LLama3-Instructions" version. There must be some subtle mistake during inference.