Paper reproduction question

[Jiang0903]

Hello, your work is fantastic, thanks for open-sourcing it. I encountered an issue while testing the released weights. Could you please clarify which weights should be used as the visual encoder when testing downstream_commentary_none_open.pth?

I tried using both pretrained_both.pth and pretrained_contrastive.pth as the visual encoder weights, but the performance was relatively poor in both cases. Below are the performance metrics:

pretrained_both.pth + downstream_commentary_none_open.pth:
{'BLEU-1': '21.895', 'BLEU-4': '3.428', 'METEOR': '20.290', 'ROUGE-L': '20.156', 'CIDER': '13.593'}

pretrained_contrastive.pth + downstream_commentary_none_open.pth:
{'BLEU-1': '22.251', 'BLEU-4': '3.037', 'METEOR': '20.946', 'ROUGE-L': '18.706', 'CIDER': '8.955'}

[jyrao]

Hi, thanks for your following and your question, here are something to clarify:

1. Which test set are you using: if your are using the testset of MatchTime, you should compare the metric numbers with Table2， if you are using SoccerReplay-1988, you should compare it with Table 5.
2. What is your input in your terminal, since different hyperparamters would represent different usage of our inference code.

To be noted, pretrained_both.pth and pretrained_contrastive.pth are both for visual encoder. As for commentary generation, you only have to input this hyperparameter with the downstream_commentary_none_open.pth:

parser.add_argument("--ckpt_path", type=str, default="FILE_PATH_TO_MODEL_CHECKPOINT")

since the dataset directly get the SigLIP features.

For some difference between none_open and all_open, you can refer: #3

[Jiang0903]

Thank you for your reply.

1. I am conducting my tests on the MatchTime test set, i.e. SN-Caption-test-align.

2. Regarding the hyperparameters, both open_visual_encoder and open_llm_decoder are set to False. The only other two changes I made were: I updated ckpt_path to the path of downstream_commentary_none_open.pth, and when initializing the MatchVoice_model_all_blocks class, I set the visual_encoder_checkpoint to the path of either pretrained_both.pth or pretrained_contrastive.pth.

Based on your response, I believe I understand what caused the performance degradation. When testing downstream_commentary_none_open.pth, I should use SigLIP as the visual feature extractor, rather than the visual encoder weights you provided.

However, I still have one question: Why didn't you include weights in the open-sourced release for a model trained on MatchVoice with both the frozen MatchVision visual encoder and the frozen LLM decoder ? I think that configuration is exactly what would be expected from a checkpoint named downstream_commentary_none_open.pth.

[jyrao]

Sorry that I think I might get it wrong after so long time, I looked back to my original codebase that it should be with the couple of pretrained_both.pth + downstream_commentary_none_open.pth. (This should be the final answer, sorry for confusing you.)

Also, as for your question, the point was that I didn't fintune the parameters in MatchVision for the downstream_commentary_none_open.pth training, so I don't want to make it duplicated.

As a result, seems you are doing correctly of your previous command, I might said there would be some uncertainties:

1. Make sure that the videos are cropped 30s (timestamp at middle, 15s before, 15s after), make them in the name of https://github.com/jyrao/UniSoccer/blob/main/train_data/json/MatchTime/classification_test.json shows.
2. Make sure the language decoder you are using is exactly "Meta-Llama-3-8B-Instruct" rather than "Meta-Llama-3-8B"

I recommend you send the predict result examples, that I might get the sense of some common situations before.

Best,
Jiayuan

[IsmaelElsharkawi]

Hi,
I have a question related to the recreation of the commentary results, why do we need to specify two different checkpoints, pretrained_both.pth AND downstream_commentary_none_open.pth, shouldn't downstream_commentary_none_open.pth have the needed weights for the visual encoder as well?

[Jiang0903]

@IsmaelElsharkawi
I guess when training the downstream_commentary_none_open.pth, the visual encoder and LLM decoder were frozen, so finally it did not include the weights of visual encoder when saving.