Foul Action Classification Prediction

[ZhouyouDong1]

Hello, your work is very valuable! I would like to ask how to use your pre-trained model to perform inference on the SoccerNet-MVFouls dataset, thanks!

[jyrao]

That is a good question that literally, in our work we compare the performance on MVFouls with those baseline models in the original paper[1]. In that paper, the basic logic is like they pretrained both the visual backbone and classification head.

As you can see in the paper, we were using the numbers in the original paper as the recorded experiment metric value. However, in our experiment, our purpose was to test the soccer understanding ability on other different downsteram tasks of the pretrained MatchVision and only pretrain the downstream task (Multi-view Fouls).

We didn't share this part of the code yet, since the performance wasn't good enough and it's almost the same as the classification pretraining code, but still several points to be noticed apart from our paper:

1. Our framework is designed for 30s video clips, but its internal logic was to crop 30 frames (1FPS) to fetch the frames and encoder them together, so the spatial-temporal attention was pretrained for 1FPS interval and it could run, but not work perfectly on MVFouls videos (since they are always around 5s, but still get 30 frames).

2. During training, what you only need to do is to use frozen MatchVision and initialize the downstream head (parameters in the paper), but there are some thing you need to focus was that there were some labels missing in the dataset, so in your dataset part you need to pay some attention on that.

Hope my answer could somehow help you understanding our paper, especially the MVFouls parts. Also, in another finished work of us (released soon), we made some test on VARS benchmark on current VLMs using multi-agent attempt, if you are interested in that you can kept following that work, hopefully we would release it in around 2 weeks.

[1] Held J, Cioppa A, Giancola S, et al. VARS: Video assistant referee system for automated soccer decision making from multiple views[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 5086-5097.