about spynet_pretrained in BasicVSR?

[luoyinTT]

what dataset is used to pretrain spynet ??

[ckkelvinchan]

SPyNet is borrowed from here. You may refer to this repo for more details.

[innerlee]

The source should be added to the doc https://mmediting.readthedocs.io/en/latest/restorers_models.html#basicvsr-cvpr-2021

[leiwen83]

Hi,

I wonder to know that whether new SOTA optical flow like RAFT would benefit BasicVSR, since it provide better optical flow accuracy.

[ckkelvinchan]

Hi,

I wonder to know that whether new SOTA optical flow like RAFT would benefit BasicVSR, since it provide better optical flow accuracy.

We haven't tried that before, but there is a possibility that it could lead to better results :) Another reason we used SPyNet is its speed.

[leiwen83]

I see,

Thx

[leiwen83]

Another question for the spynet module.
In basicVSR, spynet is trained together with other feature extraction part over the LR/HR pair. So after training complete, can spynet module behavior like the optical flow function, would its optical flow esitimation capability be downgrade?
In the video inference, like A,B,C,D sequence. in the first round, basicVSR take [B,C]&&[B,A] to compute the optical flow, meanwhile in the second round, it would take [C,D]&&[C,B] to compute. is there any relationship between the result of [C,B], and [B,C] as input to spynet? If basicVSR traning would not destory spynet optical flow function, would [C,B] [B,C]'s spynet result could be simply mapped so that the computation cost would be elimilated for the second round of basicVSR inference?

[ckkelvinchan]

The finetuned SPyNet will produce outputs that are very similar to optical flow, but they will have slight differences. If you want to save computational cost, I suggest you to fix SPyNet and then use the approach you mentioned above.