Question about "enhance_scores" and Attention Scaling

[jibin86]

Hi,

Thank you for sharing this project! I have a few questions about the implementation details and would appreciate your clarification:

1. In the code, I noticed that get_enhance_weight() seems to act like a “temperature” parameter mentioned in the paper, but I couldn’t find any reference in the paper to multiplying mean_scores.mean() by num_frames.

   enhance_scores = mean_scores.mean() * (num_frames + get_enhance_weight())

   • Why is mean_scores.mean() used here? What exactly is being averaged, and what does it represent?
   • How does this averaging influence the final enhancement process?
   • What is the rationale behind multiplying by num_frames, and how does this relate to the method described in the paper?

2. I also noticed that enhance_scores is multiplied by the attention output, and it is mentioned that this improves temporal consistency. Could you explain the theoretical or intuitive reasoning behind this? Specifically, why does scaling the attention output with enhance_scores contribute to better temporal consistency?

Any insights you can share would be greatly appreciated. Thank you in advance for your time!

[yangluo7]

1. mean_scores.mean() indicates the mean of non-diagonal elements of the attention map (Equation 6 in the paper), which measures the intensity of cross-frame attention (CFI). Multiplying CFI with num_frames aims to keep the final CFI_enhanced to be around 1, which will not affect the original attention output much in the residual connection, for moderate enhancement. The $\tau$ in the paper is calculated as(num_frames + get_enhance_weight()) instead of just get_enhance_weight(). To prevent possible confusing points, we have revised the paper to align it with the code.

2. if enhance_scores is high, meaning more cross-frame information in this attention output, we will increase the ratio of this attention output in the residual connection to push the model to consider more global information, which contributes to better temporal consistency.