Question regarding the aggregation strategy in _get_attn_k()

[zhangshuoneu]

This is a very insightful piece of work! It's impressive to see robust motion segmentation and quality enhancement for dynamic point clouds achieved in a training-free manner.

I have a question regarding the implementation details. I noticed that when aggregating the cross-attention maps for a single image (referring to the _get_attn_k() function in model.py), the aggregation is performed along the query dimension rather than the key dimension.

Is there a specific reason or intuition behind this design choice? I would really appreciate it if you could share your insights. Thanks!

[rover-xingyu]

Hi @zhangshuoneu, thanks for your interest.

Intuitively, since the attention map governs the information flow from key to query, averaging along the query dimension allows us to assess how much each token in the key view contributes to the query view as a whole. This effectively highlights the spatial significance of each token in terms of its contribution.

[zhangshuoneu]

Thanks for your reply.
If I understand correctly, doing so produces an image coordinate system aligned with the key view. Given that cross-attention tokens are typically aligned with the query view, wouldn't it be more reasonable to extract features that align with the query view instead?

[rover-xingyu]

You are right. Averaging along the query dimension indeed produces a map aligned with the Key view.

Our specific goal here is to evaluate the importance of each token in terms of contribution (Key) rather than gain (Query).

[zhangshuoneu]

Thanks for the reply!
Following up on that, have you experimented with aggregating along the key dimension instead? I'm curious if that yields different performance. If so, would it be possible to share any comparisons?