about similarity map

[KN1GHT9]

Congratulations on your paper being accepted to CVPR!
I would like to ask about the visualizations in Figure 1 (b), (c), and (d). How did you compute the similarity between the token and the image pixels to generate the heatmaps? From my understanding, the similarity is typically computed between tokens, so I am not sure how it was mapped back to the pixel level.

[rui-qian]

Congratulations on your paper being accepted to CVPR! I would like to ask about the visualizations in Figure 1 (b), (c), and (d). How did you compute the similarity between the token and the image pixels to generate the heatmaps? From my understanding, the similarity is typically computed between tokens, so I am not sure how it was mapped back to the pixel level.

Thank you. You can refer to this file: model/READ.py#L600”, model/llava/model/multimodal_encoder/clip_encoder.py#L85C1-L85C17. Also, we will release a more efficient and cleaner implementation soon. Stay tuned!

[KN1GHT9]

Thank you for your reply. I am new to this field, and I would also like to ask: are the grid points g_x and g_y the center coordinates of a grid?”

[rui-qian]

Thank you for your reply. I am new to this field, and I would also like to ask: are the grid points g_x and g_y the center coordinates of a grid?”

Yes, an image can be viewed as a collection of discretized pixels, from which continuous coordinates can be reconstructed via interpolation.

[rui-qian]

Congratulations on your paper being accepted to CVPR! I would like to ask about the visualizations in Figure 1 (b), (c), and (d). How did you compute the similarity between the token and the image pixels to generate the heatmaps? From my understanding, the similarity is typically computed between tokens, so I am not sure how it was mapped back to the pixel level.

Thank you. You can refer to this file: model/READ.py#L600”, model/llava/model/multimodal_encoder/clip_encoder.py#L85C1-L85C17. Also, we will release a more efficient and cleaner implementation soon. Stay tuned!

We recently released a high-performance framework and recommend using it for training: UGround.