Question about finetune_ignore commandline argument

[niki-amini-naieni]

Hi, thank you for your very useful repository! I have a couple of questions.

(1) My first question is about the finetune_ignore commandline argument here. From what I understand, it allows you to load a checkpoint but ignore some of the weights. So, for example, if I wanted to load a set of pretrained GroundingDINO weights but not load the transformer.tgt_embed.weight, I could specify: --finetune_ignore 'transformer.tgt_embed.weight'. Is my understanding correct?

(2) The Grounding DINO decoder is a transformer architecture, so why can I not just increase the number of decoder queries at inference time while using the pretrained model with 900 queries? When I try to do this, I get errors about 'transformer.tgt_embed.weight.' The structure of these parameters seems to depend on the number of queries, so when I load the pretrained checkpoint, I experience the error.

Thank you very much for any input you can provide!

[rziga]

I'm not the author, but maybe I can help with 2).

It's because of how DINO does query initialization for the decoder (see figure 2. in the paper). The queries have two parts -- one is a bounding box (shape [batch_size, 900, 4]) and the other is some latent query representation of shape [batch_size, 900, embed_dim]. The bounding box part is generated by selecting 900 locations with the highest class logits from the feature pyramid at the end of encoder and predicting the boxes. The second part is a trainable parameter of shape [900, embed_dim] which just get expanded along the batch dimension. A similar thing could be achieved by initializing both the boxes and the latent queries from the encoder outputs (see figure 5. b) in the paper). In that case, you could in theory just change the parameter at runtime to increase the number of queries, but this achieves a bit better detection performance.