Can RADIO be text encoded like clip? How to do it

[ChaojunZ]

Can RADIO encode text directly? If not, use the clip adapter and what to do. thank you

[gheinrich]

Hello, yes you can take a look at examples/zero_shot_imagenet.py for an example showing how to encode an image using RADIO's CLIP adapter, how to encode text using the corresponding language model, as well as how to compute similarity between the two.

[ChaojunZ]

Hello, yes you can take a look at for an example showing how to encode an image using RADIO's CLIP adapter, how to encode text using the corresponding language model, as well as how to compute similarity between the two.examples/zero_shot_imagenet.py

Thank you for your timely reply. I have completed this step. But when I extracted the RADIO intermediates using outputs = model.forward_intermediates(images, indices=[7, 15, 23, 31]), I got the following error：AttributeError: 'RADIOModel' object has no attribute 'forward_intermediates'
I used nvidia/RADIO-B

[ChaojunZ]

您好，是的，您可以查看一个示例，该示例显示了如何使用 RADIO 的 CLIP 适配器对图像进行编码，如何使用相应的语言模型对文本进行编码，以及如何计算两者之间的相似性。examples/zero_shot_imagenet.py

Hello, I have another problem. The size of the image I input is (384*384), which is [1, 576, 768] after encoding through PRDIO. Then I changed a hook way to obtain the features of the middle layer, but the extracted dimension is [1, 584, 768]. Let me see what the middle layer does with the size of the image.
self.layer_indexes = [3, 7]
self.layers = []
for l in self.layer_indexes:
self.radio_model.model.blocks[l].register_forward_hook(lambda m, _, o: self.layers.append(o))