inaccurate zero-shot inference

[askerlee]

synfundus-sel.zip

I tested the 5 images in the zip archive above. These images were selected from the SynFundus (https://github.com/parap1uie-s/SynFundus-1M) synthetic dataset, and each image comes with a caption.

The corresponding meta-information of the 5 images is:

file_md5	file_name	eyes_side	is_abnormal	is_amd	is_aon	is_crp	is_dm	is_dme	dr_grade	is_em	is_gc	is_md	is_mh	is_htr	has_lesion	is_pm	is_rvo	is_tessellated	is_treated	is_fundus	is_macular_readable	is_optic_disc_readable	is_retinal_region_readable	retinal_region_quality_score	diagnosticAdvice
8294bb6cfa7aeb46a5e5c7bf88680289	4899.png	left	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	1	1	1	1	9	疑似黄斑区域病变。
2c7070104be097670fe6e2b1fb67e185	4923.png	left	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	1	1	1	1	9	可见出血斑。
4c1149d2ec10cff917b7802ce190f10d	4977.png	left	0	0	0	1	0	0	0	0	0	1	0	0	0	0	0	0	0	1	1	1	1	9	疑似脉络膜视网膜病变，疑似黄斑区域病变。
82162645745bc1371cc2aea9ce923590	5048.png	left	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1	1	1	5	疑似高血压性视网膜病变（中度）。
108247fbe1947933a3022b74154d8010	5053.png	left	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	1	1	1	1	9	疑似青光眼。

The label file is:

Normal Healthy 正常,健康
Diabetic Retinopathy 糖尿病视网膜病变
Age-related Macular Degeneration 年龄相关性黄斑变性
Anomalies of the Optic Nerve 视神经异常
Choroidal Retinal Vascular 脉络膜视网膜血管
Diabetic Macular Edema 糖尿病黄斑水肿
Epimacular Membrane 黄斑上膜
Glaucoma 青光眼
Hypertensive Retinopathy 高血压视网膜病变
Myopia 近视
Retinal Vein Occlusion 视网膜静脉阻塞
黄斑区域病变
可见出血斑
脉络膜视网膜病变

However, all 5 images were best matched with "Myopia 近视". This is weird, as myopia (is_pm, "is pathological myopia") is all 0 for all the 5 images, and the captions don't mention myopia.

[sStonemason]

Sorry for the delayed reply. I have two messages to share with you:

1. I would recommend using a single image as both img_l and img_r as input, i.e. img_features = model(imgs,imgs,None). Also, the text information should be obtained by text_projection at patient level. This matches the design of the training phase and has been tested by me on multiple downstream tasks, which leads to optimal results, far superior to the results of just using monocular level features. However, please understand that I cannot publicize this part of the experimental results for now.
2. Noted that the image and text data you used are generated by AI models, after my testing, there is indeed such a issue as you described. In addition to our model, I have also tested other foundation models designed for fundus images, and again, I was unable to accurately achieve this zero-shot task. You can try the method I mentioned in 1. and observe if the accuracy improves. However, I think there are natural problems with AI-generated data and it is not suitable for use as model training and testing.

[sStonemason]

I think the core of the problem lies in the fact that the image features output by the model are not processed by the linear layer when monocular input is given, but the model is trained with a linear layer (see model.py for details). Therefore, when performing the zero-shot task, the image features cannot be well aligned with the text features. So in addition to the method mentioned above, you can also try just using monocular features, but it requires you to simply rewrite the code as below. This is a flaw in our design because when performing the downstream tasks of linear probing and fine-tuning, whether or not the image features pass through the linear layer doesn't make a difference to the final result, so we didn't take this into account, sorry.

def encode_image(self, img_l, img_r, mask_ratio=0):
    if img_r is None:
        if isinstance(self.visual, ModifiedResNet):
            # mask_ratio > 0 (FLIP strategy) is currently only implemented for VisualTransformer.
            vision_feature = self.visual(img_l.type(self.dtype))
            return self.left_feature_mapping(vision_feature)
        vision_feature = self.visual(img_l.type(self.dtype), mask_ratio)
        return self.left_feature_mapping(vision_feature)
    if img_l is None:
        if isinstance(self.visual, ModifiedResNet):
            # mask_ratio > 0 (FLIP strategy) is currently only implemented for VisualTransformer.
            vision_feature = self.visual(img_r.type(self.dtype))
            return self.right_feature_mapping(vision_feature)
        vision_feature = self.visual(img_r.type(self.dtype), mask_ratio)
        return self.right_feature_mapping(vision_feature)
    if isinstance(self.visual, ModifiedResNet):
        # mask_ratio > 0 (FLIP strategy) is currently only implemented for VisualTransformer.
        left_feature = self.visual(img_l.type(self.dtype))
        right_feature = self.visual(img_r.type(self.dtype))
        vision_feature = torch.cat(
            (left_feature, right_feature), dim=1)
        return self.global_feature_mapping(vision_feature), self.single_feature_mapping(
            left_feature), self.single_feature_mapping(right_feature)
    left_feature = self.visual(img_l.type(self.dtype), mask_ratio)
    right_feature = self.visual(img_r.type(self.dtype), mask_ratio)
    vision_feature = torch.cat(
        (left_feature, right_feature), dim=1)
    return self.global_feature_mapping(vision_feature), self.left_feature_mapping(
        left_feature), self.right_feature_mapping(right_feature)