PhenoLIP: Integrating Phenotype Ontology Knowledge into Medical Vision–Language Pretraining

| 🌐 Project Page | 📑 Paper | Github Repo | PhenoLIP | PhenoBench |

Cheng Liang^1,2, Chaoyi Wu¹, Weike Zhao^1,2, Ya Zhang^1,2, Yanfeng Wang^1,2, Weidi Xie^1,2

¹ Shanghai Jiao Tong University, ² Shanghai AI Laboratory.

The official codes for "PhenoLIP: Integrating Phenotype Ontology Knowledge into Medical Vision–Language Pretraining".

Project Structure

• Alignment/: Sub-figure level image-text alignment
• Augment/: Sub-figure level caption augmentation
• Cls/: Image classification
• Cluster/: Data clustering
• Detection/: Sub-figure detection
• Filter/: Article filtering from PubMed
• OCR/: Image content recognition
• eval/: Benchmark construction and evaluation code

Environment

You can install the code environment used for training our model.

conda create -n phenolip python==3.10
conda activate phenolip
pip install torch==2.6.0 torchvision==0.21.0

• Python: Version >= 3.10
• CUDA: Version >= 12.1
• VLLM: Version >= 0.7

Citation

If you find this work is relevant with your research or applications, please feel free to cite our work!

@misc{liang2026phenolipintegratingphenotypeontology,
      title={PhenoLIP: Integrating Phenotype Ontology Knowledge into Medical Vision-Language Pretraining}, 
      author={Cheng Liang and Chaoyi Wu and Weike Zhao and Ya Zhang and Yanfeng Wang and Weidi Xie},
      year={2026},
      eprint={2602.06184},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2602.06184}, 
}