About data leakage on zero-shot classification?

[LTEnjoy]

Hello!

Thanks for your great work! I have tested the zero-shot classification given your released checkpoint and it did a good performance. But I am confused that whether there exists some data leakage problem? Your model was fine-tuned on Swiss-Prot database and the DeepLoc dataset was also constructed from UniProt database. Did you do some filtering when you tested zero-shot performance?

Looking forward to your reply! Thanks in advance!

[KatarinaYuan]

Hi,
Thank you being interested in our work!

Please see the pre-training dataset

ProtST/protst/dataset.py

Line 22 in db53a76

"swiss_prot": "https://miladeepgraphlearningproteindata.s3.us-east-2.amazonaws.com/uniprotdata/uniprot_sprot_filtered.tsv",

. It does not expose test labeled data of each benchmark dataset that has not been observed during multimodal pre-training nor downstream fine-tuning.

[LTEnjoy]

Hi,

Thank you for the reply and I'll check it out!