Issue with Dataset Evaluation and Accuracy

[komi786]

Hello,
I used the evaluation code provided and used model_id = https://huggingface.co/cambridgeltl/SapBERT-from-PubMedBERT-fulltext
It shows lower accuracy for various datasets . For AAP dataset the accuracy @1 is 0.58% which is close to one reported in paper for vanilla SAP without further fine tunning with pubmed or biosyn.
Are further fine tunned model available ? Or this model SapBERT-from-PubMedBERT-fulltext is the fine tunned version?

[Gbone3176]

During my testing process, I also tested this model from HF, and its performance on NCBI, BC5CDR-d, and BC5CDR-c was significantly lower than the performance reported in the paper. I am quite puzzled by this. Looking forward to replying, thanks a lot~~

[LakshanKarunathilake]

i htink the evaluation dataset dict is smaller than MSH or UMLS complete dictionary. I am also trying to re-produce the results but it is no where near the reported results. I dont understand the logic behind the evaluation dictionary generation