How to pre-install a Huggingface transformer model

[bpben]

Hi all,
I was digging around for this, but couldn't seem to find an answer. I'm trying to pre-download a huggingface model for use in the spacy training workflow. The train CLI appears to download any huggingface model available on Hub (example: https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT) just based on the string name of the model (example: emilyalsentzer/Bio_ClinicalBERT). It appears to use the recommended installation:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

However, if I run this ahead of time, the spacy train workflow doesn't seem to find it and tries to download it again. Is there a way to pre-download a huggingface model in a way that spacy train can find it later? Let me know.

Thanks for all ya'll do!

[adrianeboyd]

spacy train shouldn't download it again (well, that's not entirely true, I think it does download it again if you have internet access and there are updates), so double-check that you haven't modified the cache settings/paths in any of the environments you're working in, or that there haven't been updates upstream since the last time you used it.

Internally spacy is calling the same methods as above, so there's nothing custom about how it's getting loaded from huggingface in the background.

You can also download/save it to a local directory and use that in place of emilyalsentzer/Bio_ClinicalBERT above, and then you know for sure that it's just loading from the local path AND that it doesn't get modified if there's an external update. (I think we should add in a revision setting in the config so you can specify the git revision for reproducibility.)

[bpben]

Thanks for this! I realized what I did wrong, I had only initialized the tokenizer! It works great (as usual) - thanks!