Load Biobert pre-trained weights into Bert model with Pytorch bert hugging face run_classifier.py code

[sheetalsh456]

These are the steps I followed to get Biobert working with the existing Bert hugging face pytorch code.

1. I downloaded the pre-trained weights 'biobert_pubmed_pmc.tar.gz' from the Releases page.

2. I ran this command to convert the tf checkpoint to pytorch model

python pytorch-pretrained-BERT/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py --tf_checkpoint_path="biobert/pubmed_pmc_470k/biobert_model.ckpt.index" --bert_config_file="biobert/pubmed_pmc_470k/bert_config.json" --pytorch_dump_path="biobert/pubmed_pmc_470k/Pytorch/biobert.model"

This created a file 'biobert.model' in the specified path.

3. As mentioned in this link , I compressed 'biobert.model' created above and 'biobert/pubmed_pmc_470k/bert_config.json' together into a biobert_model.tar.gz

4. I then ran the run_classifier.py of hugging face bert with the following command, using the tar.gz created above.

python pytorch-pretrained-BERT/examples/run_classifier.py --data_dir="Data/" --bert_model="biobert_model.tar.gz" --task_name="qqp" --output_dir="OutputModels/Pretrained/" --do_train --do_eval --do_lower_case

I get the error

'UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte' 

in the line

tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)

Am I doing something wrong?

I just wanted to run run_classifier.py code provided by hugging face with biobert pretrained weights in the same way that we run bert with it. Is there a way to do this?

[MeRajat]

You can first convert these weights using https://github.com/MeRajat/SolvingAlmostAnythingWithBert/blob/master/convert_to_pytorch_wt.ipynb this notebook then try again.

[sheetalsh456]

Yeah as I've mentioned above (2), I converted the tf checkpoint to pytorch model.

[MeRajat]

Last time i checked (maybe this isn't the case now not sure) there was some issues with the pytorch-pretrained-BERT version with biobert weights. Please use the one that i have shared to make it compatible.

[datduong]

Hi, to follow up on this issue. Can you clarify what is the last function in your notebook convert_to_pytorch_wt.ipynb ? In the last part of this notebook, you define a function convert_tf_checkpoint_to_pytorch. Are we supposed to use this function instead of https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py ? Thanks.

[happypanda5]

@sheetalsh456 I am having that error

'UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte'

for InferSent sentence encoder from Facebook. I wonder if you have found a solution to it