Finetune ErrorInfo: TypeError: decode() got an unexpected keyword argument 'skip_special_tokens'

[jxz2021114]

When I used the official finetuning code sample to finetune the whisper model, I met this error information:
Traceback (most recent call last):
File "main.py", line 102, in
text = wtokenizer.decode(token, skip_special_tokens=False)
File "/ldata/crj/anaconda3/envs/whisper/lib/python3.8/site-packages/whisper/tokenizer.py", line 166, in decode
return self.encoding.decode(token_ids, **kwargs)
TypeError: decode() got an unexpected keyword argument 'skip_special_tokens'

Here is the code context.
86 # Data loader
87 woptions = whisper.DecodingOptions(language="ja", without_timestamps=True)
88 wmodel = whisper.load_model("base")
89 wtokenizer = whisper.tokenizer.get_tokenizer(True, language="ja", task=woptions.task)
90
91 # Confirm Dataloading
92 dataset = JvsSpeechDataset(eval_audio_transcript_pair_list, wtokenizer, SAMPLE_RATE)
93 loader = torch.utils.data.DataLoader(dataset, batch_size=2, collate_fn=WhisperDataCollatorWhithPadding())
94
95 for b in loader:
96 print(b["labels"].shape)
97 print(b["input_ids"].shape)
98 print(b["dec_input_ids"].shape)
99
100 for token, dec in zip(b["labels"], b["dec_input_ids"]):
101 token[token == -100] = wtokenizer.eot
102 text = wtokenizer.decode(token, skip_special_tokens=False)
103 print(text)
104
105 dec[dec == -100] = wtokenizer.eot
106 text = wtokenizer.decode(dec, skip_special_tokens=False)
107 print(text)
108
109
110 break

Have anyone seen this error before? How can I solve this?

[glangford]

TypeError: decode() got an unexpected keyword argument 'skip_special_tokens'

It looks like you are using a whisper tokenizer here (ie. as defined in this repository)

89 wtokenizer = whisper.tokenizer.get_tokenizer(True, language="ja", task=woptions.task)

But get_tokenizer from this whisper does not have a skip_special_tokens option, hence the error

102 text = wtokenizer.decode(token, skip_special_tokens=False)

Should wtokenizer instead be a WhisperTokenizer from transformers ? I don't use the transformers library so this is just a possibility to review.

HuggingFace - whisper

• https://huggingface.co/docs/transformers/en/model_doc/whisper

[jxz2021114]

Hi,
Thank you for your suggestion!
When I tried your solution, I met new bugs. This is because whisper.tokenizer is quite different from transformers.WhisperTokenizer. Firstly, transformers.WhisperTokenizer has no parameter 'multilingual‘, which is set to be True in the whisper.tokenizer sample code.
Secondly, they also have different attributes, which leads to an AttributeError:
Traceback (most recent call last):
File "main.py", line 99, in
for b in loader:
File "/ldata/crj/anaconda3/envs/whisper2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 640, in next
data = self._next_data()
File "/ldata/crj/anaconda3/envs/whisper2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 684, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/ldata/crj/anaconda3/envs/whisper2/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/ldata/crj/anaconda3/envs/whisper2/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/ldata/crj/jiangxz/whisper_finetune/sdaa/data_utils.py", line 33, in getitem
text = [*self.tokenizer.sot_sequence_including_notimestamps] + self.tokenizer.encode(text)
AttributeError: 'WhisperTokenizer' object has no attribute 'sot_sequence_including_notimestamps'
Therefore, I don't think transformers.WhisperTokenizer would be an easy to implement alternative. Thanks.