The Intermediate checkpoint can't be load by from_pretrained during Fine tune Whisper

[shuaijiang]

I fine tune the whisper using code https://github.com/shuaijiang/Whisper-Finetune/blob/master/finetune_all.py

# 定义训练器
trainer = Seq2SeqTrainer(args=training_args,
                         model=model,
                         train_dataset=train_dataset,
                         eval_dataset=test_dataset,
                         data_collator=data_collator,
                         tokenizer=processor.feature_extractor,
                         )
model.config.use_cache = False
trainer._load_from_checkpoint = load_from_checkpoint

# 开始训练
trainer.train(resume_from_checkpoint=args.resume_from_checkpoint)

# 保存最后的模型
trainer.save_state()
if training_args.local_rank == 0 or training_args.local_rank == -1:
    model.save_pretrained(os.path.join(output_dir, "checkpoint-final"))

I save the last 5 checkpoint by setting save_total_limit=5 in the Seq2SeqTrainingArguments
To do the inference, I load the Intermediate model by WhisperForConditionalGeneration.from_pretrained(ckpt_path) , it says Some weights of the model checkpoint at [checkpoint_path] were not used when initializing WhisperForConditionalGeneration: [list of weights]. And it doesn't work, the model only generates 0

However, When I load the final model, which saved by model.save_pretrained(), It works.

Thanks for any advice!

[psiyou]

Hi @shuaijiang, having the same issue. have you found any solution??
thanks