How do I load a custom Whisper model (from HuggingFace)?

[athu16]

I want to load this fine-tuned model using my existing Whisper installation. I have a Python script which uses the whisper.load_model() function, but it only accepts strings like "small", "base", etc. How can I load a custom model for inference?

[jongwook]

The load_model() function also checks if the name string is a path to an existing file:

whisper/whisper/__init__.py

Lines 130 to 139 in c09a7ae

	if name in _MODELS:
	checkpoint_file = _download(_MODELS[name], download_root, in_memory)
	alignment_heads = _ALIGNMENT_HEADS[name]
	elif os.path.isfile(name):
	checkpoint_file = open(name, "rb").read() if in_memory else name
	alignment_heads = None
	else:
	raise RuntimeError(
	f"Model {name} not found; available models = {available_models()}"
	)

So you can simply supply the checkpoint path. It should still have the same format, containing a dictionary with "dims" and "model_state_dict" fields.

[anonymouscnuser]

This only load a .pt file. But in huggingface, most libraries don't provide checkpoints, but pytorch_model.bin files. And whisper library can't load them easily. Could you please offer a example that load a model from huggingface like https://huggingface.co/openai/whisper-medium or https://huggingface.co/vumichien/whisper-medium-jp ?

[phineas-pta]

you can inspire from this: #830 (comment)