How to use FP16 model precision for inference ?

[jerin-scalers-ai]

I believe the default model precision of whisper models if FP32. so how can i use FP16 model using openai-whisper package.

[glangford]

--fp16 True

fp16 is true by default. See whisper --help.

[jerin-scalers-ai]

Thanks. I am using python openai-whisper package for loading and transcribing the audio. The --fp16 flag is part of transcribe() but the actual model is loaded into device using load_model().

[glangford]

Right. As @phineas-pta points out, the audio is loaded in fp16 not the model (and presumably mixed precision is used by pytorch).

There is no parameter for fp16 in load_model, see here

whisper/whisper/__init__.py

Line 99 in e58f288

def load_model(

[jerin-scalers-ai]

Got it.
I was printing model dtype and i could see the all the layers with float32 as shown below. I believe the default precision is Float32

Parameter: encoder.conv1.weight, dtype: torch.float32
Parameter: encoder.conv1.bias, dtype: torch.float32
Parameter: encoder.conv2.weight, dtype: torch.float32
Parameter: encoder.conv2.bias, dtype: torch.float32
Parameter: encoder.blocks.0.attn.query.weight, dtype: torch.float32
Parameter: encoder.blocks.0.attn.query.bias, dtype: torch.float32
Parameter: encoder.blocks.0.attn.key.weight, dtype: torch.float32
Parameter: encoder.blocks.0.attn.value.weight, dtype: torch.float32
Parameter: encoder.blocks.0.attn.value.bias, dtype: torch.float32
Parameter: encoder.blocks.0.attn.out.weight, dtype: torch.float32
Parameter: encoder.blocks.0.attn.out.bias, dtype: torch.float32
Parameter: encoder.blocks.0.attn_ln.weight, dtype: torch.float32

[ex3ndr]

tiny one is float16 though

[phineas-pta]

fp16 is a misleading option, it loads the audio in float16 not the model, see #622

for truly float16 use other interfaces like transformers whisper.cpp faster-whisper

[jerin-scalers-ai]

Thanks for the response. I will try out some of the suggested options.