inference using int8 or other methods?

[silvacarl2]

has anyone tried doing whisper inference using either int8 or 4-bit?

[phineas-pta]

see #454

[silvacarl2]

from that post, key take aways that seem correct:

this would only be useful if you was transcribing a large amount of audio

yup. for less audio, the difference is much smaller in speed up.

This library doesn't work with GPUs because PyTorch doesn't support Dynamic Quantization with GPUs on CUDA. Your best bet would >be to try something like TensorRT.

have not tried it yet using TensorRT vs PyTorch, but not sure overall it is worth pursuing anyways bceause the payback might not be there.

The OpenAI whisper model is basically just a Transformer model with Mel spectrogram inputs which are fed into a CNN and then the >Transformer component and then the predicted tokens are outputted. Dynamic Quantisation doesn’t support CNNs right now so only >the linear layers within the Transformer are quantised using this method.

yup. so the only part being affected is Transformer layer, again not seeing very significant performance increases on shorter audio.

[silvacarl2]

ok, so this is cool but did not make a hige speed difference.

however, if it makes a GPU memory footprint difference, that would be great.

have any seen any examples which can reduce the memory footprint of whisper models?

[phineas-pta]

https://github.com/guillaumekln/faster-whisper