flash attention 2, batch size, etc.?

[silvacarl2]

does anyone have any experience on using flash attention 2 and different batch size or other parameters like that to make whisper as fast as possible?

carl

[phineas-pta]

https://github.com/Vaibhavs10/insanely-fast-whisper

the author working at huggingface so he should be very well informed

[silvacarl2]

i checked that. i am not entirely sure what technique can be used to speed up whisper in general.

https://github.com/igorcosta/insanely-fast-whisper-cli simply says this:

import torch
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition",
"openai/whisper-large-v2",
torch_dtype=torch.float16,
model_kwargs={"use_flash_attention_2": True},
device="cuda:0")

outputs = pipe("<FILE_NAME>",
chunk_length_s=30,
batch_size=24,
return_timestamps=True)

outputs["text"]

i was looking for examples that call whisper directly versus going through pipeline.

[phineas-pta]

that's exactly what u looking for right ? batch size, flash attention 2, float16

if u want something else maybe onnx or tensorrt ?