Is it possible to do model adaptation?

[talentcoco]

Is it possible to add a corpus of specific words, names of places etc, technical jargon to improve accuracy?

[jongwook]

That is possible to some degree using prompt engineering; I added the --initial_prompt option in 2037b65, and you could do

whisper audio.mp3 --initial_prompt "So we were just talking about DALL·E"

to make the model more likely to output "DALL·E" rather than "Dolly".

[talentcoco]

Thanks so much!! Is it possible to "boost" specific words? It will be very helpful as I don't know what the initial prompt will be. Ideally I would like to "boost" a few likely words e.g. "DALL-E", "WALL-E", "R2-D2", and "HAL9000"

[jongwook]

Currently there's no interface for this other than giving the initial_prompt like the above; you could hack something with logit biasing, that effectively boosts the predicted probability of certain tokens. The LogitFilter class is designed to support this:

whisper/whisper/decoding.py

Lines 366 to 370 in 62fe7f1

	class LogitFilter:
	def apply(self, logits: Tensor, tokens: Tensor) -> None:
	"""Apply any filtering or masking to logits in-place