Guillemets inside result text

[apwhitelaw]

I have been playing around with whisper and I know some French so I tried saying a sentence. One of the times I tried, the result looked like this

Aujourd'hui, je fais « J'ai visé pour le test ».

but what I actually said was

Aujourd'hui, je vais reviser pour le test.

For those that don't know, the characters « and » are called guillemets, and are basically quotation marks used in French (and some other languages, I believe).

Is whisper adding those in because it thought it was a quotation? Or do those characters mean something specific to whisper?

It hasn't happened since, so I wasn't sure if it was only using those characters because it is French or if those characters mean something specific and could appear with any language.

[jongwook]

Thanks for reporting, this happened because it (wrongly) predicted that it was a quotation, which is possible because there must've been many examples of French text in the training data which had guillemets. We tried to avoid this by suppressing tokens corresponding to these letters:

whisper/whisper/tokenizer.py

Lines 248 to 256 in 8cf36f3

	symbols = list("\"#()*+/:;<=>@[\\]^_`{|}~「」『』")
	symbols += "<< >> <<< >>> -- --- -( -[ (' (\" (( )) ((( ))) [[ ]] {{ }} ♪♪ ♪♪♪".split()
	# symbols that may be a single token or multiple tokens depending on the tokenizer.
	# In case they're multiple tokens, suppress the first token, which is safe because:
	# These are between U+2640 and U+267F miscellaneous symbols that are okay to suppress
	# in generations, and in the 3-byte UTF-8 representation they share the first two bytes.
	miscellaneous = set("♩♪♫♬♭♮♯")
	assert all(0x2640 <= ord(c) <= 0x267F for c in miscellaneous)

But «, » were not included in the default set of tokens to suppress, unintentionally. You can add them to the list in your code to avoid this, and I will also consider adding them to the default.

[apwhitelaw]

Thanks, I was able to add them, works great!

Since the other symbols are being suppressed, I would think it makes sense to add these to the default list. But that's up to you to decide, at least now I know I can change it.