regarding long form transcription after changing the tokenizer

[20IM30007]

Hey for my use case, I have modified the vocabulary so that space can be considered as a separate token and I have a new model finetuned. And since my new tokenizer considers space as a separate token, more spaces are being predicted and hence the maximum prediction budget of 224 tokens is being exhausted without completely transcribing the audio and hence the output of the audio at the end is getting truncated. Is there something we can do to solve this?

@jongwook

[MahmoudAshraf97]

You'll have to retrain the decoder with a larger maximum output length or use shorter audio files, btw the maximum is 448 tokens including the prompt