Transcribe with a fixed set of words

[jsaluja]

Use case - Non English
Training - Person is reading sentences from a fixed text. Audio dataset is 22000 audio clips of < 30 seconds each.
Inference - Different readers read different passages in different order from the same fixed text.
Expected - Transcription to contain words only from the fixed set of words in the training dataset. No words outside of that.

What I tried
Fine tuned whisper small, large-v2, large-v3 on the same non English dataset.

Observed
Inferences from the fine tuned model contain some invalid predictions.

1. Invalid words that do not exist in the language at all
2. Valid words in the right langauge, but outside my fixed dataset

Fine tuned Whisper small appears to do slightly better than fine tuned Large-v2, Large-v3

I need inferences to contain words specifically from a valid fixed set of words, not one word outside of it.

Any recommendation / documentation for such use case ?