Using Whisper to clean glitches/halucinations in a TTS generated (cloned) audio

[Sambsop]

I'm using coquiTTS to generate audio files from text (in french) but using the cloning feature the output files present audio glitches and sometimes ghostly/hallucination voices. Because the randomness of the output audio (talking about prosody) the quantity of glitches change also. To avoid that i fix the randomness seed to be able to reproduce the possible best result.
I haven't success to get a clean audio in my multiple tests, so (here comes the idea) i'm thinking about using Whisper to identify these glitches because they aren't really 'words' and removing them.
I start to write a script to compare the WER & CER and using jiwer library to get some 'confidance value' i could use. I would like some guidance or suggestion.
Thanks