finetune-gradio "test model" function has inference bug with speech & text mismatch, caused by clipped ref_audio and unchanged ref_text (sampled from dataset)

[KangZENG50025543]

Checks

• This template is only for bug reports, usage problems go with 'Help Wanted'.
• I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
• I have searched for existing issues, including closed ones, and couldn't find a solution.
• I am using English to submit this issue to facilitate community communication.

This Question is already sovled. I'll show here to help others who face the same issue.

Your model is probably fine. The bug actually occurs in the infer part.

The code I provided has been tested under Chinese dataset. The adaptation of English still need to judge.
utils_infer_changed_version.py

Problem Description

When analyzing the F5-TTS preprocessing pipeline when testing the model (it's the issue occured in "test model" of the GUI), a critical issue affecting training quality was discovered.

Current Behavior:

• When the reference audio exceeds 12 seconds, it gets truncated to the first 12 seconds or shorter
• However, the corresponding reference text remains completely unchanged
• This causes the model to learn an incorrect mapping: short audio ↔ long text

This leads to:

• ❌ Attention mechanism confusion (Generated content shows repeated phrases: e.g., Input reference text: "I am the Shore Keeper, your assistant" → Output generated audio: "your assistant, I am the Shore Keeper, your assistant, Shore Keeper, assistant")
• ❌ Duration prediction distortion (Generated speech is extremely short and spoken very fast)
• ❌ Prosody learning errors (Pauses are completely wrong, punctuation is entirely ignored)

Root Cause:
In the preprocess_ref_audio_text() function in file 'src/f5_tts/infer/utils_infer.py':

# Audio cutting:
aseg = aseg[:12000]  # Or cutting based on silence

# But text remains the same:
return ref_audio, ref_text  # ❌ No text cutting!

The solution is provided in the python file. It actually add the function of cutting the reference text in a right way inorder to make model reference the "well-matched" audio-text.

Steps to Reproduce

conda activated f5-tts

f5-tts_finetune-gradio

……(training part)

Converting audio...
Audio is over 12s, clipping short.

Generating audio in 3 batches...
100%|████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:06<00:00, 2.01s/it]

When listening to the audio, the audion is really fast and word is misordered.

✔️ Expected Behavior

The audio generated should be well paced and ordered.

❌ Actual Behavior

When listening to the audio, the audion is really fast and word is misordered.

[KangZENG50025543]

Note on Text Preparation
For optimal alignment results (this solution), please ensure your reference text has proper punctuation marking longer pauses, such as commas (,) for short breaks and periods (.) for full stops.

Text Preparation Guidelines for Better Alignment

The cropping algorithm relies on punctuation to detect natural breakpoints. For best results:

• Use , ， for brief pauses between phrases
• Use ; ； for moderate pauses
• Use . 。 ! ！ ? ？ for sentence endings
• Use …… for dramatic pauses or trailing thoughts

Well-punctuated text enables the algorithm to crop at semantically appropriate boundaries, preserving the natural rhythm and meaning.

Example

Good: "I am the Shore Keeper, your assistant. I have guarded this coast for ten years…"
Better cut points for the matching between cut text and cut audio

Avoid: "I am the Shore Keeper your assistant I have guarded this coast for ten years"
Hard to find safe cut points ❌

[SWivid]

This happens cuz finetune-gradio has no check on input, and only happens in test function inside finetune-gradio.
infer_utils will not influence the training process.

F5-TTS/src/f5_tts/train/finetune_gradio.py

Lines 1784 to 1801 in 1dcb4e1

	random_sample_infer.click(
	fn=get_random_sample_infer, inputs=[cm_project], outputs=[ref_text, gen_text, ref_audio]
	)
	with gr.Row():
	txt_info_gpu = gr.Textbox("", label="Inference on Device :")
	seed_info = gr.Textbox(label="Used Random Seed :")
	check_button_infer = gr.Button("Inference")
	gen_audio = gr.Audio(label="Generated Audio", type="filepath")
	check_button_infer.click(
	fn=infer,
	inputs=[
	cm_project,
	cm_checkpoint,
	exp_name,
	ref_text,

F5-TTS/src/f5_tts/infer/utils_infer.py

Lines 353 to 365 in 1dcb4e1

	if not ref_text.strip():
	global _ref_text_cache
	if audio_hash in _ref_text_cache:
	# Use cached asr transcription
	show_info("Using cached reference text...")
	ref_text = _ref_text_cache[audio_hash]
	else:
	show_info("No reference text provided, transcribing reference audio...")
	ref_text = transcribe(ref_audio)
	# Cache the transcribed text (not caching custom ref_text, enabling users to do manual tweak)
	_ref_text_cache[audio_hash] = ref_text
	else:
	show_info("Using custom reference text...")

The code I provided has been tested under Chinese dataset. The adaptation of English still need to judge.
utils_infer_changed_version.py

To separately clip audio and text, there rises the problem that the content of two are probably mismatched.

[KangZENG50025543]

Yes, the model itself is fine.

But is bug would happen if the user use your infer function.

Actually, I found this problem because of my university project use your infer function. I guess our TA also found that something went wrong when the time of audio longer than 12s. So he wrote "Ensure your audio no longer than 12s". 😂

Moreover, there should be some high efficient algorithms to cut the textand audio together (I guess?). My solution is the silly one (only use time to judge).

[SWivid]

But is bug would happen if the user use your infer function.

As mentioned above: This happens cuz finetune-gradio has no check on input, and only happens in test function inside finetune-gradio (#1140 (comment)).
The test function inside finetune-gradio is an add-on from original contributor.

The actual infer function can either be infer-gradio, infer-cli, runtime, api, or server as mentioned in infer README.md.

[SWivid]

We have some context here:

• Issue and Fix for Long Reference Audio in Voice Cloning Feature #1139

[KangZENG50025543]

我们这里有一些背景信息：

• 语音克隆功能中长引用音频的问题和修复 #1139

或许应该提示如果实在不好对源数据剪切，用户应该将过长ref_audio的ref_text留白？这样也许可以帮助更多疑惑的人。

或者，对于过长的ref_audio，还可以这样做：系统剪切后将其视作无ref_text，然后按照无ref_text的情况来处理：1.自动调用ARF 2.生成对应短ref_audio的新短ref_text。

[SWivid]

如果你感兴趣的话其实可以直接起pr，finetune-gradio有很多地方其实欠打磨的（所以我们一直都说用train.py最好😂，但是gui确实更简易上手）。

最简单的思路：

F5-TTS/src/f5_tts/train/finetune_gradio.py

Line 1148 in 1dcb4e1

random_item = random.choice(list_data)

如果这里取到超过12秒的就重新随机

[KangZENG50025543]

的确是简单粗暴的思路😂😂😂

如果你感兴趣的话其实可以直接起pr，finetune-gradio有很多地方其实欠打磨的（所以我们一直都说用 train.py 最好😂，但是gui确实更简易上手）。

最简单的思路：

F5-TTS/src/f5_tts/train/finetune_gradio.py

1148 号线 在1DCB4E1

random_item = 随机。选择（list_data)

如果这里取到超过12秒的就重新随机