Question about `bad_cvec` in CFG: using downsampled `cvec` without training exposure

[Stareven233]

Hello, I have another question about CFG and was hoping you could help me with it.

In the sampling code, the “unconditional” input bad_cvec is set to the downsampled cvec. However, from the training code, it looks like bad_cvec is not used during training. This raises a concern: could this lead to a train-inference mismatch?

Furthermore, CFG typically relies on contrasting a conditional input with an unconditional one. Since bad_cvec is merely a downsampled version, it arguably still carries conditions. I’m curious to know if using this “imperfect” unconditional input might offer some advantages.

Thanks in advance for your clarification!

[Pur1zumu]

Thank you for your insightful questions!

Regarding Question 1:

Yes, you've identified a key point! We intentionally leverage this train-inference mismatch so that a trained model produces a lower-quality noise prediction when given the downsampled version of cvec. We then use this lower-quality prediction to guide the model away from such outputs during inference. If we were to use bad_cvec during training, the model would learn to generate good results conditioned on bad_cvec, which would defeat our purpose.

You can actually verify how the model's output degrades with downsampled cvec by patching the downsampled version as the sole condition input to the model and observing the response.

For more theoretical background, I'd recommend referring to "Guiding a Diffusion Model with a Bad Version of Itself" (https://arxiv.org/abs/2406.02507).

Regarding Question 2:

The main reason I didn't use a completely unconditional input (dropping content condition entirely) was my gut feeling about how this task differs from general generation tasks (like TTS or SVC). Voice conversion is closer to a translation task and heavily relies on content (timbre is what needs to be modified). So completely dropping all content conditions didn't seem to make much sense, at least for the SVC task.

However, after your question prompted me to revisit recent research, I came across this paper: "CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion" (https://arxiv.org/abs/2411.18918), which appears to directly drop the content condition as unconditional input. So maybe it's not a bad choice after all? I might experiment with this approach when I have more free time in the future.

[Stareven233]

Thank you so much for your prompt and detailed response!

I’ve learned a great deal from it. Regarding Question 2, I agree that voice conversion tasks generally prioritize content. However, in SVC task, users typically expect a vivid and accurate imitation of the target singer’s performance.

Ideally, both timbre and subtle nuances should be faithfully reproduced. These nuances include articulation, vocal techniques, breath control (My phrasing may not be precise) and so on.

Therefore, allowing more flexibility in “content” might be a worthwhile approach to explore. I’m really looking forward to future updates of the project!