Issuses regarding perturbation generation 关于扰动生成的一些问题。

[Yuki-zik]

感谢您提供的开源代码！我在阅读 SafeSpeech 项目时，针对 protect.py 文件有以下问题想请教：

1. 关于 loss_mel 的梯度传播
   在 protect.py 中计算 loss_mel 时，p_wav 似乎未进行 detach() 操作。在反向传播时，梯度是否会通过两条路径传播到 p_wav？

   • 一条路径：通过 TTS 模型传播。
   • 另一条路径：直接传播到 p_wav，不经过 TTS 模型。

2. 关于 p_wav 和 wav_hat 的长度一致性
   在您的实现中，TTS 模型如何保证 p_wav 和 wav_hat 的长度相同？这两个音频是否基于相同的文本内容生成？

[wxzyd123]

嗨，感谢你对项目的关注。
问题一：初始想法是按照原本的重构损失计算wav和wav_hat，但是发现使用包含梯度信息的p_wav更有利于优化，所以在最后反向传播时，p_wav经过TTS模型会得到wav_hat，不过此处TTS模型的梯度已经被禁用，不会影响扰动的更新。
问题二：wav_hat为重构的损失，所以基本和p_wav保持一定的长度，当然可以使用下面代码保证两个变量始终为同一长度

min_len = min(wav_hat.size(1), p_wav.size(1))
p_wav = p_wav[, :min_len]
wav_hat = wav_hat[, :min_len]

Here is an English translation of this issue.

Question:

Q1: Gradient Propagation of loss_mel:
In protect.py when calculating loss_mel, p_wav doesn't seem to use detach(). During backpropagation, will gradients propagate through two paths to p_wav?
Path 1: Through the TTS model.
Path 2: Directly to p_wav without passing through the TTS model.

Q2: Length Consistency between p_wav and wav_hat:
How does your implementation ensure consistent lengths between p_wav and wav_hat? Are these two audios generated from the same text content?

Reply:

Hi, thank you for your interest in the project.

A1: The initial idea was to compute reconstruction loss between the original wav and wav_hat. However, we found that using p_wav containing gradient information better facilitates optimization. During final backpropagation: p_wav passes through the TTS model to get wav_hat. But the gradients of the TTS model are disabled here, so they won't affect the perturbation updates.

A2: Since wav_hat is the reconstructed audio, it should maintain length consistency with p_wav. You can use this code to ensure identical lengths:

min_len = min(wav_hat.size(1), p_wav.size(1))
p_wav = p_wav[:, :min_len]
wav_hat = wav_hat[:, :min_len]