Attacks on Huggingface ASR model

[filbert-c]

Hi ,

I am trying to use ART to test the vulnerability of pretrained ASR models from huggingface. I read through the code of PyTorchEspresso and ImperceptibleASRPyTorch.

But came across an error where the value self.global_optimal_delta.grad inside ImperceptibleASRPyTorch do not get updated.
Not sure why the optimiser_1 does not update the gradients in self.global_optimal_delta.grad after loss.backward() is called. The estimator code is similar to PyTorchEspresso's/Deepspeech2 compute_loss_and_decoded_output

   def compute_loss_and_decoded_output(
        self, masked_adv_input: "torch.Tensor", original_output: np.ndarray, **kwargs
    ) -> Tuple["torch.Tensor", np.ndarray]:
        """
        Compute loss function and decoded output.
        :param masked_adv_input: The perturbed inputs.
        :param original_output: Target values of shape (nb_samples). Each sample in `original_output` is a string and
                                it may possess different lengths. A possible example of `original_output` could be:
                                `original_output = np.array(['SIXTY ONE', 'HELLO'])`.
        :return: The loss and the decoded output.
        """
        self._model.train()
        self._model.freeze_feature_encoder()
        self.set_batchnorm(train=False)
        
        inputs = self._preprocess_transform_model_input(x=masked_adv_input.to(self.device),
                                                        y=original_output,
                                                        )
        loss, logits = self._model(**inputs,return_dict=False)
        pred_ids = torch.argmax(logits, dim=-1)
        decoded_output = np.array(self._processor.batch_decode(pred_ids))
        
        self._model.eval()
        self.set_batchnorm(train=True)
        return loss, decoded_output

The loss, local_delta,decoded_output, masked_adv_input is shown in the screenshot.

I am using Wav2Vec2ForCTC as my model. I extended the PyTorch estimator similar to the way PyTorchEspresso was written

[filbert-c]

I found the reason why... so when

local_delta = self.global_optimal_delta[:local_batch_size, :local_max_length]
local_delta_rescale = torch.clamp(local_delta, -self.eps, self.eps).to(self.estimator.device)
local_delta_rescale *= torch.tensor(rescale).to(self.estimator.device)
adv_input = local_delta_rescale + torch.tensor(original_input).to(self.estimator.device)
masked_adv_input = adv_input * torch.tensor(input_mask).to(self.estimator.device)

masked_adv input is sent to compute_loss_and_decode function,
this calls _preprocess_transform_model_input and _transform_model_input.
But i am using huggingface api "processor" which converts inputs into a suitable format for the model input.

I believe during this step, the computation graph for global_optimal_delta becomes destroyed. Hence why the gradients are never updated. Instead, i added the local_delta_rescale and input mask after processing the input with huggingface. But i have to ammend the code in ImperceptiblePytorch in this case. Is there a better solution?

[beat-buesser]

Hi @filbert-c Thank you very much for starting this discussion and please apologise my delayed response. Do you think we could solve the issue with a new estimator specific for the pretrained ASR models from huggingface taking into account their specific processors?

[filbert-c]

Hi @beat-buesser , it could be solved if we create a estimator for it similar to PyTorchDeepSpeech and PyTorchEspresso. However, even after my naive hacks, i realised the performance of the ImperceptiblePyTorchASR was poor compared to the ImperceptibleASR class.

Correct me if i am wrong but it looks like ImperceptibleASR uses iterative FGSM which is slightly different? Since it was more difficult to hit the targeted text by using a optimiser like Adam (ImperceptiblePyTorchASR). When trying to attack a model like WavLM or Data2Vec, i had to use a scheduler with LROnPlateau to hit the targeted text within 1000 iterations and a high initial LR. Or perhaps there is just a optimal set of hyperparameters that i am unaware of. Also, when trying for stage 2 attacks, i have never been able to make the sound imperceptible and the targeted text is no longer maintained.

[filbert-c]

Hi @beat-buesser

This is the processor class which has a tokenizer and feature extractor

processor: https://github.com/huggingface/transformers/blob/main/src/transformers/models/wav2vec2/processing_wav2vec2.py
feature extractor: https://github.com/huggingface/transformers/blob/main/src/transformers/models/wav2vec2/feature_extraction_wav2vec2.py

and here is how i tried to use global optimal delta

    self._processor = Wav2Vec2Processor.from_pretrained(pretrained_model)
    self.global_optimal_delta = Variable( 
        torch.zeros(self.batch_size, self.global_max_length).type(torch.cuda.FloatTensor),
        requires_grad=True
    )
    local_delta = self.global_optimal_delta[:local_batch_size, :local_max_length]
    local_delta_rescale = torch.clamp(local_delta, -self.eps, self.eps).to(self.estimator.device)
    local_delta_rescale *= torch.tensor(rescale).to(self.estimator.device)
    adv_input = local_delta_rescale + torch.tensor(original_input).to(self.estimator.device)
    masked_adv_input = adv_input * torch.tensor(input_mask).to(self.estimator.device)
   
   # i cant find any other solution for getting the processor to accept 
   # ragged array inputs other than feeding in a list.
    x = [v.flatten().tolist() for v in masked_adv_input]
    inputs = self._processor(x,sampling_rate=self.sr,return_tensors='pt', padding=True)

But using the above method will cause global optimal delta to have None gradients. So then i thought maybe this would work

inputs = self._processor(original_input, sampling_rate=self.sr, return_tensors='pt',padding=True)
adv_input = local_delta_rescale + inputs['input_values']
masked_adv_input = adv_input * torch.tensor(input_mask).to(self.estimator.device)
inputs['input_values'] = masked_adv_input

but the inputs to the model would be different from the code in the first snippet.

[beat-buesser]

Hi @filbert-c I see, the tolist() results in a list of floats breaking the gradient flow.

Does self._processor accept Tensors as input?
Could torch.nested already be ready to be used for the ragged array?

[filbert-c]

Hi @beat-buesser , I dont think tolist() is the issue, i tried using a single example previously and removed the list method. i wrote a simple snippet to test if tolist() was the issue. the processor allows tensors as input if they are of the same shape as shown below.

import torchaudio
from torch.autograd import Variable
from transformers import Wav2Vec2Processor, Data2VecAudioForCTC
import torch

processor = Wav2Vec2Processor.from_pretrained("facebook/data2vec-audio-large-960h")
model = Data2VecAudioForCTC.from_pretrained("facebook/data2vec-audio-large-960h").to('cuda')

batch_size = 1
learning_rate_1 = 1e-4
global_max_length = torchaudio.load('en.wav')[0][0].shape[0]
global_optimal_delta = Variable(
                torch.zeros(batch_size, global_max_length).type(torch.cuda.FloatTensor),  # type: ignore
                requires_grad=True,
            )
eps = 0.05
max_iter = 10
optimizer_1 = torch.optim.Adam(params=[global_optimal_delta], lr=learning_rate_1)
processed_files = ['en.wav']
y = ['OK']
for audio_file in processed_files:
    audio_waveform = torchaudio.load(audio_file)[0][0]    
    
    rescale = np.ones([batch_size, global_max_length], dtype=np.float64)
    input_mask = np.zeros([batch_size, global_max_length], dtype=np.float64)
    original_input = audio_waveform
    input_mask[:,:] = 1
    
    for i in range(max_iter):
        optimizer_1.zero_grad()
        model.train()
        model.freeze_feature_encoder()
        
        local_delta = global_optimal_delta
        local_delta_rescale = torch.clamp(local_delta, eps, eps).to('cuda')
        local_delta_rescale *= torch.tensor(rescale).to('cuda')
        adv_input = local_delta_rescale + torch.tensor(original_input).to('cuda')
        masked_adv_input = adv_input * torch.tensor(input_mask).to('cuda')
        masked_adv_input = masked_adv_input.flatten()
        
        print(masked_adv_input.shape, masked_adv_input.dtype)
        inputs = processor(masked_adv_input, sampling_rate=16000, return_tensors="pt")
        print(inputs['input_values'].shape)
        with processor.as_target_processor():
                targets_batch = processor(y, padding=True, return_tensors='pt').input_ids
        for k in inputs.keys():
            inputs[k] = inputs[k].to('cuda')
        inputs['labels'] = targets_batch
        loss, logits = model(**inputs,return_dict=False)
        print(loss)
        predicted_ids = torch.argmax(logits, dim=-1)

        # transcribe speech
        transcription = processor.batch_decode(predicted_ids)
        loss.backward()
        print(global_optimal_delta.grad)
        global_optimal_delta.grad = torch.sign(global_optimal_delta.grad)
        optimizer_1.step()

[beat-buesser]

@filbert-c I have been able to reproduce the error above with your script. Have you already checked the code of processor for potential calls detach() between input data and the output?

[filbert-c]

@beat-buesser been busy, will take a look at it on the weekend.