How to do adversarial test in pytorch lightning?

[Xiang-Pan]

Can we transfer a parameter or define a hook to test loop?

[justusschock]

Hi,

what exactly do you mean by "adversarial test"?

[Xiang-Pan]

Adversarial training or Adversarial attack https://en.wikipedia.org/wiki/Adversarial_machine_learning.

Such as PGD for training or PGD adversarial for test?

[Xiang-Pan]

For example,

training and test definition

def main(args: Namespace) -> None:
    cifar10_dm = CIFAR10DataModule(data_dir="./cached_datasets/",batch_size=1024)
    model = DN()

    # tb_logger = pl_loggers.TensorBoardLogger('./logs/')
    trainer = pl.Trainer(max_epochs=1,gpus=1,log_every_n_steps= 1)
    print(model.log)
    trainer.fit(model, cifar10_dm)
    model.test_mode = "8_0.8_20"
    model.set_pgd_attack()
    trainer.test()

---

model definition

def test_step(self, batch, batch_idx):
  if self.test_mode == "clean":
      x, y = batch
      final_logit = self.forward(x)
      loss = self.loss(final_logit,y)
      # calculate acc
      labels_hat = torch.argmax(final_logit, dim=1)
      test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
      self.log_dict({'test_loss': loss, 'test_acc': test_acc})
  elif self.test_mode == "8_0.8_20":
      images, labels = batch
      self.ori_net.train()
      adv_images, _ = self.pgd_attack(images, labels)
      final_logit = self.forward(adv_images)
      self.ori_net.eval()
      # loss = self.loss(final_logit,labels)
      # calculate acc
      labels_hat = torch.argmax(final_logit, dim=1)
      test_acc = torch.sum(labels == labels_hat).item() / (len(labels) * 1.0)
      self.log_dict({'test_acc': test_acc})

---

adv_images, _ = self.pgd_attack(images, labels) \ is used to generate the adversarial images, however, it shows
Traceback (most recent call last):
File "tasks/DN/train_DN.py", line 63, in
main(hparams)
File "tasks/DN/train_DN.py", line 27, in main
trainer.test()
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in test
results = self._run(model)
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
self.dispatch()
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 793, in dispatch
self.accelerator.start_evaluating(self)
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 99, in start_evaluating
self.training_type_plugin.start_evaluating(trainer)
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 148, in start_evaluating
self._results = trainer.run_stage()
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 804, in run_stage
return self.run_evaluate()
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1044, in run_evaluate
eval_loop_results = self.run_evaluation()
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in run_evaluation
output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 170, in evaluation_step
output = self.trainer.accelerator.test_step(args)
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 245, in test_step
return self.training_type_plugin.test_step(*args)
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 164, in test_step
return self.lightning_module.test_step(*args, **kwargs)
File "./models/DN/DN.py", line 78, in test_step
adv_images, _ = self.pgd_attack(images, labels)
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "./pgdattacks/attacks/pgd_norm.py", line 80, in forward
grad = torch.autograd.grad(cost, adv_images, retain_graph=False, create_graph=False)[0] ##### non_norm !!!
File "/home/xiangpan/.conda/envs/pytorch/lib/python3.8/site-packages/torch/autograd/init.py", line 223, in grad
return Variable._execution_engine.run_backward(
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

---

I guess that it was caused by test or eval model( with no_grad setting for pytorch).

[justusschock]

If you want to train your mode, it does require a training_step where you define training logic. The test step should only run inference and calculate metrics

[Xiang-Pan]

The training part is ok.

The problem is, when we do the adversarial test, we need to calculate the model gradient, (I guess that's the problem).

[justusschock]

Yes, that's a problem. You need to manually enable gradient computation by torch.set_grad_enabled