Multiple Issues with attack_adversarial_patch_pytorch_yolo.ipynb

[alexdevassy]

Describe the bug
Multiple issues were observed while execution of attack_adversarial_patch_pytorch_yolo.ipynb in Jupyter & Colab notebooks.

Issue 1 - While running Detector Predictions on Benign Images code getting bounding boxes on black images without rendering the actual image.

Issue 2 - Error in Adversarial Patch Generation

To Reproduce
Executed the code as it is from https://github.com/Trusted-AI/adversarial-robustness-toolbox/blob/main/notebooks/adversarial_patch/attack_adversarial_patch_pytorch_yolo.ipynb without any changes

System information:

• Colab, Jupyter notebook in windows 10
• Colab - Python 3.10.12, Jupyter - Python 3.11.4
• ART 1.15.0
• Colab - torch 2.0.1+cu118, Jupyter - torch 2.0.1

[beat-buesser]

Hi @alexdevassy Do you observe the same ValueError with torch 1.x?

[alexdevassy]

Thanks @beat-buesser could you let me know which version of yolov5 library was used in below command
pip install yolov5

[beat-buesser]

@alexdevassy I think it might be related to torch 2.x which we haven't yet tested. Did you try running the notebook with torch 1.13?

[f4str]

Hi @alexdevassy thank you for pointing out these issues. I've investigated the and believe have found some answers.

Issue 1 - ART will try to reuse as much memory as possible to avoid making copies. PyTorch operations like x_tensor = torch.from_numpy(x) and x_tensor /= 255 will reuse the same memory as the numpy array. Therefore, when we resize the torch tensor from [0, 255] to [0, 1], it also resizes the original numpy array (since its the same internal memory). This is a bug that we'll fix. For now a temporary workaround create a copy before running predict:

- dets = detector.predict(coco_images)
+ dets = detector.predict(coco_images.copy())

Issue 2 - Our current implementation of YOLO v3 uses the pytorchyolo library (https://github.com/eriklindernoren/PyTorch-YOLOv3) which torch < 1.13 and torchvision < 0.14. Unfortunately, this is simply a limitation of the library, please try again using torch==1.12.1 and torchvision==0.13.1 which should be the latest supported versions. You also may get some other issues with gradients, but this should have fixed in the ART 1.15.1 release. So please upgrade to the newest version of ART. Let us know if you get any additional errors.

[dirtyoldman65]

Hello,
I'm exploring universal adversarial patches that cause image classifiers to label inputs as “dog,” regardless of the original content. I’m particularly interested in using a pre-generated patch image if available, because I don't have access to a GPU or local environment to run patch training.

I found your project and thought you might already have a patch targeting the “dog” class, or a sample image generated in your experiments. Would you be willing to share it or point me to where I might download one?

I'm using it solely for personal exploration around privacy. I'll adhere to any licensing requirements or usage terms you specify.

Thank you for your work and time.

Best regards,
Terry