Empirical Robustness Metric Support for all Evasion attacks

[shreyakhandelwal07]

Hi, I can see that the attack methods currently supported for evaluating empirical robustness are Fast Gradient Sign Method and Hop Skip Jump.

adversarial-robustness-toolbox/art/metrics/metrics.py

Line 97 in 3d86b8d

:param attack_name: A string specifying the attack to be used. Currently supported attacks are {`fgsm', `hsj`}

But in theory, it can be evaluated for all evasion attacks, right?

Are there any plans to support more attack methods for evaluating empirical robustness?

[beat-buesser]

Hi @shreyakhandelwal07 Thank you very much for your question! I have moved your issue to our new Discussion forum.

Yes, empirical_robustness should be applicable to all evasion attacks. The SUPPORTED_METHODS is not exclusive and would need to be adapted by the user. A few drawbacks and reason for caution are that the implementation of empirical_robustness includes the default values for the attacks in SUPPORTED_METHODS which might need adaptations to a specific problem, it does not adapt to models and defences, and it attacks the predictions of the model instead of the stronger attack against true labels. I think these are the main reasons why it has not been updated recently.

Would you be interested to update and extend empirical_robustness?