Merge pull request #334 from IBM/dev_1.2.0

Merge ART v1.2.0

Author: beat-buesser

.coveragerc

@@ -0,0 +1,19 @@
+[run]
+branch = True
+source = art
+
+[report]
+exclude_lines =
+    if self.debug:
+    pragma: no cover
+    raise NotImplementedError
+    if __name__ == .__main__.:
+ignore_errors = True
+omit =
+    data/*
+    docs/*
+    examples/*
+    mlops/*
+    models/*
+    notebooks/*
+    tests/*

.gitignore

@@ -105,5 +105,9 @@ demo/pics/*
 *.ipynb
 .DS_Store
 
+# ignore PyCharm project files
+.idea/
+
 # Exceptions for notebooks/
 !notebooks/*.ipynb
+!notebooks/adaptive_defence_evaluations/*.ipynb

.travis.yml

@@ -1,14 +1,15 @@
 dist: xenial
 language: python
 env:
-  - KERAS_BACKEND=tensorflow TENSORFLOW_V=1.15.0 KERAS_V=2.2.5
+  - KERAS_BACKEND=tensorflow TENSORFLOW_V=1.15.2 KERAS_V=2.2.5
   - KERAS_BACKEND=tensorflow TENSORFLOW_V=2.1.0 KERAS_V=2.3.1
 python:
-   - "3.6"
+  - "3.6"
+  - "3.7"
 matrix:
   include:
     - python: 3.6
-      env: KERAS_BACKEND=tensorflow TENSORFLOW_V=1.15.0 KERAS_V=2.2.5
+      env: KERAS_BACKEND=tensorflow TENSORFLOW_V=1.15.2 KERAS_V=2.2.5
       script:
         - (pycodestyle --max-line-length=120 art || exit 0) && (pylint --disable=C0415,E1136 -rn art || exit 0)
         - py.test --pep8 -m pep8

README-cn.md

@@ -1,10 +1,19 @@
-# Adversarial Robustness 360 Toolbox (ART) v1.1
+# Adversarial Robustness Toolbox (ART) v1.1
 <p align="center">
   <img src="docs/images/art_logo.png?raw=true" width="200" title="ART logo">
 </p>
 <br />
 
-[![Build Status](https://travis-ci.org/IBM/adversarial-robustness-toolbox.svg?branch=master)](https://travis-ci.org/IBM/adversarial-robustness-toolbox) [![Documentation Status](https://readthedocs.org/projects/adversarial-robustness-toolbox/badge/?version=latest)](http://adversarial-robustness-toolbox.readthedocs.io/en/latest/?badge=latest) [![GitHub version](https://badge.fury.io/gh/IBM%2Fadversarial-robustness-toolbox.svg)](https://badge.fury.io/gh/IBM%2Fadversarial-robustness-toolbox) [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/IBM/adversarial-robustness-toolbox.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/IBM/adversarial-robustness-toolbox/context:python) [![Total alerts](https://img.shields.io/lgtm/alerts/g/IBM/adversarial-robustness-toolbox.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/IBM/adversarial-robustness-toolbox/alerts/)
+[![Build Status](https://travis-ci.org/IBM/adversarial-robustness-toolbox.svg?branch=master)](https://travis-ci.org/IBM/adversarial-robustness-toolbox)
+[![Documentation Status](https://readthedocs.org/projects/adversarial-robustness-toolbox/badge/?version=latest)](http://adversarial-robustness-toolbox.readthedocs.io/en/latest/?badge=latest)
+[![GitHub version](https://badge.fury.io/gh/IBM%2Fadversarial-robustness-toolbox.svg)](https://badge.fury.io/gh/IBM%2Fadversarial-robustness-toolbox)
+[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/IBM/adversarial-robustness-toolbox.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/IBM/adversarial-robustness-toolbox/context:python)
+[![Total alerts](https://img.shields.io/lgtm/alerts/g/IBM/adversarial-robustness-toolbox.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/IBM/adversarial-robustness-toolbox/alerts/)
+[![codecov](https://codecov.io/gh/IBM/adversarial-robustness-toolbox/branch/master/graph/badge.svg)](https://codecov.io/gh/IBM/adversarial-robustness-toolbox)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/adversarial-robustness-toolbox)](https://pypi.org/project/adversarial-robustness-toolbox/)
+[![slack-img](https://img.shields.io/badge/chat-on%20slack-yellow.svg)](https://ibm-art.slack.com/)
 
 Adversarial Robustness Toolbox（ART）是一个Python库，支持研发人员保护机器学习模型（深度神经网络，梯度提升决策树，支持向量机，随机森林，Logistic回归，高斯过程，决策树，Scikit-learn管道，等）抵御对抗性威胁，使AI系统更安全。机器学习模型容易受到对抗性示例的影响，这些示例是经过特殊修改的输入（图像，文本，表格数据等），以通过机器学习模型达到预期的效果。 ART提供了构建和部署防御的工具， 并使用对抗性攻击对其进行测试。
 防御机器学习模型主要用于验证模型的稳健性和模型强化. 所用方法包括前期处理输入，利用对抗样本增加训练数据以及利用实时检测方法来标记可能已被对手修改的输入等。 ART中实施的攻击使用目前最先进的威胁模型测试防御， 以此来制造机器学习模型的对抗性攻击。 
@@ -29,6 +38,8 @@ ART正在不断发展中。 我们欢迎您的反馈，错误报告和对ART建
 ## ART中实施的攻击，防御，检测，指标，认证和验证
 
 **逃避攻击：**
+* Threshold Attack ([Vargas et al., 2019](https://arxiv.org/abs/1906.06026))
+* Pixel Attack ([Vargas et al., 2019](https://arxiv.org/abs/1906.06026), [Su et al., 2019](https://ieeexplore.ieee.org/abstract/document/8601309/citations#citations)) 
 * HopSkipJump攻击 ([Chen et al., 2019](https://arxiv.org/abs/1904.02144))
 * 高可信度低不确定性对抗性例子 ([Grosse et al., 2018](https://arxiv.org/abs/1812.02606))
 * 预计梯度下降 ([Madry et al., 2017](https://arxiv.org/abs/1706.06083))
@@ -51,11 +62,13 @@ ART正在不断发展中。 我们欢迎您的反馈，错误报告和对ART建
 **提取攻击:**
 * 功能等效提取 ([Jagielski et al., 2019](https://arxiv.org/abs/1909.01838))
 * Copycat CNN ([Correia-Silva et al., 2018](https://arxiv.org/abs/1806.05476))
+* KnockoffNets ([Orekondy et al., 2018](https://arxiv.org/abs/1812.02766))
 
 **中毒攻击**
 * 对SVM的中毒攻击 ([Biggio et al., 2013](https://arxiv.org/abs/1206.6389))
+* Backdoor Attack ([Gu, et. al., 2017](https://arxiv.org/abs/1708.06733))
 
-**防御：**
+**防御 - 预处理器：**
 * 温度计编码 ([Buckman et al., 2018](https://openreview.net/forum?id=S18Su--CW))
 * 总方差最小化 ([Guo et al., 2018](https://openreview.net/forum?id=SyJ7ClWCb))
 * PixelDefend ([Song et al., 2017](https://arxiv.org/abs/1710.10766))
@@ -65,15 +78,21 @@ ART正在不断发展中。 我们欢迎您的反馈，错误报告和对ART建
 * JPEG压缩 ([Dziugaite et al., 2016](https://arxiv.org/abs/1608.00853))
 * 标签平滑 ([Warde-Farley and Goodfellow, 2016](https://pdfs.semanticscholar.org/b5ec/486044c6218dd41b17d8bba502b32a12b91a.pdf))
 * 虚拟对抗训练 ([Miyato et al., 2015](https://arxiv.org/abs/1507.00677))
-* 对抗训练 ([Szegedy et al., 2013](http://arxiv.org/abs/1312.6199))
 
-**提取防御:**
+**防御 - 后处理器:**
 * 反向乙状结肠 ([Lee et al., 2018](https://arxiv.org/abs/1806.00054))
 * 随机噪声 ([Chandrasekaranet al., 2018](https://arxiv.org/abs/1811.02054))
 * 类标签 ([Tramer et al., 2016](https://arxiv.org/abs/1609.02943), [Chandrasekaranet al., 2018](https://arxiv.org/abs/1811.02054))
 * 高信心 ([Tramer et al., 2016](https://arxiv.org/abs/1609.02943))
 * 四舍五入 ([Tramer et al., 2016](https://arxiv.org/abs/1609.02943))
 
+**防御 - 培训师:**
+* 对抗训练 ([Szegedy et al., 2013](http://arxiv.org/abs/1312.6199))
+* 对抗训练 Madry PGD ([Madry et al., 2017](https://arxiv.org/abs/1706.06083))
+
+**防御 - 变压器:**
+* 防御蒸馏 ([Papernot et al., 2015](https://arxiv.org/abs/1511.04508))
+
 **稳健性指标，认证和验证：**
 * Clique方法稳健性验证 ([Hongge et al., 2019](https://arxiv.org/abs/1906.03849))
 * 随机平滑 ([Cohen et al., 2019](https://arxiv.org/abs/1902.02918))

README.md

@@ -1,14 +1,23 @@
-# Adversarial Robustness 360 Toolbox (ART) v1.1
+# Adversarial Robustness Toolbox (ART) v1.1
 <p align="center">
   <img src="docs/images/art_logo.png?raw=true" width="200" title="ART logo">
 </p>
 <br />
 
-[![Build Status](https://travis-ci.org/IBM/adversarial-robustness-toolbox.svg?branch=master)](https://travis-ci.org/IBM/adversarial-robustness-toolbox) [![Documentation Status](https://readthedocs.org/projects/adversarial-robustness-toolbox/badge/?version=latest)](http://adversarial-robustness-toolbox.readthedocs.io/en/latest/?badge=latest) [![GitHub version](https://badge.fury.io/gh/IBM%2Fadversarial-robustness-toolbox.svg)](https://badge.fury.io/gh/IBM%2Fadversarial-robustness-toolbox) [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/IBM/adversarial-robustness-toolbox.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/IBM/adversarial-robustness-toolbox/context:python) [![Total alerts](https://img.shields.io/lgtm/alerts/g/IBM/adversarial-robustness-toolbox.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/IBM/adversarial-robustness-toolbox/alerts/)
+[![Build Status](https://travis-ci.org/IBM/adversarial-robustness-toolbox.svg?branch=master)](https://travis-ci.org/IBM/adversarial-robustness-toolbox)
+[![Documentation Status](https://readthedocs.org/projects/adversarial-robustness-toolbox/badge/?version=latest)](http://adversarial-robustness-toolbox.readthedocs.io/en/latest/?badge=latest)
+[![GitHub version](https://badge.fury.io/gh/IBM%2Fadversarial-robustness-toolbox.svg)](https://badge.fury.io/gh/IBM%2Fadversarial-robustness-toolbox)
+[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/IBM/adversarial-robustness-toolbox.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/IBM/adversarial-robustness-toolbox/context:python)
+[![Total alerts](https://img.shields.io/lgtm/alerts/g/IBM/adversarial-robustness-toolbox.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/IBM/adversarial-robustness-toolbox/alerts/)
+[![codecov](https://codecov.io/gh/IBM/adversarial-robustness-toolbox/branch/master/graph/badge.svg)](https://codecov.io/gh/IBM/adversarial-robustness-toolbox)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/adversarial-robustness-toolbox)](https://pypi.org/project/adversarial-robustness-toolbox/)
+[![slack-img](https://img.shields.io/badge/chat-on%20slack-yellow.svg)](https://ibm-art.slack.com/)
 
 [中文README请按此处](README-cn.md)
 
-Adversarial Robustness 360 Toolbox (ART) is a Python library supporting developers and researchers in defending Machine 
+Adversarial Robustness Toolbox (ART) is a Python library supporting developers and researchers in defending Machine 
 Learning models (Deep Neural Networks, Gradient Boosted Decision Trees, Support Vector Machines, Random Forests, 
 Logistic Regression, Gaussian Processes, Decision Trees, Scikit-learn Pipelines, etc.) against adversarial threats 
 (including evasion, extraction and poisoning) and helps making AI systems more secure and trustworthy. Machine Learning 
@@ -42,6 +51,8 @@ Get in touch with us on [Slack](https://ibm-art.slack.com) (invite [here](https:
 ## Implemented Attacks, Defences, Detections, Metrics, Certifications and Verifications
 
 **Evasion Attacks:**
+* Threshold Attack ([Vargas et al., 2019](https://arxiv.org/abs/1906.06026))
+* Pixel Attack ([Vargas et al., 2019](https://arxiv.org/abs/1906.06026), [Su et al., 2019](https://ieeexplore.ieee.org/abstract/document/8601309/citations#citations))
 * HopSkipJump attack ([Chen et al., 2019](https://arxiv.org/abs/1904.02144))
 * High Confidence Low Uncertainty adversarial samples ([Grosse et al., 2018](https://arxiv.org/abs/1812.02606))
 * Projected gradient descent ([Madry et al., 2017](https://arxiv.org/abs/1706.06083))
@@ -64,11 +75,13 @@ Get in touch with us on [Slack](https://ibm-art.slack.com) (invite [here](https:
 **Extraction Attacks:**
 * Functionally Equivalent Extraction ([Jagielski et al., 2019](https://arxiv.org/abs/1909.01838))
 * Copycat CNN ([Correia-Silva et al., 2018](https://arxiv.org/abs/1806.05476))
+* KnockoffNets ([Orekondy et al., 2018](https://arxiv.org/abs/1812.02766))
 
 **Poisoning Attacks:**
 * Poisoning Attack on SVM ([Biggio et al., 2013](https://arxiv.org/abs/1206.6389))
+* Backdoor Attack ([Gu, et. al., 2017](https://arxiv.org/abs/1708.06733))
 
-**Defences:**
+**Defences - Preprocessor:**
 * Thermometer encoding ([Buckman et al., 2018](https://openreview.net/forum?id=S18Su--CW))
 * Total variance minimization ([Guo et al., 2018](https://openreview.net/forum?id=SyJ7ClWCb))
 * PixelDefend ([Song et al., 2017](https://arxiv.org/abs/1710.10766))
@@ -78,15 +91,21 @@ Get in touch with us on [Slack](https://ibm-art.slack.com) (invite [here](https:
 * JPEG compression ([Dziugaite et al., 2016](https://arxiv.org/abs/1608.00853))
 * Label smoothing ([Warde-Farley and Goodfellow, 2016](https://pdfs.semanticscholar.org/b5ec/486044c6218dd41b17d8bba502b32a12b91a.pdf))
 * Virtual adversarial training ([Miyato et al., 2015](https://arxiv.org/abs/1507.00677))
-* Adversarial training ([Szegedy et al., 2013](http://arxiv.org/abs/1312.6199))
 
-**Extraction Defences:**
+**Defences - Postprocessor:**
 * Reverse Sigmoid ([Lee et al., 2018](https://arxiv.org/abs/1806.00054))
 * Random Noise ([Chandrasekaranet al., 2018](https://arxiv.org/abs/1811.02054))
 * Class Labels ([Tramer et al., 2016](https://arxiv.org/abs/1609.02943), [Chandrasekaranet al., 2018](https://arxiv.org/abs/1811.02054))
 * High Confidence ([Tramer et al., 2016](https://arxiv.org/abs/1609.02943))
 * Rounding ([Tramer et al., 2016](https://arxiv.org/abs/1609.02943))
 
+**Defences - Trainer:**
+* Adversarial training ([Szegedy et al., 2013](http://arxiv.org/abs/1312.6199))
+* Adversarial training Madry PGD ([Madry et al., 2017](https://arxiv.org/abs/1706.06083))
+
+**Defences - Transformer:**
+* Defensive Distillation ([Papernot et al., 2015](https://arxiv.org/abs/1511.04508))
+
 **Robustness Metrics, Certifications and Verifications**:
 * Clique Method Robustness Verification ([Hongge et al., 2019](https://arxiv.org/abs/1906.03849))
 * Randomized Smoothing ([Cohen et al., 2019](https://arxiv.org/abs/1902.02918))
@@ -122,7 +141,7 @@ The most recent version of ART can be downloaded or cloned from this repository:
 git clone https://github.com/IBM/adversarial-robustness-toolbox
 ```
 
-Install ART with the following command from the project folder `art`:
+Install ART with the following command from the project folder `adversarial-robustness-toolbox`:
 ```bash
 pip install .
 ```
@@ -149,10 +168,10 @@ and overview and more information.
 
 Adding new features, improving documentation, fixing bugs, or writing tutorials are all examples of helpful 
 contributions. Furthermore, if you are publishing a new attack or defense, we strongly encourage you to add it to the 
-Adversarial Robustness 360 Toolbox so that others may evaluate it fairly in their own work.
+Adversarial Robustness Toolbox so that others may evaluate it fairly in their own work.
 
 Bug fixes can be initiated through GitHub pull requests. When making code contributions to the Adversarial Robustness 
-360 Toolbox, we ask that you follow the `PEP 8` coding standard and that you provide unit tests for the new features.
+Toolbox, we ask that you follow the `PEP 8` coding standard and that you provide unit tests for the new features.
 
 This project uses [DCO](https://developercertificate.org/). Be sure to sign off your commits using the `-s` flag or 
 adding `Signed-off-By: Name<Email>` in the commit message.

art/__init__.py

@@ -18,34 +18,14 @@
 # pylint: disable=C0103
 
 LOGGING = {
-    'version': 1,
-    'disable_existing_loggers': False,
-    'formatters': {
-        'std': {
-            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s',
-            'datefmt': '%Y-%m-%d %H:%M'
-        }
+    "version": 1,
+    "disable_existing_loggers": False,
+    "formatters": {"std": {"format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s", "datefmt": "%Y-%m-%d %H:%M"}},
+    "handlers": {
+        "default": {"class": "logging.NullHandler",},
+        "test": {"class": "logging.StreamHandler", "formatter": "std", "level": logging.INFO},
     },
-    'handlers': {
-        'default': {
-            'class': 'logging.NullHandler',
-        },
-        'test': {
-            'class': 'logging.StreamHandler',
-            'formatter': 'std',
-            'level': logging.INFO
-        }
-    },
-    'loggers': {
-        'art': {
-            'handlers': ['default']
-        },
-        'tests': {
-            'handlers': ['test'],
-            'level': 'INFO',
-            'propagate': True
-        }
-    }
+    "loggers": {"art": {"handlers": ["default"]}, "tests": {"handlers": ["test"], "level": "INFO", "propagate": True}},
 }
 logging.config.dictConfig(LOGGING)
 logger = logging.getLogger(__name__)

art/attacks/__init__.py

@@ -1,7 +1,7 @@
 """
 Module providing adversarial attacks under a common interface.
 """
-from art.attacks.attack import Attack, EvasionAttack, PoisoningAttack, ExtractionAttack
+from art.attacks.attack import Attack, EvasionAttack, PoisoningAttackBlackBox, PoisoningAttackWhiteBox, ExtractionAttack
 
 from art.attacks.evasion.adversarial_patch import AdversarialPatch
 from art.attacks.evasion.boundary import BoundaryAttack
@@ -20,7 +20,10 @@
 from art.attacks.evasion.universal_perturbation import UniversalPerturbation
 from art.attacks.evasion.virtual_adversarial import VirtualAdversarialMethod
 from art.attacks.evasion.zoo import ZooAttack
+from art.attacks.evasion.pixel_threshold import PixelAttack
+from art.attacks.evasion.pixel_threshold import ThresholdAttack
 
+from art.attacks.poisoning.backdoor_attack import PoisoningAttackBackdoor
 from art.attacks.poisoning.poisoning_attack_svm import PoisoningAttackSVM
 
 from art.attacks.extraction.functionally_equivalent_extraction import FunctionallyEquivalentExtraction