Add a wrapper package mart_armory for running MART attacks in Armory

examples/mart_armory/README.md

@@ -0,0 +1,117 @@
+## Introduction
+
+We demonstrate how to configure and run a MART attack against object detection models in ARMORY.
+
+The demo attack here is about 30% faster than the baseline attack implementation in ARMORY, because we specify to use 16bit-mixed-precision in the MART attack configuration.
+
+MART is designed to be modular and configurable. It should empower users to evaluate adversarial robustness of deep learning models more effectively and efficiently.
+
+Please reach out to [Weilin Xu](mailto:[email protected]) if you have any question.
+
+## Installation
+
+Download the code repositories.
+
+```shell
+# You can start from any directory other than `~/coder/`, since we always use relative paths in the following commands.
+mkdir ~/coder; cd ~/coder
+
+git clone https://github.com/twosixlabs/armory.git
+# Make sure we are on the same page.
+cd armory; git checkout tags/v0.19.0 -b r0.19.0; cd ..
+
+git clone https://github.com/IntelLabs/MART.git -b example_armory_attack
+```
+
+Create and activate a Python virtualen environment.
+
+```shell
+cd armory
+python -m venv .venv
+source .venv/bin/activate
+```
+
+Install ARMORY, MART and the glue package mart_armory in editable mode.
+
+```shell
+pip install -e .[engine]
+pip install tensorflow tensorflow-datasets
+# PyTorch 2.0+ is already in the dependency of MART.
+pip install -e ../MART
+pip install -e ../MART/examples/mart_armory
+```
+
+Make sure PyTorch works on CUDA.
+
+```console
+$ CUDA_VISIBLE_DEVICES=0 python -c "import torch; print(torch.cuda.is_available())"
+True
+```
+
+> You may need to install a different PyTorch distribution if your CUDA is not 12.0.
+
+> Here's my `nvidia-smi` output: `| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |`
+
+## Usage
+
+1. Generate a YAML configuration of attack, using Adam as the optimizer.
+
+```shell
+python -m mart.generate_config \
+--config_dir=../MART/examples/mart_armory/mart_armory/configs \
+--config_name=assemble_attack.yaml \
+batch_converter=object_detection \
+batch_c15n=data_coco \
+attack=adversary \
+[email protected]=sgd \
+attack.optimizer.maximize=true \
++attack.optimizer.lr=13 \
++attack.max_iters=500 \
++attack/composer/perturber/projector=mask_range \
++attack/enforcer=default \
++attack/enforcer/constraints=[mask,pixel_range] \
++attack/composer/perturber/initializer=uniform \
+attack.composer.perturber.initializer.max=255 \
+attack.composer.perturber.initializer.min=0 \
++attack/composer/functions=overlay \
++attack/gradient_modifier=sign \
++attack/gain=rcnn_training_loss \
++attack.precision=16 \
+attack.optimizer.optimizer.path=torch.optim.Adam \
+~attack.optimizer.momentum \
+attack.objective=null \
+model_transform=armory_objdet \
+> mart_objdet_attack_adam500.yaml
+```
+
+2. Run the MART attack on one example.
+
+```shell
+cat scenario_configs/eval7/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json \
+| jq 'del(.attack)' \
+| jq '.attack.knowledge="white"' \
+| jq '.attack.use_label=true' \
+| jq '.attack.module="mart_armory"' \
+| jq '.attack.name="MartAttack"' \
+| jq '.attack.kwargs.mart_adv_config_yaml="mart_objdet_attack_adam500.yaml"' \
+| jq '.scenario.export_batches=true' \
+| CUDA_VISIBLE_DEVICES=0 armory run - --no-docker --use-gpu --gpus=1 --num-eval-batches 1
+```
+
+```
+2023-10-13 12:05:33 1m14s METRIC   armory.instrument.instrument:_write:743 adversarial_object_detection_mAP_tide on adversarial examples w.r.t. ground truth labels: {'mAP': {0.5: 0.0, 0.55: 0.0, 0.6: 0.0, 0.65: 0.0, 0.7: 0.0, 0.75: 0.0, 0.8: 0.0, 0.85: 0.0, 0.9: 0.0, 0.95: 0.0}, 'errors': {'main': {'dAP': {'Cls': 0.0, 'Loc': 0.0, 'Both': 0.0, 'Dupe': 0.0, 'Bkg': 0.0, 'Miss': 0.0}, 'count': {'Cls': 0, 'Loc': 0, 'Both': 0, 'Dupe': 0, 'Bkg': 100, 'Miss': 21}}, 'special': {'dAP': {'FalsePos': 0.0, 'FalseNeg': 0.0}, 'count': {'FalseNeg': 21}}}}
+```
+
+## Comparison
+
+Run the baseline attack on the same example for comparison. The MART attack is ~30% faster due to the 16-bit mixed precision.
+
+```shell
+cat scenario_configs/eval7/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json \
+| jq '.scenario.export_batches=true' \
+| CUDA_VISIBLE_DEVICES=0 armory run - --no-docker --use-gpu --gpus=1 --num-eval-batches 1
+```
+
+```console
+2023-10-13 12:11:50 1m33s METRIC   armory.instrument.instrument:_write:743 adversarial_object_detection_mAP_tide on adversarial examples w.r.t. ground truth labels: {'mAP': {0.5: 0.0, 0.55: 0.0, 0.6: 0.0, 0.65: 0.0, 0.7: 0.0, 0.75: 0.0, 0.8: 0.0, 0.85: 0.0, 0.9: 0.0, 0.95: 0.0}, 'errors': {'main': {'dAP': {'Cls': 0.0, 'Loc': 0.0, 'Both': 0.0, 'Dupe': 0.0, 'Bkg': 0.0, 'Miss': 0.0}, 'count': {'Cls': 0, 'Loc': 0, 'Both': 0, 'Dupe': 0, 'Bkg': 100, 'Miss': 21}}, 'special': {'dAP': {'FalsePos': 0.0, 'FalseNeg': 0.0}, 'count': {'FalseNeg': 21}}}}
+```

examples/mart_armory/hydra_plugins/hydra_mart_armory/mart_armory.py

@@ -0,0 +1,8 @@
+from hydra.core.config_search_path import ConfigSearchPath
+from hydra.plugins.search_path_plugin import SearchPathPlugin
+
+
+class HydraMartSearchPathPlugin(SearchPathPlugin):
+    def manipulate_search_path(self, search_path: ConfigSearchPath) -> None:
+        # Add mart.configs to search path
+        search_path.append("hydra-mart", "pkg://mart_armory.configs")

examples/mart_armory/mart_armory/__init__.py

@@ -0,0 +1,5 @@
+from importlib import metadata
+
+from mart_armory.attack_wrapper import MartAttack
+
+__version__ = metadata.version(__package__ or __name__)

examples/mart_armory/mart_armory/attack_wrapper.py

@@ -0,0 +1,76 @@
+#
+# Copyright (C) 2022 Intel Corporation
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+
+from __future__ import annotations
+
+import hydra
+from omegaconf import OmegaConf
+
+
+class MartAttack:
+    """A minimal wrapper to run MART adversary in Armory against PyTorch-based models.
+
+    1. Instantiate an adversary that runs attack in MART;
+    2. Instantiate batch_converter that turns Armory's numpy batch into the PyTorch batch;
+    3. The adversary.model_transform() extracts the PyTorch model from an ART Estimator and makes other changes to easier attack;
+    4. The adversary returns adversarial examples in the PyTorch format;
+    5. The batch_converter reverts the adversarial examples into the numpy format.
+    """
+
+    def __init__(self, model, mart_adv_config_yaml):
+        """_summary_
+
+        Args:
+            model (Callable): An ART Estimator that contains a PyTorch model.
+            mart_adv_config_yaml (str): File path to the adversary configuration.
+        """
+        # Instantiate a MART adversary.
+        adv_cfg = OmegaConf.load(mart_adv_config_yaml)
+        adv = hydra.utils.instantiate(adv_cfg)
+
+        # Transform the ART estimator to an attackable PyTorch model.
+        self.model_transform = adv.model_transform
+
+        # Convert the Armory batch to a form that is expected by the target PyTorch model.
+        self.batch_converter = adv.batch_converter
+
+        # Canonicalize batches for the Adversary.
+        self.batch_c15n = adv.batch_c15n
+
+        self.adversary = adv.attack
+
+        self.device = model.device
+
+        # Move adversary to the same device.
+        self.adversary.to(self.device)
+
+        # model_transform
+        self.model_transformed = self.model_transform(model)
+
+    def model(self, input, target):
+        # Wrap a model for the Adversary which works with the canonical (input, target) format.
+        batch = self.batch_c15n.revert(input, target)
+        output = self.model_transformed(*batch)
+        return output
+
+    def generate(self, **batch_armory_np):
+        # Armory format -> torchvision format
+        batch_tv_pth = self.batch_converter(batch_armory_np, device=self.device)
+
+        # Attack
+        # Canonicalize input and target for the adversary, and revert it at the end.
+        input, target = self.batch_c15n(batch_tv_pth)
+        self.adversary.fit(input, target, model=self.model)
+        input_adv, target_adv = self.adversary(input, target)
+        batch_adv_tv_pth = self.batch_c15n.revert(input_adv, target_adv)
+
+        # torchvision format -> Armory format
+        batch_adv_armory_np = self.batch_converter.revert(*batch_adv_tv_pth)
+
+        # Only return adversarial input in the original numpy format.
+        input_key = self.batch_converter.input_key
+        input_adv_np = batch_adv_armory_np[input_key]
+        return input_adv_np

examples/mart_armory/mart_armory/batch_converter.py

@@ -0,0 +1,88 @@
+#
+# Copyright (C) 2022 Intel Corporation
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+
+from functools import reduce
+
+import torch
+
+from mart.transforms.batch_c15n import BatchC15n
+
+
+class ObjectDetectionBatchConverter(BatchC15n):
+    def __init__(
+        self,
+        input_key: str = "x",
+        target_keys: dict = {
+            "y": ["area", "boxes", "id", "image_id", "is_crowd", "labels"],
+            "y_patch_metadata": [
+                "avg_patch_depth",
+                "gs_coords",
+                "mask",
+                "max_depth_perturb_meters",
+            ],
+        },
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.input_key = input_key
+        self.target_keys = target_keys
+
+    def _convert(self, batch: dict):
+        input = batch[self.input_key]
+
+        target = []
+        all_targets = [batch[key] for key in self.target_keys]
+
+        # Merge several target keys.
+        for dicts in zip(*all_targets):
+            joint_target = reduce(lambda a, b: a | b, dicts)
+            target.append(joint_target)
+
+        target = tuple(target)
+
+        return input, target
+
+    def _revert(self, input: tuple[torch.Tensor], target: tuple[dict]) -> dict:
+        batch = {}
+
+        batch[self.input_key] = input
+
+        # Split target into several self.target_keys
+        for target_key, sub_keys in self.target_keys.items():
+            batch[target_key] = []
+            for target_i_dict in target:
+                target_key_i = {sub_key: target_i_dict[sub_key] for sub_key in sub_keys}
+                batch[target_key].append(target_key_i)
+
+        return batch
+
+
+class SelectKeyTransform:
+    def __init__(self, *, key, transform, rename=None):
+        self.key = key
+        self.transform = transform
+        self.rename = rename
+
+    def __call__(self, target: dict):
+        new_key = self.rename or self.key
+
+        target[new_key] = self.transform(target[self.key])
+        if self.rename is not None:
+            del target[self.key]
+
+        return target
+
+
+class Method:
+    def __init__(self, *args, name, **kwargs):
+        self.name = name
+        self.args = args
+        self.kwargs = kwargs
+
+    def __call__(self, obj):
+        method = getattr(obj, self.name)
+        ret = method(*self.args, **self.kwargs)
+        return ret

examples/mart_armory/mart_armory/configs/assemble_attack.yaml

@@ -0,0 +1,9 @@
+# @package _global_
+
+# specify here default training configuration
+defaults:
+  - _self_
+  - attack: ???
+  - batch_converter: ???
+  - batch_c15n: ???
+  - model_transform: ???

examples/mart_armory/mart_armory/configs/attack/gain/rcnn_training_loss.yaml

@@ -0,0 +1,8 @@
+_target_: mart.nn.CallWith
+module:
+  _target_: mart.nn.Sum
+_call_with_args_:
+  - "training.loss_objectness"
+  - "training.loss_rpn_box_reg"
+  - "training.loss_classifier"
+  - "training.loss_box_reg"

examples/mart_armory/mart_armory/configs/batch_c15n/data_coco.yaml

@@ -0,0 +1,10 @@
+defaults:
+  - tuple
+  - [email protected]: pixel_1to255
+  - [email protected]: pixel_255to1
+
+transform:
+  _target_: mart.transforms.TupleTransforms
+
+untransform:
+  _target_: mart.transforms.TupleTransforms

examples/mart_armory/mart_armory/configs/batch_c15n/transform/pixel_1to255.yaml

@@ -0,0 +1,13 @@
+_target_: torchvision.transforms.Compose
+transforms:
+  - _target_: mart.transforms.Denormalize
+    center: 0
+    scale: 255
+  # Fix potential numeric error.
+  - _target_: torch.fake_quantize_per_tensor_affine
+    _partial_: true
+    # (x/1+0).round().clamp(0, 255) * 1
+    scale: 1
+    zero_point: 0
+    quant_min: 0
+    quant_max: 255

examples/mart_armory/mart_armory/configs/batch_c15n/transform/pixel_255to1.yaml

@@ -0,0 +1,3 @@
+_target_: torchvision.transforms.Normalize
+mean: 0
+std: 255