Keuper Labs
← All publications

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2023) · 2023

Neural Architecture Design and Robustness -- A Dataset

Steffen Jung1, Jovita Lukasik, Margret Keuper

Neural Architecture SearchAdversarial RobustnessDatasetsBenchmarking

Why this publication matters

Choosing a network architecture also means choosing how it will behave when its inputs change. This benchmark records robustness across a large collection of designs and demonstrates ways to use those measurements. It makes the relationship between architecture and reliability easier to investigate and compare.

Abstract

Finding architectures that are (more) robust against perturbations requires expensive evaluations. We introduce a database on neural architecture design and robustness evaluations to facilitate research in this direction. For this, we evaluate a whole neural architecture search space (NAS-Bench-201) on a range of common adversarial attacks and corruption types. We further present three exemplary use cases of this dataset, in which we (i) benchmark robustness measurements based on Jacobian and Hessian matrices for their robustness predictability, (ii) perform neural architecture search on robust accuracies, and (iii) provide an initial analysis of how architectural design choices affect robustness. We find that carefully crafting the topology of a network can have substantial impact on its robustness, where networks with the same parameter count range in mean adversarial robust accuracy from 20% −41%. Code and data is available at http://robustness.vision/.

Abstract source ↗

Figures

NAS-Bench-201 macro architecture and candidate cell operations.
Figure 1. (top) Macro architecture. Gray highlighted cells differ between architectures, while the other components stay fixed. (bottom) Cell structure and the set of possible, predefined operations. (Figure adapted from [5]) View in source ↗
(top) Accuracy (FGSM [6] and PGD [11]) boxplots over all 6 466 unique architectures in NAS-Bench-201 for different perturbation magnitudes evaluated on CIFAR-10.
Figure 2. (top) Accuracy (FGSM [6] and PGD [11]) boxplots over all 6 466 unique architectures in NAS-Bench-201 for different perturbation magnitudes evaluated on CIFAR-10. (bottom) Accuracy boxplots for different corruption types at different severity levels evaluated on CIFAR-10-C. Red line corresponds to guessing. View in source ↗

Cite this paper

Download .bib 
@inproceedings{jung12023neuralarchitecturedesign29,
  title = {{Neural Architecture Design and Robustness -- A Dataset}},
  author = {Steffen Jung1 and Jovita Lukasik and Margret Keuper},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2023)},
  year = {2023},
  url = {https://robustart.github.io/short_paper/33.pdf}
}

Figures and abstract are reproduced from the linked research sources. Credit remains with the authors and publishers.