Keuper Labs
← All publications

Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVw 2023) · 2023

On the unreasonable vulnerability of transformers for image restoration and an easy fix

Shashank Agnihotri, Kanchana Vaishnavi Gandikota, Julia Grabinski, Paramanand Chandramouli, Margret Keuper

Image RestorationVision TransformersAdversarial Robustness

Why this publication matters

An image-restoration model can produce excellent clean examples yet break badly under small, deliberate changes to its input. This work investigates that gap in transformer-based restoration and explores changes that improve resistance. It makes robustness part of judging restoration quality, alongside how sharp the output looks.

Abstract

Following their success in visual recognition tasks, Vision Transformers(ViTs) are being increasingly employed for image restoration. As a few recent works claim that ViTs for image classification also have better robustness properties, we investigate whether the improved adversarial robustness of ViTs extends to image restoration. We consider the recently proposed Restormer model, as well as NAFNet and the “Baseline network” which are both simplified versions of a Restormer. We use Projected Gradient Descent (PGD) and CosPGD, a recently proposed adversarial attack tailored to pixel-wise prediction tasks for our robustness evaluation. Our experiments are performed on realworld images from the GoPro dataset for image deblurring. Our analysis indicates that contrary to as advocated by ViTs in image classification works, these models are highly susceptible to adversarial attacks. We attempt to improve their robustness through adversarial training. While this yields a significant increase in robustness for Restormer, results on other networks are less promising. Interestingly, the design choices in NAFNet and Baselines, which were based on iid performance, and not on robust generalization, seem to be at odds with the model robustness. Thus, we investigate this further and find a fix.

Abstract source ↗

Figures

Image-restoration outputs corrupted by adversarial attacks.
Figure 1. Comparing images reconstructed by all considered models after 5 iterations of CosPGD attack . We observe strong spectral artifacts in the reconstructed images. View in source ↗
Modified visualization of repeating blocks of the architectures from [7] the considered Intermediate network from [7] (please refer to (c)) and Intermediate + ReLU network (please refer to (d)).
Figure 2. Modified visualization of repeating blocks of the architectures from [7] the considered Intermediate network from [7] (please refer to (c)) and Intermediate + ReLU network (please refer to (d)). View in source ↗

Cite this paper

Download .bib 
@inproceedings{agnihotri2023ontheunreasonable33,
  title = {{On the unreasonable vulnerability of transformers for image restoration and an easy fix}},
  author = {Shashank Agnihotri and Kanchana Vaishnavi Gandikota and Julia Grabinski and Paramanand Chandramouli and Margret Keuper},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVw 2023)},
  year = {2023},
  url = {https://arxiv.org/abs/2307.13856}
}

Figures and abstract are reproduced from the linked research sources. Credit remains with the authors and publishers.