Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV 2025) · 2025

AIM - Amending Inherent Interpretability via Self-Supervised Masking

Eyad Alshami, Shashank Agnihotri, Bernt Schiele, Margret Keuper

Why this publication matters

A useful explanation should reflect what a classifier actually uses, rather than being added only after the decision. AIM uses masking during training to encourage reliance on meaningful object features. It improves the connection between recognition and visible evidence without requiring extra region annotations.

Abstract

It has been observed that deep neural networks (DNNs) often use both genuine as well as spurious features. In this work, we propose “Amending Inherent Interpretability via Self-Supervised Masking” (AIM), a simple yet interestingly effective method that promotes the network’s utilization of genuine features over spurious alternatives without requiring additional annotations. In particular, AIM uses features at multiple encoding stages to guide a selfsupervised, sample-specific feature-masking process. As a result, AIM enables the training of well-performing and inherently interpretable models that faithfully summarize the decision process. We validate AIM across a diverse range of challenging datasets that test both out-of-distribution generalization and fine-grained visual understanding. These include general-purpose classification benchmarks such as ImageNet100, HardImageNet, and ImageWoof, as well as fine-grained classification datasets such as Waterbirds, TravelingBirds, and CUB-200. AIM demonstrates significant dual benefits: interpretability improvements, as measured by the Energy Pointing Game (EPG) score, and accuracy gains over strong baselines. These consistent gains across domains and architectures provide compelling evidence that AIM promotes the use of genuine and meaningful features that directly contribute to improved generalization and human-aligned interpretability.

Abstract source ↗

Figures

Attribution maps showing where different image classifiers focus. — **Figure 1.** AIM uses self-supervised masking to focus more on the object of interest, relying only on the image label. As shown, and in terms of attribution localization, it outperforms baseline methods, even in challenging scenarios like the Water- Birds dataset. View in source ↗

**Figure 2.** Abstract Diagram of the [backbone]+AIM Architecture. The architecture consists of a bottom-up backbone and a top-down masking pathway. The bottom-up pathway has four encoding stages (L = 0 to L = 3). The top-down pathway mirrors this structure, with each stage T corresponding to a bottom-up stage L. Each top-down stage has two parallel branches: one estimates a binary mask via a convolutional network with Gumbel-softmax, and the other processes features using a structure inspired by [21]. The estimated binary mask is element-wise multiplied with the processed features to create a spatially sparse feature map. These sparse maps are then iteratively combined with the output of the subsequent top-down stage through element-wise summation. View in source ↗

Cite this paper

Download .bib

@inproceedings{alshami2025aimamendinginherent74,
  title = {{AIM - Amending Inherent Interpretability via Self-Supervised Masking}},
  author = {Eyad Alshami and Shashank Agnihotri and Bernt Schiele and Margret Keuper},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV 2025)},
  year = {2025},
  url = {https://arxiv.org/abs/2508.11502}
}

Figures and abstract are reproduced from the linked research sources. Credit remains with the authors and publishers.