16.10.2025

 

SIC: Making AI Image Classification Understandable

MCML Research Insight - With Tom Nuno Wolf, Emre Kavak, Fabian Bongratz, and Christian Wachinger

The Best of Both Worlds: Combining Intuition with Rigor

Imagine a radiologist identifying a condition. They instinctively compare the scan to thousands of past cases they’ve seen, a process known as case-based reasoning.

SIC leverages the same intuition and integrates a similarity-based classification mechanism and B-cos neural networks, which provide faithful, pixel-level contribution maps. First, SIC learns a set of class-representative latent vectors to act as “textbook” examples (Support Samples). A test sample is classified by computing and summing similarity scores of its latent vector and the latent vectors of the support samples.

As shown in Figure 1, this provides multifaceted explanations that include the predicted class’s support samples and contribution maps, their numerical evidence, and the test sample’s contribution maps.

©Tom Nuno Wolf et al.

Figure 1: The multi-faceted explanation provided by SIC. For a given test image, SIC provides a set of learned Support Samples for each class. The Contribution Maps are generated via the B-cos encoder, faithfully highlighting the pixels that contribute to the similarity score between the test sample and the latent vectors of the support samples. The Evidence score quantifies this similarity, showing the influence of each Support Sample on the final classification. This allows a user to interrogate the model's decision by examining which Support Samples were most influential and what specific image features drove that influence.

Findings and Implications for Medical Image Analysis

It is often argued that interpretability comes at the cost of model performance. However, researchers working in the domain have continuously provided evidence that may be a misconception. The authors showed that SIC achieves comparable performance across a number of tasks, ranging from fine-grained to multi-label to medical classification. Moreover, the theoretical evaluation shows that the explanations satisfy established axioms, which manifest in their empirical evaluation with the synthetic FunnyBirds framework. These results and findings are what the authors were looking for in interpretable methods for deep learning - a transparent classifier providing theoretically grounded and easily accessible explanations for deployment in clinical settings.

Interested in Exploring Further?

Check out the code and the full paper accepted at the A*-conference ICCV 2025, one of the most prestigious conferences in the field of computer vision.

T. N. Wolf, E. Kavak, F. Bongratz and C. Wachinger.
SIC: Similarity-Based Interpretable Image Classification with Neural Networks.
ICCV 2025 - IEEE/CVF International Conference on Computer Vision. Honolulu, Hawai’i, Oct 19-23, 2025. To be published. Preprint available. arXiv GitHub

Abstract

The deployment of deep learning models in critical domains necessitates a balance between high accuracy and interpretability. We introduce SIC, an inherently interpretable neural network that provides local and global explanations of its decision-making process. Leveraging the concept of case-based reasoning, SIC extracts class-representative support vectors from training images, ensuring they capture relevant features while suppressing irrelevant ones. Classification decisions are made by calculating and aggregating similarity scores between these support vectors and the input’s latent feature vector. We employ B-Cos transformations, which align model weights with inputs, to yield coherent pixel-level explanations in addition to global explanations of case-based reasoning. We evaluate SIC on three tasks: fine-grained classification on Stanford Dogs and FunnyBirds, multi-label classification on Pascal VOC, and pathology detection on the RSNA dataset. Results indicate that SIC not only achieves competitive accuracy compared to state-of-the-art black-box and inherently interpretable models but also offers insightful explanations verified through practical evaluation on the FunnyBirds benchmark. Our theoretical analysis proves that these explanations fulfill established axioms for explanations. Our findings underscore SIC’s potential for applications where understanding model decisions is as critical as the decisions themselves.

MCML Authors

Tom Nuno Wolf

→ Group Christian Wachinger
 Artificial Intelligence in Medical Imaging

Emre Kavak

→ Group Christian Wachinger
 Artificial Intelligence in Medical Imaging

Fabian Bongratz

→ Group Christian Wachinger
 Artificial Intelligence in Medical Imaging

Christian Wachinger

Prof. Dr.

Principal Investigator

Artificial Intelligence in Medical Imaging