Language Models Meet Anomaly Detection for Better Interpretability and Generalizability
MCML Authors
Lina Felsner
Dr.
Daniel Rückert
Prof. Dr.
Director
Benedikt Wiestler
Prof. Dr.
Core PI
Julia Schnabel
Prof. Dr.
Core PI
Cosmin Bercea
Dr.
Abstract
Lina Felsner
Dr.
Daniel Rückert
Prof. Dr.
Director
Benedikt Wiestler
Prof. Dr.
Core PI
Julia Schnabel
Prof. Dr.
Core PI
Cosmin Bercea
Dr.
Abstract
This research explores the integration of language models and unsupervised anomaly detection in medical imaging, addressing two key questions: (1) Can language models enhance the interpretability of anomaly detection maps? and (2) Can anomaly maps improve the generalizability of language models in open-set anomaly detection tasks? To investigate these questions, we introduce a new dataset for multi-image visual question-answering on brain magnetic resonance images encompassing multiple conditions. We propose KQ-Former (Knowledge Querying Transformer), which is designed to optimally align visual and textual information in limited-sample contexts. Our model achieves a 60.81% accuracy on closed questions, covering disease classification and severity across 15 different classes. For open questions, KQ-Former demonstrates a 70% improvement over the baseline with a BLEU-4 score of 0.41, and achieves the highest entailment ratios (up to 71.9%) and lowest contradiction ratios (down to 10.0%) among various natural language inference models. Furthermore, integrating anomaly maps results in an 18% accuracy increase in detecting open-set anomalies, thereby enhancing the language model's generalizability to previously unseen medical conditions.
inproceedings LKM+24
MMMI @MICCAI 2024
5th International Workshop on Multiscale Multimodal Medical Imaging at the 27th International Conference on Medical Image Computing and Computer Assisted Intervention. Marrakesh, Morocco, Oct 06-10, 2024.Authors
J. Li • S. H. Kim • P. Müller • L. Felsner • D. Rückert • B. Wiestler • J. A. Schnabel • C. I. BerceaLinks
DOI GitHubResearch Area
BibTeXKey: LKM+24