Home  | Publications | LKM+24

Language Models Meet Anomaly Detection for Better Interpretability and Generalizability

MCML Authors

Link to Profile Benedikt Wiestler PI Matchmaking

Benedikt Wiestler

Prof. Dr.

Principal Investigator

AI for Image-Guided Diagnosis and Therapy

Link to Profile Julia Schnabel PI Matchmaking

Julia Schnabel

Prof. Dr.

Principal Investigator

Computational Imaging and AI in Medicine

Abstract

This research explores the integration of language models and unsupervised anomaly detection in medical imaging, addressing two key questions: (1) Can language models enhance the interpretability of anomaly detection maps? and (2) Can anomaly maps improve the generalizability of language models in open-set anomaly detection tasks? To investigate these questions, we introduce a new dataset for multi-image visual question-answering on brain magnetic resonance images encompassing multiple conditions. We propose KQ-Former (Knowledge Querying Transformer), which is designed to optimally align visual and textual information in limited-sample contexts. Our model achieves a 60.81% accuracy on closed questions, covering disease classification and severity across 15 different classes. For open questions, KQ-Former demonstrates a 70% improvement over the baseline with a BLEU-4 score of 0.41, and achieves the highest entailment ratios (up to 71.9%) and lowest contradiction ratios (down to 10.0%) among various natural language inference models. Furthermore, integrating anomaly maps results in an 18% accuracy increase in detecting open-set anomalies, thereby enhancing the language model's generalizability to previously unseen medical conditions.

inproceedings

MMMI @MICCAI 2024

5th International Workshop on Multiscale Multimodal Medical Imaging at the 27th International Conference on Medical Image Computing and Computer Assisted Intervention. Marrakesh, Morocco, Oct 06-10, 2024.

Authors

J. Li • S. H. Kim • P. Müller • L. Felsner • D. Rückert • B. Wiestler • J. A. Schnabel • C. I. Bercea

Links

DOI GitHub

Research Area

 C1 | Medicine

BibTeXKey: LKM+24

#p_rueckert #p_schnabel #p_wiestler
Back to Top