ORQA: A Benchmark and Foundation Model for Holistic Operating Room Modeling

MCML Authors

Ege Özsoy

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

Chantal Pellegrini

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

David Bani-Harouni

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

Kun Yuan

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

Matthias Keicher

Dr.

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

Nassir Navab

Prof. Dr.

Principal Investigator

Computer Aided Medical Procedures & Augmented Reality

Abstract

The real-world complexity of surgeries necessitates surgeons to have deep and holistic comprehension to ensure precision, safety, and effective interventions. Computational systems are required to have a similar level of comprehension within the operating room. Prior works, limited to single-task efforts like phase recognition or scene graph generation, lack scope and generalizability. In this work, we introduce ORQA, a novel OR question answering benchmark and foundational multimodal model to advance OR intelligence. By unifying all four public OR datasets into a comprehensive benchmark, we enable our approach to concurrently address a diverse range of OR challenges. The proposed multimodal large language model fuses diverse OR signals such as visual, auditory, and structured data, for a holistic modeling of the OR. Finally, we propose a novel, progressive knowledge distillation paradigm, to generate a family of models optimized for different speed and memory requirements. We show the strong performance of ORQA on our proposed benchmark, and its zero-shot generalization, paving the way for scalable, unified OR modeling and significantly advancing multimodal surgical intelligence. We will release our code and data upon acceptance.

misc

Preprint

Authors

Links

Research Area

 C1 | Medicine