Research Group Nassir Navab
Nassir Navab
holds the Chair of Computer Aided Medical Procedures & Augmented Reality at TU Munich.
His research focuses on computer-aided medical procedures and augmented reality. The work involves developing technologies to improve the quality of medical intervention and bridges the gap between medicine and computer science.
Team members @MCML
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
Publications @MCML
Diffusion as Sound Propagation: Physics-Inspired Model for Ultrasound Image Generation.
27th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2024). Marrakesh, Morocco, Oct 06-10, 2024. DOI. GitHub.
Abstract
Deep learning (DL) methods typically require large datasets to effectively learn data distributions. However, in the medical field, data is often limited in quantity, and acquiring labeled data can be costly. To mitigate this data scarcity, data augmentation techniques are commonly employed. Among these techniques, generative models play a pivotal role in expanding datasets. However, when it comes to ultrasound (US) imaging, the authenticity of generated data often diminishes due to the oversight of ultrasound physics.We propose a novel approach to improve the quality of generated US images by introducing a physics-based diffusion model that is specifically designed for this image modality. The proposed model incorporates an US-specific scheduler scheme that mimics the natural behavior of sound wave propagation in ultrasound imaging. Our analysis demonstrates how the proposed method aids in modeling the attenuation dynamics in US imaging. We present both qualitative and quantitative results based on standard generative model metrics, showing that our proposed method results in overall more plausible images.
MCML Authors
ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling.
27th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2024). Marrakesh, Morocco, Oct 06-10, 2024. DOI. GitHub.
Abstract
Every day, countless surgeries are performed worldwide, each within the distinct settings of operating rooms (ORs) that vary not only in their setups but also in the personnel, tools, and equipment used. This inherent diversity poses a substantial challenge for achieving a holistic understanding of the OR, as it requires models to generalize beyond their initial training datasets. To reduce this gap, we introduce ORacle, an advanced vision-language model designed for holistic OR domain modeling, which incorporates multi-view and temporal capabilities and can leverage external knowledge during inference, enabling it to adapt to previously unseen surgical scenarios. This capability is further enhanced by our novel data augmentation framework, which significantly diversifies the training dataset, ensuring ORacle’s proficiency in applying the provided knowledge effectively. In rigorous testing, in scene graph generation, and downstream tasks on the 4D-OR dataset, ORacle not only demonstrates state-of-the-art performance but does so requiring less data than existing models. Furthermore, its adaptability is displayed through its ability to interpret unseen views, actions, and appearances of tools and equipment. This demonstrates ORacle’s potential to significantly enhance the scalability and affordability of OR domain modeling and opens a pathway for future advancements in surgical data science.
MCML Authors
Deep Spectral Methods for Unsupervised Ultrasound Image Interpretation.
27th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2024). Marrakesh, Morocco, Oct 06-10, 2024. DOI.
Abstract
Ultrasound imaging is challenging to interpret due to non-uniform intensities, low contrast, and inherent artifacts, necessitating extensive training for non-specialists. Advanced representation with clear tissue structure separation could greatly assist clinicians in mapping underlying anatomy and distinguishing between tissue layers. Decomposing an image into semantically meaningful segments is mainly achieved using supervised segmentation algorithms. Unsupervised methods are beneficial, as acquiring large labeled datasets is difficult and costly, but despite their advantages, they still need to be explored in ultrasound. This paper proposes a novel unsupervised deep learning strategy tailored to ultrasound to obtain easily interpretable tissue separations. We integrate key concepts from unsupervised deep spectral methods, which combine spectral graph theory with deep learning methods. We utilize self-supervised transformer features for spectral clustering to generate meaningful segments based on ultrasound-specific metrics and shape and positional priors, ensuring semantic consistency across the dataset. We evaluate our unsupervised deep learning strategy on three ultrasound datasets, showcasing qualitative results across anatomical contexts without label requirements. We also conduct a comparative analysis against other clustering algorithms to demonstrate superior segmentation performance, boundary preservation, and label consistency.
MCML Authors
AMONuSeg: A Histological Dataset for African Multi-organ Nuclei Semantic Segmentation.
27th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2024). Marrakesh, Morocco, Oct 06-10, 2024. DOI. GitHub.
Abstract
Nuclei semantic segmentation is a key component for advancing machine learning and deep learning applications in digital pathology. However, most existing segmentation models are trained and tested on high-quality data acquired with expensive equipment, such as whole slide scanners, which are not accessible to most pathologists in developing countries. These pathologists rely on low-resource data acquired with low-precision microscopes, smartphones, or digital cameras, which have different characteristics and challenges than high-resource data. Therefore, there is a gap between the state-of-the-art segmentation models and the real-world needs of low-resource settings. This work aims to bridge this gap by presenting the first fully annotated African multi-organ dataset for histopathology nuclei semantic segmentation acquired with a low-precision microscope. We also evaluate state-of-the-art segmentation models, including spectral feature extraction encoder and vision transformer-based models, and stain normalization techniques for color normalization of Hematoxylin and Eosin-stained histopathology slides. Our results provide important insights for future research on nuclei histopathology segmentation with low-resource data.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion.
18th European Conference on Computer Vision (ECCV 2024). Milano, Italy, Sep 29-Oct 04, 2024. To be published. Preprint at arXiv. arXiv.
MCML Authors
SecondPose: SE(3)-Consistent Dual-Stream Feature Fusion for Category-Level Pose Estimation.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024). Seattle, WA, USA, Jun 17-21, 2024. DOI.
MCML Authors
MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024). Seattle, WA, USA, Jun 17-21, 2024. DOI.
MCML Authors
HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024). Seattle, WA, USA, Jun 17-21, 2024. DOI.
MCML Authors
Implicit Neural Representations for Breathing-compensated Volume Reconstruction in Robotic Ultrasound Aorta Screening.
IEEE International Conference on Robotics and Automation (ICR4 2024). Yokohoma, Japan, May 13-17, 2024. To be published. Preprint at arXiv. arXiv.
MCML Authors
CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graphs.
37th Conference on Neural Information Processing Systems (NeurIPS 2023). New Orleans, LA, USA, Dec 10-16, 2023. URL.
MCML Authors
Self-supervised Probe Pose Regression via Optimized Ultrasound Representations for US-CT Fusion.
International Conference on Medical Imaging and Computer-Aided Diagnosis (MICAD 2023). Vienna, Austria, Dec 09-10, 2023. DOI.
MCML Authors
Anatomy-Aware Masking for Inpainting in Medical Imaging.
3rd Workshop on Shape in Medical Imaging (ShapeMI 2023) at the 26th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2023). Vancouver, Canada, Oct 08-12, 2023. DOI.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
Transformers pay attention to convolutions leveraging emerging properties of vits by dual attention-image network.
IEEE/CVF International Conference on Computer Vision (ICCV 2023). Paris, France, Oct 02-06, 2023. DOI.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
Scenegenie: Scene graph guided diffusion models for image synthesis.
Workshops at the IEEE/CVF International Conference on Computer Vision (ICCV 2023). Paris, France, Oct 02-06, 2023. DOI.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
DisPositioNet: Disentangled Pose and Identity in Semantic Image Manipulation.
33rd British Machine Vision Conference (BMVC 2022). London, UK, Nov 21-24, 2022. URL.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
Y-Net: A Spatiospectral Dual-Encoder Network for Medical Image Segmentation.
25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022). Singapore, Sep 18-22, 2022. DOI.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
FedAP: Adaptive Personalization in Federated Learning for Non-IID Data.
3rd Workshop on Distributed, Collaborative, and Federated Learning, and Affordable AI and Healthcare for Resource Diverse Global Health (DeCaF FAIR 2022) at the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022). Singapore, Sep 18-22, 2022. DOI.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
MetaMedSeg: Volumetric Meta-learning for Few-Shot Organ Segmentation.
4th Workshop on Domain Adaptation and Representation Transfer (DART 2022) at the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022). Singapore, Sep 18-22, 2022. DOI.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
Interpretable Vertebral Fracture Diagnosis.
Workshop on Interpretability of Machine Intelligence in Medical Image Computing (iMIMIC 2022) at the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022). Singapore, Sep 18-22, 2022. DOI.
MCML Authors
Analyzing the Effects of Handling Data Imbalance on Learned Features from Medical Images by Looking Into the Models.
2nd Workshop on Interpretable Machine Learning in Healthcare (IMLH 2022) at the the 39th International Conference on Machine Learning (ICML 2022). Baltimore, MD, USA, Jul 17-23, 2022. arXiv.
MCML Authors
Mina Rezaei
Dr.
Statistical Learning & Data Science
Education Coordination
Bernd Bischl
Prof. Dr.
Statistical Learning & Data Science
Shape-Aware Masking for Inpainting in Medical Imaging.
Preprint at arXiv (Jul. 2022). arXiv.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
Do Explanations Explain? Model Knows Best.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022). New Orleans, LA, USA, Jun 19-24, 2022. DOI.
MCML Authors
Few-shot Structured Radiology Report Generation Using Natural Language Prompts.
Preprint at arXiv (Mar. 2022). arXiv.
MCML Authors
Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information.
Track on Datasets and Benchmarks at the 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Virtual, Dec 06-14, 2021. URL.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
MIGS: Meta Image Generation from Scene Graphs.
32nd British Machine Vision Conference (BMVC 2021). Virtual, Nov 22-25, 2021. URL.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
Unconditional Scene Graph Generation.
IEEE/CVF International Conference on Computer Vision (ICCV 2021). Virtual, Oct 11-17, 2021. DOI.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative
Towards Semantic Interpretation of Thoracic Disease and COVID-19 Diagnosis Models.
24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021). Strasbourg, France, Sep 27-Oct 01, 2021. DOI.
MCML Authors
Explaining COVID-19 and Thoracic Pathology Model Predictions by Identifying Informative Input Features.
24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021). Strasbourg, France, Sep 27-Oct 01, 2021. DOI.
MCML Authors
Neural Response Interpretation through the Lens of Critical Pathways.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021). Virtual, Jun 19-25, 2021. DOI.
MCML Authors
TeCNO: Surgical Phase Recognition with Multi-stage Temporal Convolutional Network.
23rd International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2020). Virtual, Oct 04-08, 2020. DOI.
MCML Authors
Spatio-temporal learning from longitudinal data for multiple sclerosis lesion segmentation.
Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (BrainLes 2020) at the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2020). Virtual, Oct 04-08, 2020. DOI.
MCML Authors
Inverse Distance Aggregation for Federated Learning with Non-IID Data.
Workshop on Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning (DART DCL 2020) at the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2020). Virtual, Oct 04-08, 2020. DOI.
MCML Authors
Azade Farshad
Dr.
Computer Aided Medical Procedures & Augmented Reality
Junior Representative