Home | Publications | KGP+26

3D Masked Autoencoders Are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy

MCML Authors

Amirhossein Kardoost

Dr.

→ Group Carsten Marr
Artificial Intelligence in Cell Therapy and Hematology

Carsten Marr

Prof. Dr.

Core PI

Artificial Intelligence in Cell Therapy and Hematology

Abstract

Self-supervised learning in fluorescence microscopy often relies on 2D projections, despite the inherently three-dimensional nature of cells. We present a systematic comparison of 2D and 3D masked autoencoders (MAE-2D vs. MAE-3D) on volumetric microscopy data. Under matched architectures and training protocols, MAE-3D consistently outperforms 2D max-projection and slice-based variants on downstream single-cell tasks. We further align visual representations with a pretrained protein language model (ESM2) and show that cross-modal supervision yields larger gains for volumetric models. Channel cross-attention and frequency-domain regularization are critical for leveraging 3D spatial context. On a protein--protein interaction task, MAE-3D achieves a ROC--AUC of 0.865, outperforming prior methods by up to +0.025. For protein localization, our best 3D model attains state-of-the-art AUC_{rm micro} (0.952) and F1_{rm micro} (0.742), improving over previous approaches by +0.003 and +0.010 absolute, respectively. Overall, these results demonstrate the advantages of native 3D modeling and multimodal alignment for representation learning in single-cell microscopy.

inproceedings KGP+26

MICCAI 2026

29th International Conference on Medical Image Computing and Computer Assisted Intervention. Strasbourg, France, Sep 27-Oct 01, 2026. To be published. Preprint available.