Covariance Descriptors Meet General Vision Encoders: Riemannian Deep Learning for Medical Image Classification

MCML Authors

Anna Reithmeir

→ Group Julia Schnabel
 Computational Imaging and AI in Medicine

Julia Schnabel

Prof. Dr.

Core PI

Computational Imaging and AI in Medicine

Abstract

Covariance descriptors capture second-order statistics of image features. They have shown strong performance in general computer vision tasks, but remain underexplored in medical imaging. We investigate their effectiveness for both conventional and learning-based medical image classification, with a particular focus on SPDNet, a classification network specifically designed for symmetric positive definite (SPD) matrices. We propose constructing covariance descriptors from features extracted by pre-trained general vision encoders (GVEs) and comparing them with handcrafted descriptors. Two GVEs - DINOv2 and MedSAM - are evaluated across eleven binary and multi-class datasets from the MedMNSIT benchmark. Our results show that covariance descriptors derived from GVE features consistently outperform those derived from handcrafted features. Moreover, SPDNet yields superior performance to state-of-the-art methods when combined with DINOv2 features. Our findings highlight the potential of combining covariance descriptors with powerful pretrained vision encoders for medical image analysis.

inproceedings MRD+26

---

ISBI 2026

IEEE 23rd International Symposium on Biomedical Imaging. London, UK, Apr 08-11, 2026. To be published. Preprint available.

---

Authors

J. Mayr • A. Reithmeir • M. Di Folco • J. A. Schnabel

---

Links

arXiv

---

Research Area

 C1 | Medicine