Home | Publications | GBU+26

Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation With Nested Embeddings

MCML Authors

Link to Profile Peter Schüffler PI Matchmaking

Peter Schüffler

Prof. Dr.

Core PI

Computational Pathology

Abstract

Foundation models (FMs) have driven significant progress in computational pathology. These models can easily exceed a billion parameters and produce high–dimensional embeddings, thus limiting their applicability for research or clinical use when computing resources are tight. Here we introduce Pathryoshka, a novel multi–teacher distillation framework inspired by agglomerative models and Matryoshka representation learning to reduce pathology FM sizes while allowing for adaptable embedding dimensions. We evaluate our framework with a distilled model on ten public pathology benchmarks with varying downstream tasks. Compared to its much larger teachers, Pathryoshka reduces the model size by 86–92% at on–par performance. It outperforms state–of–the–art single–teacher distillation models of comparable size by a median margin of 7.0 and other pathology multi–teacher distillation by 5.3 percentage points in accuracy. By enabling efficient deployment without sacrificing accuracy or representational richness, Pathryoshka democratizes access to state–of–the–art pathology FMs for the broader research and clinical community.

inproceedings GBU+26

ECCV 2026

19th European Conference on Computer Vision. Malmö, Sweden, Sep 08-12, 2026.
Conference logo
A* Conference

Authors

C. Grashei • C. Brechenmacher • R. M. Umer • J. Liu • C. Marr • E. Szczurek • P. J. Schüffler

Links

DOI

Research Area

 C1 | Medicine

BibTeXKey: GBU+26

#p-marr #p-schueffler
Back to Top