Routing-Free Mixture-of-Experts
MCML Authors
Volker Tresp
Prof. Dr.
Core PI
Abstract
Volker Tresp
Prof. Dr.
Core PI
Abstract
Standard Mixture-of-Experts (MoE) designs rely heavily on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE, which eliminates all hard-coded centralized components, including external routers, Softmax, TopK, and fixed load balancing. All related functionalities become encapsulated within individual experts, allowing each expert to determine its activation entirely on its own. We introduce a unified, adaptively scheduled load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a customizable interpolation, letting optimal resource allocation pattern emerge locally during training. Extensive experiments demonstrate that Routing-Free MoE achieves consistently better performance, efficiency, and robustness than baseline approaches at both training and deployment. We further analyze its emergent behavior in detail and offer insights that may facilitate future MoE design and optimization.
inproceedings LHY+26
EMNLP 2026
Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026. To be published. Preprint available.
Authors
Y. Liu • J. Han • S. Yan • V. Tresp • Y. MaLinks
URL GitHubResearch Area
BibTeXKey: LHY+26