Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success

MCML Authors

Suvrit Sra

Prof. Dr.

Core PI

Resource Aware Machine Learning

Abstract

Non-Euclidean optimisation methods with matrix-valued updates, such as Muon and Scion, have recently shown strong empirical performance for training Transformer models, yet their theoretical advantages over Euclidean methods remain poorly understood. We address this gap in the heavy-tailed non-convex regime, where stochastic gradients have bounded p-th central moments, p∈(1,2]. We show that certain non-Euclidean methods achieve optimal sample complexity under stronger stationarity measures, while Euclidean methods incur additional dimension-dependent costs. As a consequence, for m×n matrices, Muon finds an ε-stationary point in nuclear norm within (min{m,n}Δ1Lε2(σε)pp−1) samples, absorbing heavy-tailed noise without extra dimension dependence, unlike Euclidean methods. We further prove this sample complexity, including its dimension dependence, is optimal for all first-order methods under nuclear-norm stationarity. Experiments on large language models support our theory. Surprisingly, our results suggest that other Schatten geometries beyond the spectral geometry of Muon can perform competitively in certain settings.

misc HPS26

---

Preprint

Jun. 2026

---

Authors

F. Hübler • T. Pethick • S. Sra

---

Links

arXiv GitHub

---

Research Area

 A2 | Mathematical Foundations