Conflicting Biases at the Edge of Stability: Norm Versus Sharpness Regularization
MCML Authors
Vit Fojtik
Gitta Kutyniok
Prof. Dr.
Core PI
Johannes Maly
Prof. Dr.
Collaborating PI
Abstract
Vit Fojtik
Gitta Kutyniok
Prof. Dr.
Core PI
Johannes Maly
Prof. Dr.
Collaborating PI
Abstract
The remarkable generalization properties of overparameterized networks are often attributed to implicit biases, such as norm minimization at small learning rates and low sharpness in the Edge-of-Stability regime. In this work, we argue that a comprehensive understanding of the generalization performance of gradient descent requires analyzing the interaction between these various forms of implicit regularization. We empirically demonstrate that the learning rate interpolates between low parameter norm and low sharpness of the trained model. We furthermore prove that neither implicit bias alone minimizes the generalization error for diagonal linear networks trained on a simple regression task. These findings demonstrate that focusing on a single implicit bias is insufficient to explain good generalization, and they motivate a broader view of implicit regularization that captures the dynamic trade-off between norm and sharpness induced by non-negligible learning rates.
inproceedings MFC+26
ICML 2026
43rd International Conference on Machine Learning. Seoul, South Korea, Jul 06-11, 2026. To be published. Preprint available.
Authors
M. Matveev • V. Fojtik • H.-H. Chou • G. Kutyniok • J. MalyLinks
URLResearch Area
BibTeXKey: MFC+26