The Learning Rate Does More Than Set the Pace

A neural network can fit its training examples in many different ways. The harder question is which of these possible solutions will continue to perform well on new data, and whether the choices made during training quietly determine the answer.

The Learning Rate Shapes the Solution

In their paper Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization, MCML Junior Member Maria Matveev, former MCML member Vit Fojtik, external collaborator Hung-Hsu Chou, and MCML PIs Gitta Kutyniok and Johannes Maly examine how two forms of implicit regularization interact during neural network training. Their work was accepted at ICML 2026, one of the world’s leading conferences in machine learning.

The researchers focus on two properties that have been used to explain why highly overparameterized neural networks can nevertheless perform well on unseen data: small parameter norms and low sharpness. Their results show that these properties do not necessarily identify the same solution.

When Two Helpful Biases Disagree

A neural network can often fit the same training data in many different ways. Training therefore does more than simply reduce errors: it also determines which of these possible solutions the model ends up using. Earlier research has shown that very small learning rates can favor solutions with small parameter values. Other studies have found that larger learning rates can lead to flatter solutions, meaning that small changes to the model have less effect on its training performance.

Both properties are often linked to better performance on new, unseen data. But they do not necessarily point to the same solution. Looking at them separately therefore leaves an important question unanswered: What happens when both influence the same training process?

A Trade-Off Controlled by the Learning Rate

The researchers identify a critical learning rate that separates two regimes. Below this threshold, norm and sharpness remain close to the solutions favored by very small learning rates.

Above it, increasing the learning rate generally reduces sharpness but increases the parameter norm. The learning rate therefore does more than set the step size: it changes which solutions the training process prefers.

Neither Extreme Is Always Best

The experiments show that:

• Below the critical learning rate, norm and sharpness remain relatively stable
• Above it, sharpness decreases while parameter norms increase
• In several settings, the lowest test error occurs at an intermediate learning rate

The theoretical analysis supports this result: in a simplified regression model, neither the minimum-norm solution nor the minimum-sharpness solution achieves the lowest expected generalization error.

Why It Matters

The findings challenge explanations that focus only on small parameter norms or low sharpness. They also show that the learning rate can act as a form of regularization, influencing which solution a neural network ultimately finds.

A broader understanding of generalization therefore requires looking at how different biases interact during training.

Further Reading & Reference

Read the full paper, accepted at ICML 2026, to explore how the learning rate shifts the balance between low parameter norms and low sharpness, and why neither extreme necessarily leads to the best generalization.

M. Matveev • V. Fojtik • H.-H. Chou • G. Kutyniok • J. Maly
Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization.
ICML 2026 - 43rd International Conference on Machine Learning. Seoul, South Korea, Jul 06-11, 2026. To be published. Preprint available. URL