When Flatter Is Not Always Better

Imagine hiking through a mountain landscape. A broad, flat valley may look like the easiest route. Yet the flattest path does not necessarily lead where you want to go.

A similar idea appears in the mathematical landscapes used to describe how neural networks learn. New research explores what happens when training reaches exceptionally flat regions and why the flattest destination is not always the best one.

Understanding the Landscape

In their ICML 2026 paper Flatland: The Adventures of Gradient Descent with Large Step Sizes, MCML Junior Members Leonardo Galli and Wiebke Bartolomaeus, MCML PI Holger Rauhut, and collaborators Curtis Fox and Mark Schmidt study how large the steps in neural network training can become while training still converges.

Neural network training can be pictured as movement through a landscape. Each location represents a possible configuration of the network. The height represents the training loss, a measure of how poorly the network currently performs on the training data. Training searches for locations with low loss.

The shape of the surrounding landscape also matters. A narrow valley changes steeply when the network parameters are adjusted. In a broad, flat valley, small changes have much less effect on the loss. This sensitivity is often described in terms of sharpness: flatter regions have lower sharpness. Such regions have attracted considerable attention because earlier research has linked low sharpness with better performance on unseen data, although this relationship remains debated.

Taking Larger Steps

The paper starts from a practical problem. Neural networks are commonly trained by repeatedly adjusting their parameters to reduce the loss - back to the analogy above, this is the path of the multi-dimensional hiker. The size of each adjustment is controlled by the learning rate. Large steps can make training faster, but steps that are too large can cause training to fail. Finding the largest useful step while keeping training stable has therefore been a longstanding challenge.

The researchers define what constitutes a large step size and develop adaptive methods that select such steps automatically using gradient and function information. This allows training to stay close to the largest step size that still keeps optimization stable from the beginning of training.

When they push the step size toward this limit, training often moves into regions with the lowest possible sharpness in their setting. The paper refers to this exceptionally flat part of the loss landscape as Flatland.

When Flatness Becomes a Problem

Some training runs reach these extremely flat regions very early and then make little progress. Their accuracy remains close to random guessing, while the gradients used to update the hidden layers become very small. As a result, these layers can no longer learn useful features. The authors describe these runs as unsuccessfully-flat.

Other runs behave differently. They stay close to the flattest regions without reaching the absolute minimum in sharpness and often achieve low training loss and high test accuracy. The authors refer to these runs as successfully-flat. The researchers then adjust their method to limit the largest steps and show that this can turn unsuccessfully-flat runs into successfully-flat ones.

Rethinking Flatness

The findings do not imply that flatness is unimportant. Instead, they show that reaching the lowest possible sharpness is not enough for successful training. In the experiments, networks that enter extremely flat regions too early can stop making meaningful progress, while slightly sharper regions lead to better training outcomes.

The work also leaves open questions. One is why using very large step sizes drives training so efficiently toward globally flat saddle points. The authors also note that further theoretical work is needed to understand convergence in the stochastic setting.

Why This Matters

The work provides a systematic way to study neural network training with unusually large steps. It also adds an important qualification to the common intuition that flatter solutions are preferable: a network can reach an extremely flat region and still fail to learn the task.

The results offer a more nuanced way to examine the relationship between learning rates, the geometry of the loss landscape, and successful feature learning, without treating maximum flatness as a goal in itself.

©AI-generated

Further Reading & Reference

Want to see how the researchers define large step sizes and investigate what happens in “Flatland”? The full paper presents the theoretical framework, algorithms, and experiments behind these results.

L. Galli • C. Fox • W. Bartolomaeus • M. Schmidt • H. Rauhut
Flatland: The Adventures of Gradient Descent with Large Step Sizes.
ICML 2026 - 43rd International Conference on Machine Learning. Seoul, South Korea, Jul 06-11, 2026. To be published. Preprint available. URL GitHub