Width Independent Bounds for the Local Lipschitz Constant of Deep Neural Networks at Random Initialization and After Lazy Training

MCML Authors

Apostolos Evangelidis

→ Group Massimo Fornasier
 Applied Numerical Analysis

Felix Krahmer

Prof. Dr.

Principal Investigator

* Former Principal Investigator

Abstract

A plethora of recent works has shown that for wide, overparameterized neural networks, training with Stochastic Gradient Descent (SGD) often leads to interpolation of the training data without sacrificing generalization performance. A key parameter that is not only closely connected to generalization properties, but is also closely tied to other desiderata such as robustness and resistance to adversarial perturbations is the Lipschitz constant of the neural network. While empirically, the Lipschitz constant has been shown not to increase with network width, theoretical findings only provide bounds with logarithmic growth in the width and only for the random initialization of ReLU-networks. In this work, we close this gap for neural networks with smooth activations by showing that, both at random initialization and throughout lazy training, the local Lipschitz constant of deep neural networks does not increase with network width. More precisely, we establish novel non-asymptotic (finite width) upper bounds and corroborate them by numerical experiments.

inproceedings EK26

---

ICML 2026

43rd International Conference on Machine Learning. Seoul, South Korea, Jul 06-11, 2026. To be published. Preprint available.

---

Authors

A. Evangelidis • F. Krahmer

---

Links

URL

---

Research Area

 A2 | Mathematical Foundations