Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

MCML Authors

Vincent Bürgin

→ Group Stefanie Jegelka
 Foundations of Deep Neural Networks

Daniel Herbst

→ Group Stefanie Jegelka
 Foundations of Deep Neural Networks

Ya-Wei Eileen Lin

→ Group Stefanie Jegelka
 Foundations of Deep Neural Networks

Stefanie Jegelka

Prof. Dr.

Core PI

Foundations of Deep Neural Networks

Abstract

Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.

inproceedings BHL+26

---

ICML 2026

43rd International Conference on Machine Learning. Seoul, South Korea, Jul 06-11, 2026. To be published. Preprint available.

---

Authors

V. Bürgin • D. Herbst • Y.-W. E. Lin • S. Jegelka

---

Links

URL GitHub

---

Research Area

 A3 | Computational Models