Munich AI Lectures

Solvable High-Dimensional Models of Attention: A Theory of Generalization on Token Sequences

Lenka Zdeborova, École Polytechnique Fédérale de Lausanne (EPFL)

   29.01.2026

   3:00 pm - 4:00 pm

   TUM, Campus Munich, Theresienstr. 90, Room 0101.Z1.090

The lecture is dedicated to the theoretical analysis of attention layers, which today form the heart of modern machine learning architectures. The focus is on the question of how such systems generalize from data and what principles determine their learning behavior on sequences of tokens. Using analyzable high-dimensional models, learning and generalization performances are characterized in closed form for the first time.

The focus is on supervised learning scenarios in which the models enable precise theoretical predictions and provide mechanistic insights into the representational learning of attention-based architectures. Finally, an outlook is given on how these results pave the way for manageable theoretical models for self-supervised and generative training with attention.

Lenka Zdeborová is a professor of physics and computer science at the École Polytechnique Fédérale de Lausanne (EPFL) and leads the Statistical Physics of Computation Laboratory there. Her research combines methods of statistical physics with questions from machine learning, inference, and optimization. She has received numerous awards, including the CNRS Bronze Medal, the Irène Joliot-Curie Prize, and ERC Grants, and works on theoretical models that explain how modern AI systems learn, generalize, and scale.