Differentiable Attention Sparsity via Structured D-Gating

MCML Authors

Chris Kolb

Dr.

→ Group David Rügamer
 Statistics, Data Science and Machine Learning

Bernd Bischl

Prof. Dr.

Director

Statistical Learning and Data Science

David Rügamer

Prof. Dr.

Core PI

Statistics, Data Science and Machine Learning

Abstract

A core component of modern large language models is the attention mechanism, but its immense parameter count necessitates structured sparsity for resource-efficient optimization and inference. Traditional sparsity penalties, such as the group lasso, are non-smooth and thus incompatible with standard stochastic gradient descent methods. To address this, we propose a deep gating mechanism that reformulates the structured sparsity penalty into a fully differentiable optimization problem, allowing effective and principled norm-based group sparsification without requiring specialized non-smooth optimizers. Our theoretical analysis and empirical results demonstrate that this approach enables structured sparsity with simple stochastic gradient descent or variants while maintaining predictive performance.

inproceedings KBR25

---

SLLM @ICLR 2025

Workshop on Sparsity in LLMs at the 13th International Conference on Learning Representations. Singapore, Apr 24-28, 2025.

---

Authors

C. Kolb • B. Bischl • D. Rügamer

---

Links

URL

---

Research Area

 A1 | Statistical Foundations & Explainability