Variational Learning Is Effective for Large Deep Networks

MCML Authors

Yuesong Shen

Dr.

* Former Member

→ Group Daniel Cremers
 Computer Vision & Artificial Intelligence

Daniel Cremers

Prof. Dr.

Director

Computer Vision & Artificial Intelligence

Abstract

We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks. We show that an optimizer called Improved Variational Online Newton (IVON) consistently matches or outperforms Adam for training large networks such as GPT-2 and ResNets from scratch. IVON’s computational costs are nearly identical to Adam but its predictive uncertainty is better. We show several new use cases of IVON where we improve finetuning and model merging in Large Language Models, accurately predict generalization error, and faithfully estimate sensitivity to data. We find overwhelming evidence that variational learning is effective.

inproceedings SDC+24

---

ICML 2024

41st International Conference on Machine Learning. Vienna, Austria, Jul 21-27, 2024.

---

Authors

Y. Shen • N. Daheim • B. Cong • P. Nickl • G. M. Marconi • C. Bazan • R. Yokota • I. Gurevych • D. Cremers • M. E. Khan • T. Möllenhoff

---

Links

URL GitHub

---

Research Area

 B1 | Computer Vision