Stein Variational Newton Neural Network Ensembles

MCML Authors

Vincent Fortuin

Dr.

Collaborating PI

Bayesian Deep Learning

Abstract

Deep neural network ensembles are powerful tools for uncertainty quantification, which have recently been re-interpreted from a Bayesian perspective. However, current methods inadequately leverage second-order information of the loss landscape, despite the recent availability of efficient Hessian approximations. We propose a novel approximate Bayesian inference method that modifies deep ensembles to incorporate Stein Variational Newton updates. Our approach uniquely integrates scalable modern Hessian approximations, achieving faster convergence and more accurate posterior distribution approximations. We validate the effectiveness of our method on diverse regression and classification tasks, demonstrating superior performance with a significantly reduced number of training epochs compared to existing ensemble-based methods, while enhancing uncertainty quantification and robustness against overfitting.

inproceedings FMF24a

---

SPIGM @ICML 2024

Workshop on Structured Probabilistic Inference & Generative Modeling at the 41st International Conference on Machine Learning. Vienna, Austria, Jul 21-27, 2024.

---

Authors

K. Flöge • M. A. Moeed • V. Fortuin

---

Links

URL

---

Research Area

 A1 | Statistical Foundations & Explainability