Home | Publications | HKS+26

On the Uncertainty in Prior-Data Fitted Network Pretraining

MCML Authors

Link to Profile David Rügamer PI Matchmaking

David Rügamer

Prof. Dr.

Core PI

Statistics, Data Science and Machine Learning

Abstract

Prior-data fitted networks (PFNs) have recently emerged as a new paradigm for supervised machine learning by approximating Bayesian posterior predictive distributions via in-context learning. While leading to state-of-the-art predictive performance across a wide range of benchmarks, quantifying the approach's uncertainty is challenging. Using an empirical risk minimization perspective, we characterize possible sources of uncertainty in PFN predictions and empirically show that certain uncertainties cannot be reduced by simply scaling the model size or pretraining. To explicitly address these overlooked sources of uncertainty, we study deep ensembles of PFNs and formulate a Bayesian neural network version of PFNs for which we obtain samples from an approximate posterior via Markov chain Monte Carlo.

inproceedings HKS+26

FMSD @ICLR 2026

2nd Workshop on Foundation Models for Structured Data at the 14th International Conference on Learning Representations. Rio de Janeiro, Brazil, Apr 23-27, 2026.

Authors

M. Hülskamp • J. Kobialka • E. Sommer • D. Rügamer

Links

URL

Research Area

 A1 | Statistical Foundations & Explainability

BibTeXKey: HKS+26

#p-ruegamer
Back to Top