Home  | Publications | SFN+25

Constructing Confidence Intervals for 'The' Generalization Error – A Comprehensive Benchmark Study

MCML Authors

Link to Profile Thomas Nagler

Thomas Nagler

Prof. Dr.

Principal Investigator

Computational Statistics & Data Science

Link to Profile Anne-Laure Boulesteix

Anne-Laure Boulesteix

Prof. Dr.

Principal Investigator

Biometry in Molecular Medicine

Link to Profile Bernd Bischl PI Matchmaking

Bernd Bischl

Prof. Dr.

Director

Statistical Learning and Data Science

Abstract

When assessing the quality of prediction models in machine learning, confidence intervals (CIs) for the generalization error, which measures predictive performance, are a crucial tool. Luckily, there exist many methods for computing such CIs and new promising approaches are continuously being proposed. Typically, these methods combine various resampling procedures, most popular among them cross-validation and bootstrapping, with different variance estimation techniques. Unfortunately, however, there is currently no consensus on when any of these combinations may be most reliably employed and how they generally compare. In this work, we conduct a large-scale study comparing CIs for the generalization error, the first one of such size, where we empirically evaluate 13 different CI methods on a total of 19 tabular regression and classification problems, using seven different inducers and a total of eight loss functions.We give an overview of the methodological foundations and inherent challenges of constructing CIs for the generalization error and provide a concise review of all 13 methods in a unified framework. Finally, the CI methods are evaluated in terms of their relative coverage frequency, width, and runtime. Based on these findings, we can identify a subset of methods that we would recommend.We also publish the datasets as a benchmarking suite on OpenML and our code on GitHub to serve as a basis for further studies.

article

Journal of Data-centric Machine Learning Research

2.6. Jan. 2025. To be published. Preprint available.

Authors

H. Schulz-Kümpel • S. Fischer • T. Nagler • A.-L. Boulesteix • B. Bischl • R. Hornung

Links

URL

Research Area

 A1 | Statistical Foundations & Explainability

BibTeXKey: SFN+25

Back to Top