The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models

MCML Authors

Timo Freiesleben

Dr.

→ Group Tom Sterkenburg
 Epistemological Foundations of Machine Learning

Abstract

Predictive benchmarking, evaluating machine learning models based on predictive performance and competitive ranking, is central to machine learning research and scientific inquiry. However, benchmark scores at best measure performance relative to a specific dataset and learning problem. Drawing substantial scientific inferences requires additional assumptions. Adapting ideas from psychological validity theory, we propose validity conditions that make these assumptions explicit. In two case studies—ImageNet and the Fragile Families Challenge—we show how benchmark results can support inferences about research progress and limits of predictability, situating predictive benchmarking as a distinct epistemic practice in machine learning.

article FZ26

---

Philosophy of Science

Early Access. Oct. 2026.

---

Authors

T. Freiesleben • S. Zezulka

---

Links

DOI

---

Research Area

 A2 | Mathematical Foundations