Home | Publications | ARS+26

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

MCML Authors

Abstract

Artificial intelligence benchmarks are an important mechanism to measure model progress and guide deployment decisions. However, benchmarks quickly “saturate”, making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches.

inproceedings ARS+26


ICML 2026

43rd International Conference on Machine Learning. Seoul, South Korea, Jul 06-11, 2026.
Conference logo
A* Conference

Authors

M. Akhtar • A. Reuel • P. Soni • S. Ahuja • P. S. Ammanamanchi • R. Rawal • V. Zouhar • S. Yadav • C. Whitehouse • D. Ki • J. Mickel • L. Choshen • M. Šuppa • J. Batzner • J. Chim • J. Sania • Y. Long • H. A. Rahmani • C. Knight • Y. Nan • J. Raj • Y. Fan • S. Singh • S. Sahoo • E. Habba • U. Gohar • S. Pawar • R. Scholz • A. Subramonian • J. Ni • M. Kochenderfer • S. Koyejo • M. Sachan • S. Biderman • Z. Talat • A. Ghosh • I. Solaiman

Links

URL

In Collaboration

 Cohere
 IBM


Research Area

 A1 | Statistical Foundations & Explainability

BibTeXKey: ARS+26

Back to Top