Home | Publications | HK26

Proxy Scoring Enables Benchmarking LLM Forecasters Without Waiting for Outcomes

MCML Authors

Abstract

Large language models are increasingly used as general-purpose forecasters, but benchmarking their forecasting ability remains slow and fragile. Retrospective benchmarks risk training-data contamination, while prospective benchmarks require waiting weeks or months for questions to resolve. We propose an instantaneous evaluation method based on proxy scoring rules. Rather than scoring each forecast against the eventual outcome, we score it against an extremized aggregate of the forecasts made by other models, drawing on the literature on information elicitation without verification. Empirically these proxy scores correlate strongly with resolved-outcome metrics such as the Brier score on existing LLM forecasting data and are almost as predictive as Brier scores of future performance and substantially less noisy across time. We further show the effectiveness of the method crucially depends on the aggregation method used: simple means and medians can perform poorly, while logit-mean aggregation followed by extremization yields consistently strong correlations.

inproceedings HK26

Forecast @ICML 2026

Workshop on Forecasting as a New Frontier of Intelligence at the 43rd International Conference on Machine Learning. Seoul, South Korea, Jul 06-11, 2026. To be published. Preprint available.

Authors

J. Hege • G. Kutyniok

Links

URL

Research Area

 A2 | Mathematical Foundations

BibTeXKey: HK26

#p-kutyniok
Back to Top