Teaching Language Models to Express Uncertainty

An AI assistant may give a confident answer even when it is wrong. But what if it could tell us how sure it really is? MCML researchers have developed a reinforcement learning approach that teaches language models to express more reliable confidence, helping us recognize when to trust an answer and when to double-check.

Why a Confident Answer Is Not Always a Reliable One

In the paper Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models, MCML Junior Members David Bani-Harouni, Chantal Pellegrini, Ege Özsoy, Kamilia Zaripova, and Matthias Keicher, together with MCML PI Nassir Navab and co-author Paul Stangel, investigate how language models can express more reliable confidence in their answers. The paper was presented at the International Conference on Learning Representations (ICLR 2026).

The growing interest in reliable AI confidence estimates is also reflected in recent developments such as TypeSafe AI’s JEV Decision Model. Independent analyses have highlighted Rewarding Doubt as earlier research addressing the same fundamental challenge: training AI systems with reinforcement learning to express confidence that matches their actual reliability. This connection shows how the MCML team’s work contributes to a research direction that is now attracting broader attention. For further context, see the discussions by Motion and Madness and Anthony Maio.

Confidence is calibrated when the model’s stated certainty matches how often it is correct. For example, among answers given with 80% confidence, roughly 80% should be correct. This matters when an AI response informs a decision that may need further checking.

Why Models Struggle to Judge Their Own Answers

Language models often provide answers in a fluent, assured tone, even when those answers contain mistakes. Simply asking a model to state its confidence does not guarantee that the number reflects its actual reliability.

Earlier approaches often estimated confidence by analyzing the model’s outputs or using a separate mechanism. The researchers instead ask whether confidence can be trained directly into the model’s response process.

Learning Through Rewards and Penalties

The researchers introduce Rewarding Doubt, a reinforcement-learning method that trains language models to express a numerical confidence estimate alongside an answer. Reinforcement learning improves a model’s behavior by assigning rewards to its outputs during training.

Their reward follows a logarithmic scoring rule: a correct answer with high confidence earns a stronger reward, while an incorrect answer given with high confidence receives a stronger penalty. The design also discourages unnecessary doubt about correct answers. During training, the answer is generated first and kept fixed while the confidence expression is optimized, allowing the researchers to focus on calibration rather than changing the answer itself.

©Bani-Harouni et al.

Figure 1: Illustration of the reward design: The model is rewarded for a high confidence if it is correct and punished if it is incorrect. To maximize the reward, the model needs to learn when to predict a higher or lower confidence, considering a possible higher punishment.

More Accurate Confidence, Including on New Tasks

The researchers evaluate their method on factual question answering with single and multiple answers. Compared with simply asking an untrained model to state its confidence, Rewarding Doubt substantially reduces calibration error. On the TriviaQA benchmark, expected calibration error falls from 0.3459 to 0.0226. A separately trained confidence probe achieves similarly strong calibration, with a slightly lower calibration error on this benchmark. Unlike Rewarding Doubt, however, this approach does not train the language model itself to express calibrated confidence.

The team also tests models of different sizes and architectures and evaluates performance on previously unseen question-answering tasks, including medical questions. The results show improved confidence estimates across these settings, while answer accuracy remains broadly stable in the reported experiments.

Knowing When to Double-Check

A confidence estimate is useful only when it corresponds to the model’s actual likelihood of being correct. Rewarding Doubt shows that language models can be trained to communicate that uncertainty more reliably, without requiring a separate confidence model at the point of use.

Whether the approach performs similarly on much larger models or more open-ended responses remains to be tested. The work provides a research basis for AI systems that make uncertainty visible rather than hiding it behind fluent language.

Further Reading & Reference

Want to see how language models can learn when to express confidence and when to express doubt? The full ICLR 2026 paper explains the reward design and evaluates how well the resulting confidence estimates match real answer accuracy.

D. Bani-Harouni • C. Pellegrini • P. Stangel • E. Özsoy • K. Zaripova • N. Navab • M. Keicher
Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models.
ICLR 2026 - 14th International Conference on Learning Representations. Rio de Janeiro, Brazil, Apr 23-27, 2026. To be published. Preprint available. URL GitHub