Home | News

29.10.2026

Teaser image to Teaching Language Models to Express Uncertainty

Teaching Language Models to Express Uncertainty

MCML Research Insight – With David Bani-Harouni, Chantal Pellegrini, Ege Özsoy, Kamilia Zaripova, Nassir Navab, and Matthias Keicher

An AI assistant may give a confident answer even when it is wrong. But what if it could tell us how sure it really is? MCML researchers have developed a reinforcement learning approach that teaches language models to express more reliable confidence, helping us recognize when to trust an answer and when to double-check.


Why a Confident Answer Is Not Always a Reliable One

 Key Insight


An AI system should express high confidence only when its answers are likely to be correct.


In the paper Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models, MCML Junior Members David Bani-Harouni, Chantal Pellegrini, Ege Özsoy, Kamilia Zaripova, and Matthias Keicher, together with MCML PI Nassir Navab and co-author Paul Stangel, investigate how language models can express more reliable confidence in their answers. The paper was presented at the International Conference on Learning Representations (ICLR 2026).

The growing interest in reliable AI confidence estimates is also reflected in recent developments such as TypeSafe AI’s JEV Decision Model. Independent analyses have highlighted Rewarding Doubt as earlier research addressing the same fundamental challenge: training AI systems with reinforcement learning to express confidence that matches their actual reliability. This connection shows how the MCML team’s work contributes to a research direction that is now attracting broader attention. For further context, see the discussions by Motion and Madness and Anthony Maio.

Confidence is calibrated when the model’s stated certainty matches how often it is correct. For example, among answers given with 80% confidence, roughly 80% should be correct. This matters when an AI response informs a decision that may need further checking.


Why Models Struggle to Judge Their Own Answers

 Big Question


Can a language model learn to say how sure it is, rather than simply sound sure?


Language models often provide answers in a fluent, assured tone, even when those answers contain mistakes. Simply asking a model to state its confidence does not guarantee that the number reflects its actual reliability.

Earlier approaches often estimated confidence by analyzing the model’s outputs or using a separate mechanism. The researchers instead ask whether confidence can be trained directly into the model’s response process.


Learning Through Rewards and Penalties

 Core Idea


Train a model to express confidence that reflects whether its answers are correct.


The researchers introduce Rewarding Doubt, a reinforcement-learning method that trains language models to express a numerical confidence estimate alongside an answer. Reinforcement learning improves a model’s behavior by assigning rewards to its outputs during training.

Their reward follows a logarithmic scoring rule: a correct answer with high confidence earns a stronger reward, while an incorrect answer given with high confidence receives a stronger penalty. The design also discourages unnecessary doubt about correct answers. During training, the answer is generated first and kept fixed while the confidence expression is optimized, allowing the researchers to focus on calibration rather than changing the answer itself.

Illustration of the reward design

Figure 1: Illustration of the reward design: The model is rewarded for a high confidence if it is correct and punished if it is incorrect. To maximize the reward, the model needs to learn when to predict a higher or lower confidence, considering a possible higher punishment.


More Accurate Confidence, Including on New Tasks

 Key Finding


Training improves how closely a model's confidence matches its accuracy, including on tasks outside its training data.


The researchers evaluate their method on factual question answering with single and multiple answers. Compared with simply asking an untrained model to state its confidence, Rewarding Doubt substantially reduces calibration error. On the TriviaQA benchmark, expected calibration error falls from 0.3459 to 0.0226. A separately trained confidence probe achieves similarly strong calibration, with a slightly lower calibration error on this benchmark. Unlike Rewarding Doubt, however, this approach does not train the language model itself to express calibrated confidence.

The team also tests models of different sizes and architectures and evaluates performance on previously unseen question-answering tasks, including medical questions. The results show improved confidence estimates across these settings, while answer accuracy remains broadly stable in the reported experiments.


Knowing When to Double-Check

 Takeaway


Better confidence estimates can help people recognize when an AI answer needs verification.


A confidence estimate is useful only when it corresponds to the model’s actual likelihood of being correct. Rewarding Doubt shows that language models can be trained to communicate that uncertainty more reliably, without requiring a separate confidence model at the point of use.

Whether the approach performs similarly on much larger models or more open-ended responses remains to be tested. The work provides a research basis for AI systems that make uncertainty visible rather than hiding it behind fluent language.


Further Reading & Reference

Want to see how language models can learn when to express confidence and when to express doubt? The full ICLR 2026 paper explains the reward design and evaluates how well the resulting confidence estimates match real answer accuracy.

A* Conference
D. Bani-Harouni • C. Pellegrini • P. Stangel • E. Özsoy • K. Zaripova • N. Navab • M. Keicher
Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models.
ICLR 2026 - 14th International Conference on Learning Representations. Rio de Janeiro, Brazil, Apr 23-27, 2026. To be published. Preprint available. URL GitHub

Share Your Research!


Get in touch with us!

Are you an MCML Junior Member and interested in showcasing your research on our blog?

We’re happy to feature your work—get in touch with us to present your paper.

#blog #research #navab

Related

Link to Enkelejda Kasneci: The Real Value of AI in Education Is More Time for Humans

17.11.2026

Enkelejda Kasneci: The Real Value of AI in Education Is More Time for Humans

Enkelejda Kasneci explores how AI can provide personalized learning support while keeping human interaction and student agency at the center.

Read more
Link to One Toolbox for Many Kinds of Uncertainty

12.11.2026

One Toolbox for Many Kinds of Uncertainty

MCML researchers introduce probly, a unified Python package for uncertainty-aware machine learning, presented orally at NeurIPS 2026.

Read more
Link to Outstanding Paper Award at EMNLP 2026 for Work on Accent Bias in Audio Language Models

09.11.2026

Outstanding Paper Award at EMNLP 2026 for Work on Accent Bias in Audio Language Models

A paper co-authored by MCML PI Valentin Hofmann on accent bias in audio language models won an Outstanding Paper Award at EMNLP 2026.

Read more
Link to Best Thesis Award for Sohir Maskey

06.11.2026

Best Thesis Award for Sohir Maskey

Former MCML Junior Member Sohir Maskey received the Best Thesis Award of the DFG Priority Programme 2298 for his PhD on graph neural networks.

Read more
Tiny logo
Link to MCML at CIKM 2026

06.11.2026

MCML at CIKM 2026

MCML researchers are represented with 1 paper at CIKM 2026.

Read more
Back to Top