04.12.2025

When to Say "I’m Not Sure": Making Language Models More Self-Aware

MCML Research Insight - With Yawei Li, David Rügamer, Bernd Bischl, and Mina Rezaei

In their ICLR 2025 paper “Calibrating LLMs with Information-Theoretic Evidential Deep Learning”, MCML Junior Members Yawei Li and Mina Rezaei, with MCML PIs David Rügamer and Bernd Bischl, offer a solution. Using statistical methods, they help LLMs stay honest about what they know and what they don’t, and when to admit it.

---

How it Works

Normally, to check how confident an AI model is, you have to run it many times and compare the answers, a bit like asking ten friends the same question and seeing if they agree. That’s slow and expensive. This paper uses a smarter approach called Evidential Deep Learning (EDL). It lets a model estimate its own confidence in one quick run. Along with an answer, the model gives a number showing how strong its “evidence” is:

• More evidence means higher confidence
• Less evidence means more uncertainty

However, when fine-tuned on small datasets, models can still fake confidence, producing too much evidence even when they shouldn’t. That’s where the Information Bottleneck (IB) comes in.

---

The “Information Bottleneck” Idea

Imagine the model’s brain as a pipeline: data goes in, predictions come out. Along the way, it gathers a lot of information, some useful, while some is noisy or misleading.

The bottleneck acts like a filter, squeezing out unnecessary details and keeping only what truly helps the model decide. For instance, when answering a science question, a model might guess “gravity” just because the word “space” appears. The bottleneck stops this by focusing only on genuinely relevant clues.

In the combined IB-EDL approach, this filter also gives a penalty to the model whenever it tries to increase its confidence using that irrelevant information, helping it stay accurate and humble about what it knows.

The expected result: predictions that are still strong, but realistically cautious.

---

Results

The team tested IB-EDL on popular large language models like Llama-2, Llama-3, and Mistral-7B using commonsense reasoning and reading-comprehension tasks.

Compared to normal fine-tuning, IB-EDL:

• Reduced overconfidence by about 70%, making the models become much better at knowing when to be unsure.
• Kept accuracy just as high or even slightly better.
• Got better at spotting unfamiliar questions, realizing when something was outside its training experience.

Even when the researchers intentionally added mistakes to some of the training data, the models trained with IB-EDL still stayed stable and reliable, showing that this method helps them stay grounded even when there is noise in the data.

---

Why it Matters

Being smart isn’t enough for AI, it must also know when to be unsure and to let us know. IB-EDL gives these large models a built-in sense of uncertainty, helping them communicate confidence realistically.

That makes them safer, fairer, and far more trustworthy, which is a key step toward truly honest AI.

---

Challenges

While this method works well for question-answering and classification, measuring uncertainty in open-ended text generation, like creative writing, remains tricky. Future work aims to extend this approach so models can also stay honest when generating longer, freer responses.

---

Further Reading & Reference

If you’d like to learn more about how the team measured and reduced overconfidence in large language models, or explore the method in your own experiments, check out the full paper appeared at ICLR 2025, one of the world’s leading machine learning conferences.

---

#blog #research #bischl #ruegamer

---