Rewarding Doubt: A Reinforcement Learning Approach to Confidence Calibration of Large Language Models

MCML Authors

David Bani-Harouni

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

Chantal Pellegrini

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

Ege Özsoy

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

Kamilia Zaripova

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

Matthias Keicher

Dr.

→ Group Nassir Navab
 Computer Aided Medical Procedures & Augmented Reality

Nassir Navab

Prof. Dr.

Principal Investigator

Computer Aided Medical Procedures & Augmented Reality

Abstract

A safe and trustworthy use of Large Language Models (LLMs) requires an accurate expression of confidence in their answers. We introduce a novel Reinforcement Learning (RL) approach for LLM calibration that fine-tunes LLMs to elicit calibrated confidence estimations in their answers to factual questions. We model the problem as a betting game where the model predicts a confidence score together with every answer, and design a reward function that penalizes both over and under-confidence. We prove that under our reward design an optimal policy would result in a perfectly calibrated confidence estimation. Our experiments demonstrate significantly improved confidence calibration and generalization to new tasks without re-training, indicating that our approach teaches a general confidence awareness. This approach enables the training of inherently calibrated LLMs.

misc

Preprint

Authors

Links

Research Area

 C1 | Medicine