Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding

MCML Authors

Esteban Garces Arias

Dr.

→ Group Barbara Plank
 AI and Computational Linguistics

Matthias Aßenmacher

Dr.

→ Group Bernd Bischl
 Statistical Learning and Data Science

Abstract

In open-ended generation, LLMs frequently fall into the ``likelihood trap'', marked by repetitive degeneration and vocabulary dullness, creating a discrepancy between machine-generated and human-written text. While post-hoc tail truncation (e.g., Top-p, Min-p) avoids sampling from the unreliable tail, it can over-sample from the uncalibrated head and misalign generation with human lexical preferences; fixed scalar repetition penalties likewise ignore variation in logit scale across inference steps, potentially disrupting semantic coherence. To address both limitations, we propose Variance-Calibrated Modulation (VCM), a training-free pre-decoding intervention that reshapes the probability distribution before truncation through two dynamic mechanisms: (1) Contextual Searchlight via PMI, which suppresses global stopwords while elevating context-evoked tokens, and (2) Adaptive Self-Debiasing, which uses real-time logit standard deviation for scale-invariant penalization. Across open-ended generation, factual QA, and mathematical reasoning, VCM consistently mitigates the likelihood trap. With negligible computational overhead, VCM integrates with existing decoding strategies, improving diversity, coherence, and, particularly at higher decoding temperatures, reasoning accuracy.

inproceedings DLG+26a

---

EMNLP 2026

Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026. To be published. Preprint available.

---

Authors

Y. Ding • M. Li • E. Garces Arias • M. Aßenmacher • C. Heumann • C. Zhang

---

Links

arXiv GitHub

---

Research Areas

 A1 | Statistical Foundations & Explainability

 B2 | Natural Language Processing