Prompt-Robust Language Models: Which Training Strategies Work?

MCML Authors

Frederic Sadrieh

→ Group Hinrich Schütze
 Computational Linguistics

Abstract

Despite their strong performance, large language models remain highly sensitive to prompt formulation. Prior work has addressed this issue through refined data construction or more complex robustness-oriented training methods. In this work, we reproduce and compare these strategies under controlled conditions, evaluating their effects on worst-case prompt sensitivity and average performance. We find that conventional instruction fine-tuning offers limited reductions in prompt sensitivity: across methods, the average gap between the best- and worst-performing prompts remains 40–57% of performance. Moreover, recent robustness-enhancing methods often fail to outperform simpler data construction strategies; in worst-case settings, at least one simple method performs better in 5 of 8 evaluations. Overall, our findings show that prompt sensitivity remains an open problem, and motivate practitioners to prioritize simpler data construction techniques before adopting more sophisticated robustness methods.

inproceedings SS26b

---

Findings @EMNLP 2026

Findings of the Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026.

---

Authors

F. Sadrieh • M. Štefánik

---

Links

DOI

---

Research Area

 B2 | Natural Language Processing