Home | Publications | NKK+26b

Who Flips? Self- And Cross-Model Counterarguments Reveal Answer Instability in LLMs

MCML Authors

Nafiseh Nikeghbal

→ Group Jana Diesner
Human-Centered Computing

Amir Hossein Kargaran

→ Group Hinrich Schütze
Computational Linguistics

Shaghayegh Kolli

→ Group Jana Diesner
Human-Centered Computing

Jana Diesner

Prof. Dr.

Core PI

Human-Centered Computing

Abstract

Standard accuracy benchmarks evaluate whether large language models (LLMs) reach correct answers. However, they do not test whether models maintain that answer when challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean 7.1pp, up to 18.7pp). Furthermore, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. From this cross-model pool, we construct MaxFlip, a curated benchmark that amplifies answer flips by up to 23.6pp over self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks.

inproceedings NKK+26b

Findings @EMNLP 2026

Findings of the Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026. To be published. Preprint available.