Home | Publications | CLP+25

References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation

MCML Authors

Link to Profile Barbara Plank

Barbara Plank

Prof. Dr.

Core PI

AI and Computational Linguistics

Abstract

Human language production exhibits remarkable richness and variation, reflecting diverse communication styles and intents. However, this variation is often overlooked in summarization evaluation. While having multiple reference summaries is known to improve correlation with human judgments, the impact of the reference set on reference-based metrics has not been ystematically investigated. This work examines the sensitivity of widely used reference-based metrics in relation to the choice of reference sets, analyzing three diverse multi-reference summarization datasets: SummEval, GUMSum, and DUC2004. We demonstrate that many popular metrics exhibit significant instability. This instability is particularly concerning for n-gram-based metrics like ROUGE, where model rankings vary depending on the reference sets, undermining the reliability of model comparisons. We also collect human judgments on LLM outputs for genre-diverse data and examine their correlation with metrics to supplement existing findings beyond newswire summaries, finding weak-to-no correlation. Taken together, we recommend incorporating reference set variation into summarization evaluation to enhance consistency alongside correlation with human judgments, especially when evaluating LLMs.

inproceedings CLP+25

INLG 2025

18th International Natural Language Generation Conference. Hanoi, Vietnam, Nov 05-09, 2025.

Authors

S. Casola • Y. J. Liu • S. Peng • O. Kraus • A. Gatt • B. Plank

Links

URL

Research Area

 B2 | Natural Language Processing

BibTeXKey: CLP+25

#p-plank
Back to Top