Home | Publications | BEM+24

Order Effects in Annotation Tasks: Further Evidence of Annotation Sensitivity

MCML Authors

Link to Profile Frauke Kreuter

Frauke Kreuter

Prof. Dr.

Core PI

Social Data Science and AI

Abstract

The data-centric revolution in AI has revealed the importance of high-quality training data for developing successful AI models. However, annotations are sensitive to annotator characteristics, training materials, and to the design and wording of the data collection instrument. This paper explores the impact of observation order on annotations. We find that annotators’ judgments change based on the order in which they see observations. We use ideas from social psychology to motivate hypotheses about why this order effect occurs. We believe that insights from social science can help AI researchers improve data and model quality.

inproceedings BEM+24

UncertaiNLP @EACL 2024

1st Workshop on Uncertainty-Aware NLP at the 18th Conference of the European Chapter of the Association for Computational Linguistics. St. Julians, Malta, Mar 17-22, 2024.

Authors

J. Beck • S. Eckman • B. Ma • R. Chew • F. Kreuter

Links

URL

Research Area

 C4 | Computational Social Sciences

BibTeXKey: BEM+24

#p-kreuter
Back to Top