Home | Publications | BEM+24

Order Effects in Annotation Tasks: Further Evidence of Annotation Sensitivity

MCML Authors

Jacob Beck

→ Group Frauke Kreuter
Social Data Science and AI

Bolei Ma

→ Group Frauke Kreuter
Social Data Science and AI

Frauke Kreuter

Prof. Dr.

Principal Investigator

Social Data Science and AI

Abstract

The data-centric revolution in AI has revealed the importance of high-quality training data for developing successful AI models. However, annotations are sensitive to annotator characteristics, training materials, and to the design and wording of the data collection instrument. This paper explores the impact of observation order on annotations. We find that annotators’ judgments change based on the order in which they see observations. We use ideas from social psychology to motivate hypotheses about why this order effect occurs. We believe that insights from social science can help AI researchers improve data and model quality.

inproceedings

UncertaiNLP @EACL 2024

1st Workshop on Uncertainty-Aware NLP at the 18th Conference of the European Chapter of the Association for Computational Linguistics. St. Julians, Malta, Mar 17-22, 2024.