Disentangling Subjectivity and Uncertainty for Hate Speech Annotation and Modeling Using Gaze

MCML Authors

Andreas Säuberli

→ Group Barbara Plank
 AI and Computational Linguistics

Barbara Plank

Prof. Dr.

Principal Investigator

AI and Computational Linguistics

Abstract

Variation is inherent in opinion-based annotation tasks like sentiment or hate speech analysis. It does not only arise from errors, fatigue, or sentence ambiguity but also from genuine differences in opinion shaped by background, experience, and culture. In this paper, first, we show how annotators’ confidence ratings can be great use for disentangling subjective variation from uncertainty, without relying on specific features present in the data (text, gaze, etc.). Our goal is to establish distinctive dimensions of variation which are often not clearly separated in existing work on modeling annotator variation. We illustrate our approach through a hate speech detection task, demonstrating that models are affected differently by instances of uncertainty and subjectivity. In addition, we show that human gaze patterns offer valuable indicators of subjective evaluation and uncertainty. Disclaimer: This paper contains sentences that may be offensive.

inproceedings AHS+25

---

EMNLP 2025

Conference on Empirical Methods in Natural Language Processing. Suzhou, China, Nov 04-09, 2025.

---

Authors

Ö. Alacam • S. Hoeken • A. Säuberli • H. Gröner • D. Frassinelli • S. Zarrieß • B. Plank

---

Links

DOI

---

Research Area

 B2 | Natural Language Processing