Uncertainty Quantification to Improve the Reliability of LLM-Based Hate Speech Detection
MCML Authors
Abstract
Abstract
While Large Language Models (LLMs) have been tested for hate speech detection, their reliability in subjective annotation settings remains underexplored. Existing improved prompting approaches for hate speech classification often rely on continuous context retrieval, which can be computationally expensive and unnecessary for confident predictions. To address this, we propose an uncertainty-aware prompting pipeline that integrates Uncertainty Quantification (UQ) into LLM-based hate speech classification. The framework uses uncertainty scores to identify ambiguous and potentially unreliable predictions through threshold-based filtering, enabling selective application of advanced context-enriched prompting only when needed. We evaluate the approach on two benchmark datasets—Measuring Hate Speech (Sachdeva et al., 2022) and CREHate (Lee et al., 2024—across five LLMs using two advanced prompting strategies: Persona-Aware and Annotation-Grounded Few-Shot Prompting. Experimental results show that the proposed pipeline consistently improves classification performance and reduces uncertainty, leading to more reliable model behavior with improved computational efficiency.
inproceedings YDF26
NLP4PI @EMNLP 2026
5th Workshop on NLP for Positive Impact at the Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026.Authors
N. I. Yuksel • D. Dementieva • A. FraserLinks
DOI GitHubResearch Area
BibTeXKey: YDF26