Aligning NLP Models With Target Population Perspectives Using PAIR: Population-Aligned Instance Replication

MCML Authors

Bolei Ma

→ Group Frauke Kreuter
 Social Data Science and AI

Christoph Kern

Prof. Dr.

Associate

Social Data Science and AI

Barbara Plank

Prof. Dr.

Principal Investigator

AI and Computational Linguistics

Frauke Kreuter

Prof. Dr.

Principal Investigator

Social Data Science and AI

Abstract

Models trained on crowdsourced labels may not reflect broader population views when annotator pools are not representative. Since collecting representative labels is challenging, we propose Population-Aligned Instance Replication (PAIR), a method to address this bias through statistical adjustment. Using a simulation study of hate speech and offensive language detection, we create two types of annotators with different labeling tendencies and generate datasets with varying proportions of the types. Models trained on unbalanced annotator pools show poor calibration compared to those trained on representative data. However, PAIR, which duplicates labels from underrepresented annotator groups to match population proportions, significantly reduces bias without requiring new data collection. These results suggest statistical techniques from survey research can help align model training with target populations even when representative annotator pools are unavailable. We conclude with three practical recommendations for improving training data quality.

inproceedings

---

NLPerspectives @EMNLP 2025

4th Workshop on Perspectivist Approaches to NLP at the Conference on Empirical Methods in Natural Language Processing. Suzhou, China, Nov 04-09, 2025. To be published. Preprint available.

---

Authors

S. Eckman • B. Ma • C. Kern • R. Chew • B. Plank • F. Kreuter

---

Links

---

Research Areas

 B2 | Natural Language Processing

 C4 | Computational Social Sciences