Helpful, Harmless, Honest? RLHF as Survey Design and Content Moderation

MCML Authors

Frauke Kreuter

Prof. Dr.

Core PI

Social Data Science and AI

Abstract

Reinforcement Learning from Human Feedback (RLHF) is widely credited as the process that transformed large language models (LLMs) from narrow text generators into general-purpose, steerable systems. RLHF is seen as the primary mechanism for aligning these models with human values such as helpfulness, harmlessness, and honesty. Instead of writing brittle, hand-designed specifications, RLHF learns an approximation of human preferences to resolve pluralistic, contested, and context-dependent preferences. In RLHF practice, however, the resulting preferences are neither free-form nor pluralistic: raters are instructed in advance how to interpret 'good' behavior, disagreements are filtered or collapsed, and the training signal encodes a highly constrained normative stance. We argue that RLHF is best understood as a hybrid of two familiar socio-technical practices: survey research, because it elicits, filters, and aggregates human judgments into a training signal, and content moderation, because it directly determines which behaviors and outputs are permitted or suppressed. We analyze current RLHF pipelines through these lenses, focusing in particular on two stages with strong normative consequences: rater selection and rater instruction. From the survey perspective, we find that contemporary RLHF pipelines fall short of well-established standards, particularly with respect to construct validity, population definition, aggregation, and interpretation. From the content moderation perspective, we find that normative constraints are embedded deeply in these systems and are correspondingly difficult to scrutinize. We present a critical reading of OpenAI's public alignment-related technical reports, system cards, and product documentation from 2017 to the present, using an interactive visualization (here) to trace how their design rationales and interaction contracts have changed over time. We show that current disclosure practices around RLHF are insufficient to support meaningful pluralistic evaluation of deployed systems. We conclude that viewing RLHF as both survey and content moderation demands far stronger standards of transparency and scrutiny.

inproceedings DKB26

---

ACM FAccT 2026

9th ACM Conference on Fairness, Accountability, and Transparency. Montréal, Canada, Jun 25-28, 2026.

---

Authors

S. D'Alonzo • F. Kreuter • S. Booth

---

Links

DOI

---

Research Area

 C4 | Computational Social Sciences