Home | Publications | Kon25

How ML-Filtered Answer Options Shape Responses and Interactions in CATI Surveys

MCML Authors

Olga Kononykhina

→ Group Frauke Kreuter
Social Data Science and AI

Abstract

Occupational coding has historically been a manual, post-survey task, but tools like OccuCoDe are shifting this process into real-time surveys using machine learning (ML). OccuCoDe dynamically filters and presents tailored answer options, allowing respondents themselves to select the description that best matches their occupation. However, our study revealed low agreement between such respondent-driven ML-based coding and post-survey manual coding, prompting us to explore how the quality of responses in automatic occupational coding relates to the quality of answer options, respondent and interviewer behaviors. We embedded OccuCoDe into a standard monthly multi-topic survey conducted by the Institute for Applied Social Science (INFAS) from 1 April to 31 June 2019. The survey was designed as a cross-sectional and panel survey with a 30:70 ratio for panel and new respondents, resulting in a representative sample of adults in Germany aged 18 and older. We received and analyzed 669 audio recordings through behavioral coding. Results showed that the quality of ML-generated suggestions significantly influenced classification accuracy, with highly accurate suggestion leading to better alignment with manual coding. Contrary to expectations, behavioral factors such as interviewer adherence to scripts or respondent mapping or comprehension issues were not the significant drivers of mismatches. Instead, familiar survey dynamics persisted: respondents often interrupted when they identified an option they liked, or interviewers skipped certain categories (e.g., 'Other'). These findings suggest that while integrating ML or other AI tools into surveys is potentially fruitful, the key to success lies in refining the precision and distinctiveness of answer options. We also demonstrate that, although both respondents and interviewers showed adaptability to the presence of an automatic component, their behaviors could not overcome mismatches caused by limitations in ML-generated suggestions. In occupational coding—and potentially other survey domains—the effectiveness of real-time ML/AI integration depends on aligning algorithmic outputs with respondent realities to achieve high-quality data.

inproceedings

AAPOR 2025

AAPOR 80th Annual Conference on Reshaping Democracy's Oracle: TransForming Polls, Surveys, and the Measurement of Public Opinion in the Age of Al. St. Louis, MO, USA, May 14-16, 2025.