Home | News

10.06.2026

Teaser image to How Should Researchers Report Their Use of LLMs?

How Should Researchers Report Their Use of LLMs?

MCML Research Insight – With Stefan Feuerriegel, Barbara Plank, Kerstin Forster, Dominique Geissler, Abdurahman Maarouf, Sebastian Maier

Large Language Models (LLMs) are increasingly becoming part of scientific research. They can generate text, analyze data, simulate participants, and support researchers in entirely new ways.

But there is a catch: even small changes in prompts, settings, or model versions can lead to substantially different results. Without clear reporting, it becomes difficult to understand what model was used, how it was configured, which prompts were given, or how outputs were validated.

MCML PI Stefan Feuerriegel, together with Junior Members Kerstin Forster, Dominique Geissler, Abdurahman Maarouf, Sebastian Maier, MCML PI Barbara Plank, and other international collaborators, developed GUIDE-LLM — a framework to help researchers use LLMs more transparently. Their work was published in Nature Human Behaviour, one of the leading journals for interdisciplinary research on human behavior and society.

 Key Insight


Research using LLMs needs new reporting standards.


A Challenge for Research Rigour

Science depends on reproducibility. Researchers need to understand how a study was conducted in order to evaluate or replicate it. That becomes harder with LLMs. The same model may behave differently depending on its version, size, runtime environment, or system instructions, and even small changes in prompts can lead to substantially different outputs. At the same time, models may also reproduce biases present in their training data.

Without proper reporting, this can lead to:

  • Replication failures
  • Misinterpretation of results
  • Difficulty comparing studies across fields
  • Reduced trust in findings using LLMs
  • Challenges for reviewers evaluating methodological quality

 Core Idea


GUIDE-LLM establishes a common reporting standard that makes AI-assisted research easier to evaluate, compare, and reproduce.


A Shared Standard for Transparent AI Research

GUIDE-LLM introduces a reporting framework that helps researchers systematically document how LLMs are used in behavioural and social science research. It was created through a global panel of 80 experts across disciplines including psychology, economics, sociology, AI, and ethics.

The final checklist contains 14 core items that help researchers clearly document:

  • Why and how LLMs were used
  • Which models and versions were used
  • Prompting and configuration choices
  • Validation procedures for LLM-generated outputs
  • Reproducible workflows and code sharing

Importantly, GUIDE-LLM does not tell researchers how to use AI. Instead, it establishes a minimum standard for transparency.


 Takeaway


Without transparent reporting, AI-assisted research risks becoming impossible to evaluate or reproduce.


Why It Matters

As LLMs become standard tools in research, transparency becomes essential for interpreting results. Without it, studies risk becoming difficult to evaluate, compare, or reproduce.

GUIDE-LLM aims to make AI-assisted research visible, structured, and reliable — helping ensure that research remains trustworthy in the age of AI.

 


Further Reading & Reference

The links below lead to the full paper, published in Nature Human Behaviour - a leading journal at the intersection of the social, behavioral, and computational sciences - as well as the project page with additional information and the downloadable checklist.

Top Journal
S. Feuerriegel • C. Barrie • M. J. Crockett • L. K. Globig • K. L. McLoughlin • D.-M. Mirea • A. Spirling • D. Yang • T. Althoff • M. Antoniak • L. P. Argyle • A. Ashokkumar • M. Atari • H. Bailey • K. Bauer • U. Bhatt • Y. Chai • T. Chakraborty • Y. Chandra • H. Chen • H. Daumé III • G. De Francisci Morales • M. Dehghani • D. Dillion • J. C. Eichstaedt • K. Forster • D. Geissler • K. Gray • T. L. Griffiths • J. Hartmann • O. P. Hauser • J. K. He • R. Hemrajani • F. Holzmeister • A. H.-C. Hwang • T. Hu • A. A. Ivanova • N. Köbis • Y. Kyrychenko • H. Lakkaraju • J. Liu • A. Maarouf • S. Maier • L. Meincke • R. Mihalcea • B. Mittelstadt • S. M. Mohammad • M. Naaman • O. Netzer • A. Oh • D. C. Ong • F. Pierri • B. Plank • I. Rahwan • T. Rahwan • P. S. B. Rao • C. E. Robertson • D. M. Rothschild • M. J. Salganik • E. Schulz • C. Shah • Y. R. Shrestha • E. Shutova • A. A. Siegel • A. Simchon • H. Sun • M. Toetzke • J. J. Van Bavel • M. Vaccaro • J. W. Vaughan • E. Vayena • P. O. S. Vaz-de-Melo • B. Vecchione • A. Wang • R. West • R. Willer • D. U. Wulff • R. Zhang • S. Zhang • S. Rathje • M. H. Ribeiro
A reporting checklist for large language models in behavioural science.
Nature Human Behaviour 10. Jun. 2026. PDF URL

Share Your Research!


Get in touch with us!

Are you an MCML Junior Member and interested in showcasing your research on our blog?

We’re happy to feature your work. Get in touch with us to present your paper.

#blog #research #feuerriegel #plank

Related

Link to Enkelejda Kasneci: The Real Value of AI in Education Is More Time for Humans

17.11.2026

Enkelejda Kasneci: The Real Value of AI in Education Is More Time for Humans

Enkelejda Kasneci explores how AI can provide personalized learning support while keeping human interaction and student agency at the center.

Read more
Link to One Toolbox for Many Kinds of Uncertainty

12.11.2026

One Toolbox for Many Kinds of Uncertainty

MCML researchers introduce probly, a unified Python package for uncertainty-aware machine learning, presented orally at NeurIPS 2026.

Read more
Link to Outstanding Paper Award at EMNLP 2026 for Work on Accent Bias in Audio Language Models

09.11.2026

Outstanding Paper Award at EMNLP 2026 for Work on Accent Bias in Audio Language Models

A paper co-authored by MCML PI Valentin Hofmann on accent bias in audio language models won an Outstanding Paper Award at EMNLP 2026.

Read more
Link to Best Thesis Award for Sohir Maskey

06.11.2026

Best Thesis Award for Sohir Maskey

Former MCML Junior Member Sohir Maskey received the Best Thesis Award of the DFG Priority Programme 2298 for his PhD on graph neural networks.

Read more
Tiny logo
Link to MCML at CIKM 2026

06.11.2026

MCML at CIKM 2026

MCML researchers are represented with 1 paper at CIKM 2026.

Read more
Back to Top