How Should Researchers Report Their Use of LLMs?

Large Language Models (LLMs) are increasingly becoming part of scientific research. They can generate text, analyze data, simulate participants, and support researchers in entirely new ways.

But there is a catch: even small changes in prompts, settings, or model versions can lead to substantially different results. Without clear reporting, it becomes difficult to understand what model was used, how it was configured, which prompts were given, or how outputs were validated.

MCML PI Stefan Feuerriegel, together with Junior Members Kerstin Forster, Dominique Geissler, Abdurahman Maarouf, Sebastian Maier, MCML PI Barbara Plank, and other international collaborators, developed GUIDE-LLM — a framework to help researchers use LLMs more transparently. Their work was published in Nature Human Behaviour, one of the leading journals for interdisciplinary research on human behavior and society.

A Challenge for Research Rigour

Science depends on reproducibility. Researchers need to understand how a study was conducted in order to evaluate or replicate it. That becomes harder with LLMs. The same model may behave differently depending on its version, size, runtime environment, or system instructions, and even small changes in prompts can lead to substantially different outputs. At the same time, models may also reproduce biases present in their training data.

Without proper reporting, this can lead to:

• Replication failures
• Misinterpretation of results
• Difficulty comparing studies across fields
• Reduced trust in findings using LLMs
• Challenges for reviewers evaluating methodological quality

---

A Shared Standard for Transparent AI Research

GUIDE-LLM introduces a reporting framework that helps researchers systematically document how LLMs are used in behavioural and social science research. It was created through a global panel of 80 experts across disciplines including psychology, economics, sociology, AI, and ethics.

The final checklist contains 14 core items that help researchers clearly document:

• Why and how LLMs were used
• Which models and versions were used
• Prompting and configuration choices
• Validation procedures for LLM-generated outputs
• Reproducible workflows and code sharing

Importantly, GUIDE-LLM does not tell researchers how to use AI. Instead, it establishes a minimum standard for transparency.

---

Why It Matters

As LLMs become standard tools in research, transparency becomes essential for interpreting results. Without it, studies risk becoming difficult to evaluate, compare, or reproduce.

GUIDE-LLM aims to make AI-assisted research visible, structured, and reliable — helping ensure that research remains trustworthy in the age of AI.

 

---

Further Reading & Reference

The links below lead to the full paper, published in Nature Human Behaviour - a leading journal at the intersection of the social, behavioral, and computational sciences - as well as the project page with additional information and the downloadable checklist.

---

Subscribe to RSS News feed