Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges

MCML Authors

Bolei Ma

→ Group Frauke Kreuter
 Social Data Science and AI

Yang Janet Liu

* Former Member

→ Group Barbara Plank
 AI and Computational Linguistics

Frauke Kreuter

Prof. Dr.

Principal Investigator

Social Data Science and AI

Barbara Plank

Prof. Dr.

Principal Investigator

AI and Computational Linguistics

Abstract

Understanding pragmatics-the use of language in context-is crucial for developing NLP systems capable of interpreting nuanced language use. Despite recent advances in language technologies, including large language models, evaluating their ability to handle pragmatic phenomena such as implicatures and references remains challenging. To advance pragmatic abilities in models, it is essential to understand current evaluation trends and identify existing limitations. In this survey, we provide a comprehensive review of resources designed for evaluating pragmatic capabilities in NLP, categorizing datasets by the pragmatics phenomena they address. We analyze task designs, data collection methods, evaluation approaches, and their relevance to real-world applications. By examining these resources in the context of modern language models, we highlight emerging trends, challenges, and gaps in existing benchmarks. Our survey aims to clarify the landscape of pragmatic evaluation and guide the development of more comprehensive and targeted benchmarks, ultimately contributing to more nuanced and context-aware NLP models.

inproceedings

---

ACL 2025

63rd Annual Meeting of the Association for Computational Linguistics. Vienna, Austria, Jul 27-Aug 01, 2025.

---

Authors

B. Ma • Y. Li • W. Zhou • Z. Gong • Y. J. Liu • K. Jasinskaja • A. Friedrich • J. Hirschberg • F. Kreuter • B. Plank

---

Links

URL

---

Research Areas

 B2 | Natural Language Processing

 C4 | Computational Social Sciences