XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs

MCML Authors

Ercong Nie

→ Group Hinrich Schütze
 Computational Linguistics

Haotian Ye

→ Group Hinrich Schütze
 Computational Linguistics

Hinrich Schütze

Prof. Dr.

Principal Investigator

Computational Linguistics

Abstract

We introduce XCOMPS in this work, a multilingual conceptual minimal pair dataset covering 17 languages. Using this dataset, we evaluate LLMs' multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. By comparing base, instruction-tuned, and knowledge-distilled models, we find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) Instruction tuning improves performance in concept understanding but does not enhance internal competence; knowledge distillation can enhance internal competence in conceptual understanding for low-resource languages with limited gains in explicit task performance. 4) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning.

inproceedings

---

SIGTYP @ACL 2025

7th Workshop on Research in Computational Linguistic Typology and Multilingual NLP at the 63rd Annual Meeting of the Association for Computational Linguistics. Vienna, Austria, Jul 27-Aug 01, 2025.

---

Authors

L. He • E. Nie • S. S. Dindar • A. Firoozi • A. Florea • V. Nguyen • C. Puffay • R. Shimizu • H. Ye • J. Brennan • H. Schmid • H. Schütze • N. Mesgarani

---

Links

URL

---

Research Area

 B2 | Natural Language Processing