Extracting Linguistic Information From Large Language Models: Syntactic Relations and Derivational Knowledge
MCML Authors
Tsedeniya Kinfe Temesgen
Marion Di Marco
Dr.
Alexander Fraser
Prof. Dr.
Core PI
Abstract
Tsedeniya Kinfe Temesgen
Marion Di Marco
Dr.
Alexander Fraser
Prof. Dr.
Core PI
Abstract
This paper presents a study of the linguistic knowledge and generalization capabilities of Large Language Models (LLMs), focusing ontheir morphosyntactic competence. We design three diagnostic tasks: (i) labeling syntactic information at the sentence level - identifying subjects, objects, and indirect objects; (ii) derivational decomposition at the word level - identifying morpheme boundaries and labeling thedecomposed sequence; and (iii) in-depth study of morphological decomposition in German and Amharic. We evaluate prompting strategies in GPT-4o and LLaMA 3.3-70B to extract different types of linguistic structure for typologically diverse languages. Our results showthat GPT-4o consistently outperforms LLaMA in all tasks; however, both models exhibit limitations and show little evidence of abstract morphological rule learning. Importantly, we show strong evidence that the models fail to learn underlying morphological structures. Therefore,raising important doubts about their ability to generalize.
inproceedings TDF25
EMNLP 2025
Conference on Empirical Methods in Natural Language Processing. Suzhou, China, Nov 04-09, 2025.
Authors
T. K. Temesgen • M. Di Marco • A. FraserLinks
DOIResearch Area
BibTeXKey: TDF25